Adapting machine learning model for domain shift data

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for adapting a machine learning model to infer against a target data set in a shift domain from a source data set used to train the machine learning model. One example method generally includes identifying one or more domain-sensitive layers in a machine learning model based on differences between outputs generated by one or more layers in the machine learning model for inputs in a source domain and inputs in a shift domain. A normalized value is updated for each respective domain sensitive layer of the one or more domain sensitive layers based on the mixing factor, the fixed normalized value for the data in the source domain, and the calculated normalized value for the data in the shift domain. The updated normalized value is applied to each respective domain-sensitive layer of the one or more domain-sensitive layers in the machine learning model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 18 / 338,174, filed on June 20, 2023, entitled “Adapting Machine Learning Models for Domain-Shifted Data,” which claims priority and the benefit of U.S. provisional patent application serial number 63 / 377,477, filed on September 28, 2022, entitled “Adapting Machine Learning Models for Domain-Shifted Data,” which is assigned to the assignee of this application, and the entire contents of these patent applications are hereby incorporated by reference in their entirety. Background Art

[0003] Aspects of the present disclosure relate to machine learning models, and more particularly to adapting machine learning models to perform accurate inference on domain-shifted datasets.

[0004] Machine learning models (such as artificial neural networks (ANNs), convolutional neural networks (CNNs), etc.) can be used to perform various actions on input data. These actions may include, for example, data compression, pattern matching (e.g., for biometric authentication), object detection (e.g., for surveillance applications, autonomous driving, etc.), natural language processing (e.g., identification of keywords in spoken speech that trigger the execution of specified operations within the system), or other inference operations in which a model is used to predict something about the state of the environment from which the input data is received. These models may generally be trained using a source data set, which may be different from a target data set that the machine learning model uses as input for inference. For example, in some scenarios where a machine learning model is trained and deployed for use in an object avoidance task in autonomous driving, the source data set may include images, videos, or other content captured in a specific environment with specific equipment in a specific state (e.g., an urban or otherwise highly constructed environment with an imaging device that has relatively clean specific noise and optical properties).

[0005] Generally speaking, machine learning models generate accurate inferences when deployed in environments similar to the environments from which the source data used to train the machine learning models was obtained (e.g., when the source and target datasets are in the same or at least substantially similar domains). However, inference performance may degrade when the domain in which the target dataset was captured is shifted relative to the source dataset. For example, a model trained in an urban or otherwise highly constructed environment may not be able to accurately perform inferences on data captured in a rural environment (and vice versa). In another example, inference performance may degrade when the target dataset includes data captured under different weather or environmental conditions, or when blur or other transformations are introduced in the imaging system. Summary of the invention

[0006] Certain aspects provide a processor-implemented method for adapting a machine learning model to infer from a source data set used to train the machine learning model against a target data set in a shifted domain. An example method generally includes identifying one or more domain-sensitive layers in the machine learning model based on differences between outputs generated by one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain. A normalization value is updated for each corresponding domain-sensitive layer in the one or more domain-sensitive layers based on a mixing factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain. The updated normalization value is applied to each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model.

[0007] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0008] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings depict certain features of various aspects of the disclosure and therefore are not to be considered limiting of the scope of the disclosure.

[0010] Figure 1An example pipeline for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain in accordance with aspects of the present disclosure is illustrated.

[0011] Figure 2 An example test-time normalization layer for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain is illustrated in accordance with aspects of the present disclosure.

[0012] Figure 3 Examples of adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain are illustrated in accordance with aspects of the present disclosure.

[0013] Figure 4 Example operations for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain are illustrated in accordance with aspects of the present disclosure.

[0014] Figure 5 Example implementations of a processing system on which adaptation of a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain are illustrated in accordance with aspects of the present disclosure.

[0015] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0016] Aspects of the present disclosure provide techniques and apparatus for adapting machine learning models to accurately perform inference on domain-shifted data.

[0017] Various applications use machine learning models to process streaming data and generate outputs that can then be used to perform various specified actions within the system. For example, streaming audio data can be captured and processed by a machine learning model to authenticate or otherwise identify a user of the system (e.g., where multiple users with different voice profiles use the same system and the system is customized based on the identity of the user). In another example, streaming video data can be captured and processed by a machine learning model to identify objects within a scene captured by a camera, identify the distances of these objects to reference fiducials, and perform other identification and ranging tasks (e.g., for autonomous driving tasks, surveillance, etc.). In yet another example, time series signal measurements in a wireless communication system (e.g., time series signal measurements of channel quality information (CQI), channel state information (CSI), etc.) can be processed by a machine learning model for various prediction signals and / or beam management techniques, such as predicting a beamforming pattern to be used for communications between a network entity (e.g., a base station) and a user equipment (UE) in a wireless communication system.

[0018] In many cases, these machine learning models may be trained on data in a different domain from the input data on which they perform inference. For example, a machine learning model may be trained using a "clean" or idealized data set that represents a best-case or idealized environment from which the data was captured. In another example, a machine learning model may be trained using a source data set captured from a specific environment. However, machine learning models may generally be deployed in environments outside of these best-case environments or in environments from which the source data set was captured, and therefore, the data input into the machine learning model at inference time may be in a shifted domain relative to the source data set. As an illustrative example, a "clean" data set in a video streaming application may include data captured in a sufficiently bright environment using an imaging device that has no damage or obstructions (e.g., scratches, dust, and / or oil on a lens element that may degrade the image captured by the imaging device), whereas a data set used as input into a machine learning model at inference time may be captured in a dim environment using an imaging device that has various properties that affect the fidelity of the image captured by the imaging device.

[0019] Because the data input into the machine learning model at inference time may be in a shifted domain relative to the source data set, the performance of the machine learning model may be reduced relative to the performance of the machine learning model when performing inference on the data in the source data set. The magnitude of the performance reduction may be related to the magnitude of the shift between the source data set and the data input into the machine learning model at inference time. In general, a smaller domain shift between the source data set and the data input into the machine learning model at inference time and a larger domain shift ratio between the source data set and the data input into the machine learning model at inference time may produce better inference performance. For example, the inference performance of a model performing inference on a received input image captured by an imaging system having optical properties different from but similar to the imaging system used to capture the source data set may be similar to the inference performance of the model performing inference on the source data set. However, the inference performance of a model performing inference on a received input image with additional transformations (such as blur or noise) may be worse than the inference performance of the same model performing inference on a received input image captured by an imaging system having optical properties different from but similar to the imaging system used to capture the source data set.

[0020] In order to account for the change in inference performance caused by the domain shift between the source data set and the target data set used to train the machine learning model, various normalization techniques can be used. For example, batch normalization can be used to normalize input activations based on the mean and variance of the training data set during training and based on the estimation of the input data statistics running at the inference time. However, machine learning models using batch normalization may not significantly compensate for the domain shift between the source data set and the target data set (e.g., the data set of the input to which the machine learning model will perform various inference operations) used to train the machine learning model. For example, in an autonomous driving scenario, a model using batch normalization may continue to be sensitive to domain shifts. Such domain shifts may include, for example, changes between the environment in which the source data set is captured (e.g., historical data used to train the machine learning model) and the environment in which the target data set is captured (e.g., at inference time). In another example, test time adaptation techniques can be used to estimate various normalized statistics from the current test input. However, these techniques may assume a large test batch size and a single or limited domain shift. Because these assumptions may not be realistic, real-world performance of machine learning models using test-time adaptation may also show that these machine learning models are sensitive to various domain shifts of the input data relative to the source datasets used to train these machine learning models.

[0021] Various aspects of the present disclosure provide techniques and apparatus for adapting a machine learning model based on a normalized value associated with a source data set and a normalized value associated with a target data set (which may be domain-shifted relative to the source data set). As discussed in further detail herein, the degree to which the normalized value is associated with the source data set and the degree to which the normalized value is associated with the target data set may be set based on the domain shift sensitivity of each layer of the machine learning model. Generally speaking, a layer that is non-domain-sensitive or has limited sensitivity to domain shift may weight the normalized value associated with the source data set more heavily, while a domain-sensitive layer may weight the normalized value associated with the target data set more heavily, so that the unique properties of the target data set are considered when performing inference on the target data set. By doing so, various aspects of the present disclosure may provide improved inference performance across different data domains. In addition, a cross-domain model does not need to be trained using a training data set that includes data from a large number of domains, which can reduce the amount of computing resources used to train the machine learning model to accurately generate inferences across these different data domains relative to the amount of resources used to train the machine learning model using a training data set that includes data from a large number of domains.

[0022] Example test-time normalization for adapting machine learning models to domain-shifted data

[0023] Figure 1 Depicted is an example pipeline 100 for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain in accordance with aspects of the present disclosure.

[0024] The pipeline 100 generally includes a plurality of layers having weights 112A-112Z of a trained machine learning model 110. The machine learning model 110 can be represented by the expression f θ :x→y represents and can be trained on the source data set in a source data set specific domain. In some aspects, the weights 112A-112Z associated with different layers of the machine learning model 110 can be generated based on a batch normalization technique applied to the source data set, in which the average value μ s and variance (or standard deviation σ s ) is a statistical normalized measure calculated over the source dataset used to train the machine learning model.

[0025] In order to allow the machine learning model to adapt to the domain shift between the source data set and the target data set on which inference is performed, the layers in the machine learning model 110 may be analyzed in a post-training phase to determine the domain sensitivity of each layer. In some aspects, in order to determine the domain sensitivity of each layer in the machine learning model 110, the source data set may be transformed into the target data set by applying one or more domain shift transformations to the source data set. For example, these transformations may include (but are not limited to) adding noise to images in the source data set, removing noise from images in the source data set, converting images in the source data set from color to black and white, introducing color shifts to images in the source data set, introducing visual artifacts (e.g., blur, position shifts, etc.) to images in the source data set, and the like. Although the above discusses transformations that can be applied to image data, it should be recognized that other transformations may be applied to image data, and different transformations may be applied to non-image data (e.g., audio, text, etc.) as appropriate.

[0026] During this post-training phase, the batch normalization layers can be transformed into test-time normalization layers by adjusting the normalization values ​​applied in each of these layers based on whether the layer is domain-sensitive and, if so, the degree to which the layer is domain-sensitive. To do so, for the lth layer and the input data mini-batch size of B, the input to the lth layer can be represented as in C represents the channel, H represents the height dimension, and W represents the width dimension. Normalized statistics and (e.g., mean and variance values), for input (For example, for a small batch input to layer l) can be calculated as follows:

[0027]

[0028] as well as

[0029]

[0030] where b∈B represents the bth part of the mini-batch, h∈H represents the height component of this part of the mini-batch, and w∈W represents the width component of this part of the mini-batch.

[0031] Normalized statistics for the source dataset and Various techniques, such as an exponential moving average over the source data set, can be used to exploit and To estimate the mean of the source data set It can be expressed by the following expression:

[0032]

[0033] In addition, the variance of the source dataset It can be expressed by the following expression:

[0034]

[0035] In a batch normalization layer (such as those in the machine learning model 110 illustrated with weights 112A-112Z), the input z in Normalized statistics can be used and to normalize, and the normalized input can use the learnable parameter γ (l) and β (l) To transform, where γ (l) corresponds to the learned scale parameter used to normalize the input and β (l) corresponds to the learned bias term used to normalize the input.

[0036] In order to allow the machine learning model to adapt to domain-shifted data and generate accurate inferences from data in a different domain from the source dataset used to train the machine learning model, normalized statistics for the source dataset and the target dataset may be combined, as illustrated in the machine learning model 120. For each layer in the machine learning model that performs normalization, a learnable interpolation weight ranging between [0, 1] is (also called a blending factor) can be applied to the normalized statistics for the source dataset. Similarly, 1 and the learnable interpolation weight α (l) The difference can be applied to the normalized statistics for the target dataset. In general, the interpolation weight (or mixing factor) indicates the extent to which layer l and channel c are affected by the domain shift between the source dataset used to train the machine learning model and the target dataset representing the data for which the machine learning model received at inference time will generate one or more inferences. (l) Larger values ​​(e.g., α (l) The value closer to 1) generally indicates that the layer has a (l) Smaller values ​​(e.g., α (l) The closer the value of is to 0), the lower the degree of domain sensitivity.

[0037] like Figure 1, weights for layer A in machine learning model 110 (e.g., weights 112A) may be replaced with frozen weights 122A and batch weights 124A in machine learning model 120. In general, as discussed in further detail below, frozen weights (e.g., 122A-122Z) may be weights associated with training data used to train machine learning model 110, whereas batch weights (e.g., 124A-124Z) may be weights associated with inputs based on which machine learning model 120 generates inferences. Similar replacements may be performed for layers B through Z in machine learning model 120, where Z is any positive integer. In this example, frozen weights 122A-122Z (collectively, “frozen weights 122”) may represent weights associated with a source data set, and batch weights 124A-124Z (collectively, “batch weights 124”) may represent weights associated with a target data set, where the relative magnitudes of frozen weights 122 and batch weights 124 for each layer in machine learning model 120 correspond to the α for the corresponding layer. (l) By freezing the weights associated with the source dataset, the knowledge gained by training the machine learning model using the source dataset can be preserved in the machine learning model to varying degrees depending on the degree to which the layers in the machine learning model are domain sensitive.

[0038] In this example, layer A may have an alpha close to 0.5. (l) , thereby indicating that frozen weights 122A and batch weights 124A have equal influence on the output generated by this layer of the machine learning model 120. In addition, layer B may have an α value of less than 0.5. (l) , indicating that the layer is relatively sensitive to domain shift and therefore the weights associated with the target dataset should have a greater impact on the output generated by layer B. Therefore, when processing the input through layer B of the machine learning model 120, the batch weights 124B may be more heavily weighted than the frozen weights 122B. Finally, as illustrated, layer Z may have an α greater than 0.5. (l) , indicating that the layer is relatively insensitive to domain shift, and therefore the weights associated with the source data set may have a greater impact on the output generated by layer Z without adversely affecting the performance of the machine learning model 120. Therefore, when processing inputs through layer Z of the machine learning model 120, the frozen weights 122Z may be more heavily weighted than the batch weights 124Z.

[0039] The normalized values ​​of layer l and channel c applied in the machine learning model 120 may be expressed as follows:

[0040]

[0041] as well as

[0042]

[0043] Figure 2 An example test-time normalization layer 200 for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain is illustrated in accordance with aspects of the present disclosure. As discussed, the test-time normalization layer 200 can be used as a batch normalization layer in the machine learning models so that the normalization layers in the machine learning models can adjust for the domain shift between the source dataset used to train the machine learning models and the target dataset comprising data for which the machine learning models will generate one or more inferences.

[0044] As illustrated, the test-time normalization layer 200 includes a normalization block 210 and a transformation block 220. The normalization block 210 receives an input z in And based on the following items to generate activation The blending factor α defined for the layer and the normalized value μ calculated for the target dataset in and σ in , and the frozen normalized value μ for the source dataset s and σ s Generally speaking, activation It can be expressed by the following formula:

[0045]

[0046] Transform block 220 uses activation And the output z is generated by the following operation out : An affine transformation using the parameters γ and β discussed above is applied to the activations Output z out It can be expressed by the following formula:

[0047]

[0048] Figure 3 An example pipeline 300 for adapting a machine learning model to perform inference from a source dataset used to train the machine learning model against a target dataset in a shifted domain is illustrated in accordance with aspects of the present disclosure.

[0049] Generally speaking, pipeline 300 may include a first stage 302 in which a priori In the second stage, the mixing factor α is based on the prior The initialization α is refined (e.g., a learned probability distribution over a set of options for the source dataset). To obtain the prior The pre-trained model (e.g., a model including one or more batch normalization layers) can be used at block 310 to generate output (e.g., inference) for an input x from a source dataset using a machine learning model including multiple convolutional layers, conditional batch normalization (CBN) layers, rectifier linear unit (ReLU) layers, and fully connected layers that generate a final output, and used at block 320 to generate output for an input x' from a target dataset. Generally, the target dataset can simulate a domain shift relative to the source dataset such that a pairing of an input in the source dataset with a corresponding shifted input in the target dataset includes semantically similar information (e.g., a color image and a monochrome image of the same subject). Because blocks 310 and 320 use the same machine learning model and thus use the same normalization parameters to generate activations for both x and x' So the activation for input x Can be considered as comparison activation The true value data used for comparison. and The difference between the two represents the input activation z caused by the shift applied to x to generate x' (l,c) With z ′(l,c) The difference between.

[0050] when and When the difference between is large (e.g., above a threshold, in the top percentile interval of the distribution of layer / channel domain sensitive metrics, etc.), it can be determined that the normalized statistic at (l,c) is the target for adapting towards the target dataset and away from the source dataset. is an affine transformation applied to the output activation The transform block (for example, Figure 2 The input of the transform block 220 illustrated in FIG. 2 is a transform block 220, so the gradient of the affine parameters γ and β can be used to measure and The difference between.

[0051] To calculate and The difference between the gradient and can be collected using a pre-trained model that uses normalized statistics generated for the source dataset and a cross entropy loss between the input x and the shifted input x' For each layer l∈[1,…,L] and for each channel c∈[1,…,C l ], the directions of the gradient pairs are compared, and the average of the similarity values ​​across the data in the target dataset is calculated. Gradient similarity s (l,c) It can be calculated according to the following formula:

[0052]

[0053] as well as

[0054]

[0055] Where (g,g ′ ) corresponds to and of and For the sake of and To perform integration, and The average value can be calculated and expressed as s (l,c) ∈[0,1].

[0056] For layers and channels in a machine learning model that are domain sensitive (e.g., have a large difference between the activations generated for input x and the shifted input x'), a larger weight may be applied to the normalized statistics associated with the target dataset to which the shifted input x' belongs rather than to the normalized statistics associated with the source dataset to which the input x belongs. To allow this larger weight to be applied to the normalized statistics associated with the target dataset, a min-max normalization may be computed by inversely computing the similarity scores for the layers and channels of the machine learning model, resulting in a calculation of a relative difference. This relative difference may be amplified to produce a priori The generation of . It can be expressed by the following formula:

[0057]

[0058] where s (l ,. ) denotes all channels in layer l and v(·) denotes the min-max normalization function.

[0059] When using a pre-trained machine learning model with a batch normalization layer to obtain a priori Thereafter, the batch normalization layer of the pre-trained machine learning model may be replaced with a test-time normalization layer that processes the input using the combined statistics of the source and target datasets. In doing so, the trained weights (including affine parameters) of the model may be frozen, and the normalization statistics may be modified to use statistics from both the source and target datasets modified by a mixing factor α. To do so, at block 330, the mixing factor α may be defined as being initialized to The learnable parameters of represent the layers and channels whose normalization statistics will be corrected using the normalization statistics associated with the target dataset.

[0060] When sampling input from an arbitrary target domain different from the source domain where the source dataset is located, the mixing factor α can be optimized or otherwise set in order to correctly normalize the data for the combination statistics. To simulate the domain shift from the source domain to the target domain, the target dataset (representing the domain-shifted version of the source dataset), the cross entropy loss and mean square error loss can be used to allow the model to consistently generate inferences given an input x from the source dataset or a shifted input x' from the target dataset and prevent α from being Diverges significantly. The mean square error loss term can be defined as follows:

[0061]

[0062] In addition, the total loss function can be defined as follows:

[0063]

[0064] where λ is the learned regularization weight hyperparameter.

[0065] Example adaptation of machine learning models to domain-shifted data

[0066] Figure 4 Example operations 400 for adapting a machine learning model to perform inference from a source data set used to train the machine learning model against a target data set in a shifted domain are illustrated in accordance with aspects of the present disclosure. Operations 400 may be performed, for example, by a computing system such as a user equipment (UE) or other computing device such as Figure 5 As illustrated in , a pre-trained machine learning model can be adjusted before deployment (e.g., during or after training) or a machine learning model can be deployed on a computing system or other computing device.

[0067] As illustrated, operations 400 begin at block 410 where one or more domain-sensitive layers in a machine learning model are identified based on differences between outputs generated by the one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain.

[0068] In some aspects, identifying one or more domain-sensitive layers in a machine learning model may be performed on a per-layer basis. For each layer in the machine learning model, a source domain gradient vector of the corresponding layer may be calculated based on forward propagation and back propagation of inputs in the source domain through the corresponding layer. Similarly, a target domain gradient vector of the corresponding layer may be calculated based on forward propagation and back propagation of inputs in the shift domain through the corresponding layer. A similarity score may be calculated between the source domain gradient vector and the domain shift gradient vector. In some aspects, the similarity score may include a distance score between the source domain gradient vector and the domain shift gradient vector, a normalized cosine similarity score, and the like. In some aspects, a domain-sensitive layer may be identified as a layer whose similarity score is above a threshold percentile of the score across similarity scores calculated for each layer in the machine learning model. In some aspects, each layer of the machine learning model to which normalization is applied may be considered a domain-sensitive layer in the machine learning model, and these layers may have varying degrees of batch sensitivity.

[0069] In some aspects, the input in the source domain may be an input corresponding to a source dataset used to train the machine learning model. The input in the shifted domain (also referred to as the target domain) may be an input generated by applying one or more transformations to the input in the source dataset.

[0070] In some aspects, the input in the shift domain may be input obtained at inference time.

[0071] At block 420 , the operation 400 continues by updating the normalization value for each respective one of the one or more domain sensitive layers based on the blending factor, the fixed normalization value for the data in the source domain, and the calculated normalization value for the data in the shifted domain.

[0072] In some aspects, the normalization value for each respective domain-sensitive layer is further updated based on a blending factor being applied to the normalization value for data in the source domain and 1 minus the blending factor being applied to the normalization value for data in the shifted domain. As discussed, larger values ​​of the blending factor may correspond to a lower degree of domain sensitivity, whereas smaller values ​​of the blending factor may correspond to a higher degree of domain sensitivity. In general, as the degree of domain sensitivity increases, the amount by which the normalized statistics of the data in the target domain (e.g., data received at inference time) influence the overall output of the layer of the machine learning model increases.

[0073] In some aspects, the normalized values ​​for the data in the shifted domain include a mean and a variance (or standard deviation) calculated from the data in the shifted domain.

[0074] In some aspects, the mixing factor can be a learnable parameter that is fixed for the machine learning model before deployment. As discussed, to learn the value of the mixing factor, a prior can be obtained based on the difference between the output activations generated for the input x in the source domain and the shifted input x' in the target domain. The mixing factor can be initialized as a prior and then refined based on a cross entropy loss between the input x and x' and a mean squared error loss between the mixing factor and the prior.

[0075] In some aspects, normalization values ​​may be defined as one or more constants in the machine learning model for data in the source domain. By doing so, the learned knowledge about the source domain may be preserved regardless of the extent to which the layers in the machine learning model are adjusted to take into account information about the data in the shifted (target) domain.

[0076] At block 430 , the operation 400 continues by applying the updated normalization value to each respective domain-sensitive layer of the one or more domain-sensitive layers in the machine learning model.

[0077] In some aspects, operation 400 also includes deploying the machine learning model with the updated normalization value for each respective one or more domain sensitive layers in the machine learning model.

[0078] In some aspects, operation 400 also includes receiving input data. One or more inferences are generated based on the received input data and the machine learning model using the updated normalization value for each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model, and one or more actions are taken based on the generated one or more inferences. These actions may include, for example, compressing data (e.g., using an encoder of an autoencoder neural network or a generative adversarial network), decompressing data (e.g., using a decoder of an autoencoder neural network), identifying objects in visual content (still images or image streams), predicting future motion of objects in visual content, and / or other tasks for which a machine learning model can be trained and used to generate inferences on input data.

[0079] Example processing system for adapting machine learning models to domain-shifted data

[0080] Figure 5 Describes a method for adapting machine learning models to domain-shifted data (such as the Figure 4 An example processing system 500 of the type described herein.

[0081] Processing system 500 includes a central processing unit (CPU) 502, which in some examples may be a multi-core CPU. Instructions executed at CPU 502 may be loaded, for example, from a program memory associated with CPU 502, or may be loaded from memory 524.

[0082] The processing system 500 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 504 , a digital signal processor (DSP) 506 , a neural processing unit (NPU) 508 , a multimedia processing unit 510 , and wireless connectivity components 512 .

[0083] An NPU, such as NPU 508, is generally a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0084] An NPU, such as NPU 508, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, the NPU may be part of a dedicated neural network accelerator.

[0085] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0086] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Typically, optimization based on error predictions involves propagating back through the layers of the model and determining the gradient to reduce the prediction error.

[0087] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can thus be configured to input new data slices and quickly process the data slices through the trained model to generate model outputs (e.g., inferences).

[0088] In some implementations, the NPU 508 is part of one or more of the CPU 502 , the GPU 504 , and / or the DSP 506 .

[0089] In some examples, wireless connectivity component 512 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity component 512 is further coupled to one or more antennas 514.

[0090] The processing system 500 may also include one or more sensor processing units 516 associated with any manner of sensors, one or more image signal processors (ISPs) 518 associated with any manner of image sensors, and / or a navigation processor 520, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0091] The processing system 500 may also include one or more input and / or output devices 522, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0092] In some examples, one or more of the processors of processing system 500 may be based on the ARM or RISC-V instruction set.

[0093] Processing system 500 also includes memory 524, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 524 includes computer-executable components that are executable by one or more of the aforementioned processors of processing system 500.

[0094] Specifically, in this example, memory 524 includes a layer identification component 524A, a normalized value update component 524B, a normalized value application component 524C, and a machine learning model 524D. The depicted components, as well as other components not depicted, may be configured to perform various aspects of the methods described herein.

[0095] Generally speaking, the processing system 500 and / or its components may be configured to perform the methods described herein.

[0096] It is worth noting that in other aspects, such as where the processing system 500 is a server computer, various elements of the processing system 500 may be omitted. For example, in other aspects, the multimedia processing unit 510, the wireless connectivity component 512, the sensor processing unit 516, the ISP 518, and / or the navigation processor 520 may be omitted. In addition, various elements of the processing system 500 may be distributed, such as training a model and using the model to generate inferences.

[0097] Sample Clauses

[0098] Specific implementation details of various aspects of the disclosure are described in the following numbered clauses.

[0099] Clause 1: A processor-implemented method comprising: identifying one or more domain-sensitive layers in a machine learning model based on differences between outputs generated by one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain; updating a normalization value for each corresponding domain-sensitive layer in the one or more domain-sensitive layers based on a mixing factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain; and applying the updated normalization value to each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model.

[0100] Clause 2: A method according to clause 1, wherein identifying the one or more domain-sensitive layers in the machine learning model includes, for each corresponding layer of the layers in the machine learning model: calculating a source domain gradient vector for the corresponding layer based on forward propagating and backpropagating the input in the source domain through the corresponding layer; calculating a domain-shifted gradient vector for the layer based on forward propagating and backpropagating the input in the shifted domain through the corresponding layer; and calculating a similarity score between the source domain gradient vector and the domain-shifted gradient vector, wherein a similarity score above a threshold percentile of the similarity scores calculated for the layers in the machine learning model indicates that the corresponding layer is a domain-sensitive layer.

[0101] Clause 3: The method of clause 2, wherein the similarity score comprises a normalized cosine similarity score.

[0102] Clause 4: A method according to any one of clauses 1 to 3, wherein the normalized value for each respective domain-sensitive layer is further updated based on the mixing factor being applied to the normalized value for the data in the source domain and 1 minus the mixing factor being applied to the normalized value for the data in the shifted domain.

[0103] Clause 5: The method of clause 4, wherein the mixing factor comprises a learnable parameter that is fixed for the machine learning model prior to deployment.

[0104] Clause 6: The method of any one of clauses 1 to 5, wherein the normalized value for the data in the shift domain comprises a mean and a standard deviation calculated from the data in the shift domain.

[0105] Clause 7: A method according to any one of clauses 1 to 6, further comprising generating the input in the shifted domain by applying one or more transformations to the input in the source domain.

[0106] Clause 8: A method as described in any of clauses 1 to 7, wherein the input in the shifted domain comprises input obtained at inference time.

[0107] Clause 9: A method according to any one of clauses 1 to 8, further comprising deploying the machine learning model using the updated normalized value for each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model.

[0108] Clause 10: A method according to any one of clauses 1 to 9, further comprising defining the normalized values ​​for the data in the source domain as one or more constants in the machine learning model.

[0109] Clause 11: A method according to any one of clauses 1 to 10, the method further comprising: receiving input data; generating one or more inferences based on the received input data and the machine learning model using the updated normalized value for each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model; and taking one or more actions based on the generated one or more inferences.

[0110] Clause 12: A processing system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 11.

[0111] Clause 13: A processing system comprising: means for performing the method according to any one of clauses 1 to 11.

[0112] Clause 14: A non-transitory computer readable medium comprising computer executable instructions which, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 11.

[0113] Clause 15: A computer program product embodied on a computer readable storage medium, the computer program product comprising: code for performing the method according to any one of clauses 1 to 11.

[0114] Additional considerations

[0115] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality or structures and functionality that are supplementary or alternative to the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present invention.

[0116] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0117] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). For example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0118] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. In addition, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. In addition, "determine" may include resolving, selecting, choosing, establishing, etc.

[0119] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate components that can perform the corresponding functions. These components may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0120] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, the reference to the singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "member for..." is used to explicitly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A processor-implemented method, comprising: identifying one or more domain-sensitive layers in the machine learning model based on differences between outputs generated by the one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain; updating a normalization value for each respective one of the one or more domain-sensitive layers based on a blending factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain; as well as Applying the updated normalization value to each respective one of the one or more domain-sensitive layers in the machine learning model.

2. The method of claim 1 , wherein identifying the one or more domain-sensitive layers in the machine learning model comprises, for each respective one of the layers in the machine learning model: computing a source domain gradient vector for the corresponding layer based on forward propagating and back propagating the input in the source domain through the corresponding layer; computing a domain-shifted gradient vector for the corresponding layer based on forward propagating and back-propagating the input in the shifted domain through the corresponding layer; as well as A similarity score is calculated between the source domain gradient vector and the domain-shifted gradient vector, wherein a similarity score above a threshold percentile of scores across similarity scores calculated for the layer in the machine learning model indicates that the corresponding layer is a domain-sensitive layer. The method of claim 2 , wherein the similarity score comprises a normalized cosine similarity score.

4. The method of claim 1 , wherein the normalized value for each respective domain-sensitive layer is further updated based on the blending factor being applied to the normalized value for data in the source domain and 1 minus the blending factor being applied to the normalized value for data in the shifted domain.

5. The method of claim 4, wherein the mixing factors comprise learnable parameters that are fixed for the machine learning model prior to deployment. 6 . The method of claim 1 , wherein the normalized value for the data in the shift domain comprises a mean and a variance calculated from the data in the shift domain.

7. The method of claim 1, further comprising generating the input in the shifted domain by applying one or more transforms to the input in the source domain. The method of claim 1 , wherein the input in the shifted domain comprises input obtained at inference time.

9. The method of claim 1 , further comprising deploying the machine learning model using the updated normalization value for each corresponding domain-sensitive layer in the one or more domain-sensitive layers in the machine learning model.

10. The method of claim 1, further comprising defining the normalized values ​​for the data in the source domain as one or more constants in the machine learning model.

11. The method according to claim 1, further comprising: Receive input data; generating one or more inferences based on the received input data and the machine learning model using the updated normalized values ​​for each respective one of the one or more domain-sensitive layers in the machine learning model; as well as One or more actions are taken based on the one or more inferences generated.

12. A system, comprising: A memory, wherein executable instructions are stored in the memory; as well as at least one processor configured to execute the executable instructions to cause the system to: identifying one or more domain-sensitive layers in the machine learning model based on differences between outputs generated by the one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain; updating a normalization value for each respective one of the one or more domain-sensitive layers based on a blending factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain; as well as Applying the updated normalization value to each respective one of the one or more domain-sensitive layers in the machine learning model.

13. The system of claim 12, wherein to identify the one or more domain-sensitive layers in the machine learning model, the at least one processor is configured to cause the system to, for each respective one of the layers in the machine learning model: computing a source domain gradient vector for the corresponding layer based on forward propagating and back propagating the input in the source domain through the corresponding layer; computing a domain-shifted gradient vector for the corresponding layer based on forward propagating and back-propagating the input in the shifted domain through the corresponding layer; as well as A similarity score is calculated between the source domain gradient vector and the domain-shifted gradient vector, wherein a similarity score above a threshold percentile of scores across similarity scores calculated for the layer in the machine learning model indicates that the corresponding layer is a domain-sensitive layer. The system of claim 13 , wherein the similarity score comprises a normalized cosine similarity score.

15. The system of claim 12, wherein the normalized value for each respective domain-sensitive layer is further updated based on the blending factor being applied to the normalized value for data in the source domain and 1 minus the blending factor being applied to the normalized value for data in the shifted domain.

16. The system of claim 15, wherein the mixing factors comprise learnable parameters that are fixed for the machine learning model prior to deployment.

17. The system of claim 12, wherein the normalized values ​​for the data in the shift domain include a mean and a standard deviation calculated from the data in the shift domain.

18. The system of claim 12, wherein the at least one processor is further configured to cause the system to generate the input in the shifted domain by applying one or more transformations to the input in the source domain.

19. The system of claim 12, wherein the input in the shifted domain comprises input obtained at inference time.

20. The system of claim 12, wherein the at least one processor is further configured to cause the system to deploy the machine learning model using the updated normalized value for each corresponding domain-sensitive layer of the one or more domain-sensitive layers in the machine learning model.

21. The system of claim 12, wherein the at least one processor is further configured to cause the system to deploy the normalized values ​​for the data in the source domain as one or more constants in the machine learning model.

22. The system of claim 12, wherein the at least one processor is further configured to cause the system to: Receive input data; generating one or more inferences based on the received input data and the machine learning model using the updated normalized values ​​for each respective one of the one or more domain-sensitive layers in the machine learning model; and One or more actions are taken based on the one or more inferences generated.

23. A system, comprising: means for identifying one or more domain-sensitive layers in the machine learning model based on differences between outputs generated by the one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain; means for updating a normalization value for each respective one of the one or more domain-sensitive layers based on a blending factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain; as well as Means for applying an updated normalization value to each respective one of the one or more domain-sensitive layers in the machine learning model.

24. The system of claim 23, wherein the means for identifying the one or more domain-sensitive layers in the machine learning model comprises, for each respective one of the layers in the machine learning model: means for computing a source domain gradient vector for the corresponding layer based on forward propagating and back propagating the input in the source domain through the corresponding layer; means for computing a domain-shifted gradient vector for the respective layer based on forward propagating and back propagating the input in the shifted domain through the respective layer; as well as means for calculating a similarity score between the source domain gradient vector and the domain-shifted gradient vector, wherein a similarity score above a threshold percentile of scores across similarity scores calculated for the layers in the machine learning model indicates that the corresponding layer is a domain-sensitive layer.

25. The system of claim 23, wherein the normalized value for each respective domain-sensitive layer is further updated based on the blending factor being applied to the normalized value for data in the source domain and 1 minus the blending factor being applied to the normalized value for data in the shifted domain.

26. The system of claim 25, wherein the mixing factors comprise learnable parameters that are fixed for the machine learning model prior to deployment.

27. The system of claim 23, further comprising means for generating the input in the shifted domain by applying one or more transformations to the input in the source domain.

28. The system of claim 23, further comprising a component for deploying the machine learning model using the updated normalization value for each respective one of the one or more domain-sensitive layers in the machine learning model.

29. The system of claim 23, further comprising: A component for receiving input data; means for generating one or more inferences based on the received input data and the machine learning model using the updated normalized values ​​for each respective one of the one or more domain-sensitive layers in the machine learning model; as well as Means for taking one or more actions based on the generated one or more inferences.

30. A non-transitory computer readable medium having executable instructions stored thereon, the executable instructions, when executed by at least one processor, performing operations comprising: identifying one or more domain-sensitive layers in the machine learning model based on differences between outputs generated by the one or more layers in the machine learning model for inputs in a source domain and inputs in a shifted domain; updating a normalization value for each respective one of the one or more domain-sensitive layers based on a blending factor, a fixed normalization value for data in the source domain, and a calculated normalization value for data in the shifted domain; as well as Applying the updated normalization value to each respective one of the one or more domain-sensitive layers in the machine learning model.

Images