Domain generalization via batch normalization statistics

Abstract

Generally, this disclosure relates to systems and methods that utilize batch normalized statistics as a means of cross-domain generalization. Specifically, example implementations of this disclosure can generate different representations for different domains by collecting independent batch normalized statistics, which can then be used for mapping between domains in a shared latent space. During testing or inference, samples from unknown test or target domains can be projected into the same shared latent space. Thus, the domain of the target samples can be expressed as a linear combination of known domains, where the combinations are weighted based on the corresponding distances between the batch normalized statistics in the latent space. This same mapping strategy can be applied during training and testing to learn latent representations and robust yet lightweight ensemble models operating within this latent space.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 985,434, filed March 5, 2020. The entire contents of U.S. Provisional Patent Application No. 62 / 985,434 are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to domain generalization in machine learning models. More specifically, this disclosure relates to performing domain generalization by exploring the latent space of batch normalization (“batchnorm”) statistics. Background Technology

[0004] Machine learning models trained on data distributions often fail to generalize to samples from different distributions. This phenomenon, commonly referred to in the literature as domain shift between training and test data, is one of the biggest limitations of data-driven algorithms. The assumption that the limited availability of annotated samples from the test domain can mitigate this problem by using explicit supervision or fine-tuning the model using domain adaptation techniques can be helpful. Unfortunately, this assumption does not always hold true in practice, as collecting samples from every possible environment for real-world applications (e.g., all possible test domains) is often impractical. For example, solutions for autonomous driving require samples from every possible road surface under every possible season and weather condition.

[0005] In contrast to domain adaptation, domain generalization refers to algorithms that address domain shifting problems by training or configuring models to be robust to unseen domains. Therefore, in domain generalization techniques, explicit samples of the test or target domain are not required (or may not be available) during training.

[0006] Most domain generalization work utilizes numerous training sets to learn domain-invariant feature extractors. Others focus on explicitly optimizing model parameters to achieve consistent performance across domains through ad hoc training strategies, while different lines of work require modifications to the model architecture to achieve domain invariance. However, none of these solutions fully utilize domain-specific training data because they explicitly attempt to discard any domain-specific information. Summary of the Invention

[0007] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0008] One example aspect of this disclosure relates to a computer-implemented method for performing domain generalization using batch normalized statistics. The method includes obtaining an ensemble model of machine learning by a computing system comprising one or more computing devices. This model includes a shared parameter portion and multiple distinct batch normalization layers correspondingly associated with multiple distinct source domains, wherein multiple distinct sets of batch normalized statistics are correspondingly associated with the multiple distinct source domains. The method includes accessing a target sample associated with a target domain. The method includes determining, by the computing system, a target set of batch normalized statistics for the target sample. The method includes determining, by the computing system, multiple similarity scores between the target set of batch normalized statistics and the multiple distinct sets of batch normalized statistics correspondingly associated with the multiple distinct source domains. The similarity scores may be distance metrics or other statistical measures of similarity. The method includes processing the target sample accordingly with the ensemble model of machine learning by the computing system to generate multiple domain-specific predictions correspondingly associated with the multiple distinct source domains. The method includes having a computational system interpolate multiple domain-specific predictions, at least in part, based on corresponding similarity scores between the target set in batch normalized statistics and multiple different sets of the batch normalized statistics, to obtain a target prediction for the target sample in the target domain. The method also includes having the computational system output the target prediction for the target sample.

[0009] Another exemplary aspect of this disclosure relates to a computational system for training an ensemble model to perform domain generalization. The computational system includes one or more processors and one or more non-transient computer-readable media, which jointly store: an ensemble model comprising one or more multi-source domain alignment layers, wherein each multi-source domain alignment layer includes a shared weight portion and a plurality of distinct batch normalization layers correspondingly associated with a plurality of source domains; and instructions, which, when executed by the one or more processors, cause the computational system to perform an operation for each of one or more training iterations. The operation includes obtaining a training batch comprising a plurality of domain-specific sets of training examples correspondingly associated with the plurality of source domains. The operation includes updating a plurality of distinct sets of batch normalization statistics for the plurality of distinct batch normalization layers correspondingly associated with the plurality of source domains. The operation includes, for each training example in a plurality of domain-specific sets of the training example: determining a training set of batch normalized statistics for the training examples; accordingly determining a plurality of similarity scores between the training set of batch normalized statistics and the plurality of different sets of batch normalized statistics correspondingly associated with the plurality of different source domains; and interpolating the plurality of domain-specific predictions, at least in part, based on the corresponding similarity scores between the training set of batch normalized statistics and the plurality of different sets of batch normalized statistics, to obtain a training prediction for the training example. The operation includes determining an aggregated loss based on the corresponding training predictions generated for each training example in the plurality of domain-specific sets of the training example. The operation includes updating one or more parameter values ​​for at least the shared weight portion of at least one of the one or more multi-source domain alignment layers.

[0010] Another example aspect of this disclosure relates to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations include obtaining an ensemble model of machine learning, the model including a shared parameter portion and a plurality of parallel batch normalization layers correspondingly associated with a plurality of different source domains. The operations include accessing a target sample associated with a target domain. The operations include processing the target sample accordingly with the ensemble model of machine learning to generate a plurality of domain-specific predictions correspondingly associated with the plurality of different source domains. The operations include correspondingly determining a target set of batch normalization statistics generated for the target sample and a plurality of different sets of batch normalization statistics correspondingly associated with the plurality of different batch normalization layers, which are correspondingly associated with the plurality of different source domains. The operations include interpolating the plurality of domain-specific predictions at least in part based on the corresponding distances between the target set of batch normalization statistics and the plurality of different sets of batch normalization statistics to obtain a target prediction for the target sample in the target domain. The operations include outputting the target prediction for the target sample.

[0011] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0013] A detailed discussion of embodiments for those skilled in the art is set forth in the description with reference to the accompanying drawings, in which:

[0014] Figure 1A -C depicts a graphical representation of the process of creating a domain-specific representation, projecting samples from an unknown domain, and locating samples relative to a known domain, according to an example embodiment of the present disclosure.

[0015] Figure 2A and 2B An example multi-source domain alignment layer according to an example embodiment of the present disclosure is depicted.

[0016] Figure 3A and 3B The present disclosure describes the application of the proposed methods to an example dataset for experimentation, according to an example embodiment of the present disclosure.

[0017] Figure 4 Pseudocode for an example training algorithm based on an example embodiment of this disclosure is provided.

[0018] Figure 5A A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0019] Figure 5B A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0020] Figure 5C A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0021] The reference numerals repeated across multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0022] Overview

[0023] Generally, this disclosure relates to systems and methods that utilize batch normalized statistics as a means of cross-domain generalization. Specifically, example implementations of this disclosure can generate different representations for different domains by collecting independent batch normalized statistics, which can then be used for mapping between domains in a shared latent space. During testing or inference, samples from unknown test or target domains can be projected into the same shared latent space. Thus, the domain of the target sample can be expressed as a linear combination of known domains, weighted based on corresponding similarity scores between batch normalized statistics in the latent space. This same mapping strategy can be applied during both training and testing to learn both the latent representation and a robust yet lightweight ensemble model operating within this latent space. The appendix contains example experiments demonstrating significant improvements (up to +12%) in classification accuracy compared to state-of-the-art techniques on popular domain generalization benchmarks: PACS, Office31, and Office-Caltech.

[0024] Therefore, in one example, the ensemble model of machine learning may include a shared parameter portion and multiple distinct batch normalization layers correspondingly associated with multiple different source domains. During training, multiple distinct sets of batch normalization statistics can be generated for the multiple different source domains. During inference, the computational system can determine the target set of batch normalization statistics for target samples associated with the target domain. Subsequently, the computational system can correspondingly determine multiple similarity scores between the target set of batch normalization statistics and the multiple distinct sets of batch normalization statistics correspondingly associated with the multiple different source domains. The computational system can process the target samples with the ensemble model of machine learning to generate multiple domain-specific predictions correspondingly associated with the multiple different source domains. The computational system can interpolate the multiple domain-specific predictions at least in part based on the corresponding similarity scores between the target set of batch normalization statistics and the multiple distinct sets of batch normalization statistics to obtain target predictions for target samples in the target domain.

[0025] More specifically, the example aspects of this disclosure explicitly facilitate domain-specific representations by collecting independent batch normalized statistics for each available domain during training. In some implementations, this results in lightweight ensembles for training domain-specific models, where most or all parameters are shared, except for the normalized statistics. Upon convergence, each domain can be mapped to a point in the latent space using the accumulated statistics.

[0026] To provide an example, Figure 1A A visualization of this space is shown for a simplified case of a single batch normalization layer that operates on the output of a convolutional layer using two filters (e.g., accumulating only two means and variances, and each domain can be represented as a 2D Gaussian; other embodiments accumulate more statistics, e.g., across multiple consecutive layers). In this space, the membership of a sample to a domain can be efficiently measured by simply looking at the distance between the instance normalized statistics of a sample and the statistics specific to each domain (e.g., the specific distance to their centroids corresponding to the accumulated population statistics). Therefore, normalized statistics can be used to efficiently learn the latent space of domains.

[0027] In particular, the same latent space representation can be used for samples from unknown domains, relying on their instance statistics to project them into the same latent space. Figure 1B The illustration visualizes this projection of samples from an unknown domain through its instance statistics. After projection, the distance between the sample and the domain centroid can be determined, effectively attempting to locate the unknown domain relative to a known domain. Figure 1C The process is outlined in the diagram, where arrows indicate the measured distance between the centroid of the known domain and the unknown domain.

[0028] Following projection, the proposed lightweight ensemble predictions for the test samples can be generated as a combination of domain-specific predictions. For example, the domain-specific predictions can be weighted according to the inverse of the distance from the known domain to the latent space. The same domain-specific model combination can be used when training on samples from the known domain. By doing so, the proposed training method forces the model to learn a meaningful latent space and Rogert probabilities that can be safely linearly combined according to the proposed weighting strategy.

[0029] Therefore, an exemplary aspect of this disclosure recognizes that batch normalized statistics (e.g., accumulated on convolutional layers) can be used to map input samples (e.g., input images) to a latent space in which membership to a domain can be measured based on the distance of the sample to the domain centroid. One effective use of this concept is to learn lightweight ensemble models that share some or all of the parameters, except for the normalized statistics. Such ensemble models can generalize better to unseen domains by interpolating various domain-specific predictions based on the distance between the instance norm statistics of the target samples and the domain centroid.

[0030] This disclosure provides several technical effects and benefits. As an example, compared to previous work, the proposed system and method do not discard domain-specific properties, but rather use them to learn the domain's latent space and map unknown domains relative to known domains. This results in significant improvements over existing techniques using different network architectures on standard domain generalization benchmarks. Therefore, the ability of computational systems to generalize to unseen domains can be enhanced.

[0031] The proposed technique can be applied to many different machine learning model architectures with batch normalized statistical features, including, as an example, any modern convolutional neural network (CNN) that relies on batch normalization layers. The proposed approach also elegantly extends to the number of domains available at training time.

[0032] As another example of technical effect and benefit, the domain generalization technique described in this paper eliminates the need to train or retrain a new model for each possible domain or to collect training samples from all possible domains. In particular, the proposed system and method can generate a single ensemble model robust to new or unseen domains. Therefore, there is no need to generate additional models or collect additional training data for these additional domains. In this way, computational resources that would otherwise be spent on model training or training data collection can be saved, thereby reducing the consumption of computational resources such as processor usage, memory usage, and / or network bandwidth.

[0033] Example domain generalization techniques

[0034] One example aspect of the proposed technique is the use of batch normalized statistics to map known and unknown domains in a shared latent space, where the domain membership of samples can be measured based on the distance between Gaussian distributions. The following sections introduce some common notation, describe example multi-source domain alignment layers that can be used to map domains in the latent space, and describe examples of how to project samples from unknown domains into the same latent space to achieve robust performance. Finally, the same predictions through the mapping strategy can be combined during training to improve model performance.

[0035] Example symbols

[0036] Let X and Y represent the input (e.g., image) and output (e.g., object category) spaces of the model. Let represent the set of K source domains available during training. This can be represented by an unknown probability distribution on the space X×Y. To describe each domain d i The goal of a machine learning model is to learn the probability distribution p of the entire training set. xy Let t be a general target domain available only at test time, and follow an unknown probability distribution in the same space.

[0037] Typically, deep learning models learn a mapping X→Y. The example implementations disclosed herein include lightweight ensembles of models that learn a mapping (X,D)→Y, which utilize domain labels to learn an ensemble of posterior distributions. The condition is domain membership. Since it's impossible to learn the target distribution during training... Therefore, one goal of the proposed method is to approximate it as the source distribution of the learning process. Mixtures (e.g., linear combinations).

[0038] For each source domain d∈D, give a value containing n d Training set of labeled samples test set From the unknown marginal distribution of the target domain t Collected m t It consists of 100 unlabeled samples. In contrast to the domain adaptation setting, the domain generalization approach used here assumes that samples from the target domain are not available during training. Furthermore, during inference, it is assumed that each unseen sample is processed independently, meaning that information from previously seen target samples is not accumulated and affects new predictions.

[0039] Example multi-source domain alignment layer

[0040] Neural networks are particularly prone to capturing dataset biases in their internal representations. The distribution of internal features is indeed highly domain-dependent. To capture and mitigate the distribution bias inherent in multi-source settings, the example implementation of this disclosure enables the batch normalization layer to normalize domain-related activations to the same reference distribution through domain-specific normalization statistics.

[0041] Therefore, its first and second moments can be used to... The activation of a domain d is nominally normalized by matching it to a reference Gaussian with zero mean and single variance:

[0042]

[0043] Where, x d It is the marginal distribution of activations from domain d. Activate the input extracted from it; and It is the overall statistic of the domain d, and ε>0 is a small constant used to avoid numerical problems.

[0044] During training, the multi-source batch normalization layer can collect and apply domain-specific batch statistics. At the same time, the overall domain statistics are updated accordingly to the moving average of the statistics for each batch b.

[0045] During inference, each test sample can be analyzed individually, and the domain label d may be unavailable. This boils down to the case where the batch size is equal to 1. The instance statistics of a single sample x can be compared with the statistics of the batch b from domain d. Comparisons are necessary. Since the overall statistic is merely a noisier estimate of the statistic of the same Gaussian distribution, the validity of this statement extends to comparisons with them.

[0046] For example, in the case of a 2D feature map with size H×W and batch size B, the analysis of batch statistics calculation is as follows:

[0047]

[0048]

[0049] Where, μ b and Correspondingly, these are the batch mean and variance, and x is the value of a single element of the feature map. If we consider any normally distributed variable... It is then clear that the instance statistics (case B=1) are estimates of the parameters of the same Gaussian, but calculated on a smaller number of samples H·W instead of B·H·W.

[0050] Because the distribution of internal features is highly domain-dependent, the accumulated statistics for each domain provide a compact representation of the corresponding domain. The next section explains how this layer can be used to map source domains and unseen samples within the same latent space.

[0051] Figure 2A and 2B An example multi-source domain alignment layer is illustrated. Specifically, as... Figure 2A As shown, this layer may include a shared weight portion to generate feature maps and multiple parallel batch normalization layers, each corresponding to a different source domain. During training, the batch normalization layers can collect and update domain-specific batch and overall statistics. During inference, as... Figure 2B As shown, to compute the final prediction, the same layer can collect instance statistics of the target samples, which can then be used to estimate their domain membership. In some implementations, although separate batch normalization statistics are maintained for each domain, the same batch normalization layer parameters gamma and beta can be learned and shared by all batch normalization layers across all domains.

[0052] Example domain localization in the Batchnorm latent space

[0053] Using the domain alignment layer proposed in the previous section to collect specific statistics allows the network to clearly learn multiple source distributions.

[0054] The result of this stopgap measure is a lightweight ensemble of models where each model shares some or all of the weights, but differs for the normalization parameters. In one example, all weights are shared, but differ for the normalization parameters at one or more layers. In another example, all weights in the shared feature extraction portion are shared, but differ for the normalization parameters at one or more layers, and the difference lies in that each source domain has a domain-specific prediction head.

[0055] Because this lightweight integration embodies multiple source distributions Therefore, this disclosure proposes to approximate the target distribution by performing optimal interpolation across these distributions. This reduces domain shift in the target domain. The generated target distribution is a weighted mixture of the distributions in the ensemble. In some implementations, the weights are chosen based on the distance of the test sample relative to each source domain in the latent space.

[0056] Therefore, the example implementation is based on its overall statistics. Map individual domains in the latent space, where, and It is a vector of the cumulative population mean and variance of the domain d for all layers l∈B={1,2,...,}. B is the set of batch normalized layers in the selected model architecture. The set of batch normalized layers included in set B can include all or fewer of the batch normalized layers included in the model. Set B can include any number of layers (e.g., 1, 2, 20, etc.).

[0057] Specifically, the potential space L l The model is spanned by activation statistics at layer l. In this space, individual samples x are mapped by their instance statistics at layer l, while the aggregate statistics accumulated for each domain at the same layer l are used to represent the domain centroids in this space. Intuitively, since activations in a neural network are highly domain-dependent, clusters of points in this latent space correspond to a specific domain, and the aggregate statistics accumulated for that specific domain provide a compact representation.

[0058] Therefore, in some implementations, the potential embedding of a domain d can be defined as:

[0059]

[0060] It is a vector of the cumulative total statistics of all fields d in all layers l∈B.

[0061] Similarly, for unseen target samples x t Its projection can be derived by forward propagation through the network and normalized using the statistics of its activated instances. Therefore, the target sample x t Potential embeddings It can be defined as a stacked vector of instance statistics at different batch normalization layers in the network:

[0062]

[0063] Potential embedding Each tuple represents the instance statistics collected at a certain layer l during forward propagation and can be used to map onto the latent space L at layer l. l Sample x in t .

[0064] Once the embeddings of the test samples are available, we can use this information to map onto the batch-normalized latent space L = {L l} l∈B The samples in the sample, where the target sample x can be... t The membership degree of a domain d is determined, for example, by the inverse of the distance metric between the target embedding and the domain embedding. This allows any test sample to be soft-domain classified into each source domain.

[0065] To compute the potential domain space L in layer ll The distance metric between two points treats the moving mean and moving variance of the corresponding batch normalization layer as parameters of a multivariate Gaussian distribution. This distance can be expressed as a distance in a probability measure space, i.e., a symmetric positive definite function satisfying the triangle inequality. An example distance function is the Wasserstein distance, a special case for two multivariate Gaussian distributions.

[0066] Let p ~ N(μ) p C p ) and q~N(μ q C q ) is R n The two normal distributions on the above have corresponding expected values ​​μ. p and μ q ∈R n and C p C q ∈R n×n The covariance matrix. Then the 2-Wasserstein distance is:

[0067]

[0068] Among them, ||·|| 2 It is R n The Euclidean norm on the surface.

[0069] The example implementation utilizes the Wasserstein metric to measure the value of the test sample x in the following manner. t Embedding of domain d to z d Distance between: summed over batch normalization layer l∈B at activation embedding and Distance between:

[0070]

[0071] Here, B is the set of batch normalization layers in the selected network architecture.

[0072] In some implementations, the test sample x t The membership degree of a domain d can be defined as the reciprocal of the distance to that domain:

[0073]

[0074] By examining Equations 2 and 3, it can be seen that the only difference between instance and batch statistics is the estimation of their sample size, and therefore it is fair to compare them by calculating the Wasserstein distance between the two multivariate Gaussian distributions they represent.

[0075] Once the membership degrees for all source domains have been calculated, they can be used to finally distribute the target. Restore the source distribution of learning to a weighted average of the corresponding domain membership degrees. Mixtures (e.g., linear combinations):

[0076]

[0077] in, It is the test sample x t The membership degree of the domain d.

[0078] Final prediction f(x) t It can be similarly computed as, for example, a linear combination of multiple predictions obtained under different domain assumptions:

[0079]

[0080] Where, f(x) t |d) is the model learned from the domain d for sample x. t The obtained predictions. In some implementations, the final prediction can be computed at the softmax layer of the ensemble model. In other implementations, the final prediction can be computed at the output layer of the ensemble model. In still other implementations, the final prediction can be computed individually at each layer of the ensemble model, and the final prediction can be passed from each layer to the next sequential layer.

[0081] As an example used for illustration, Figure 3A and 3B The illustration shows an example of applying this process to a PACS dataset. This dataset consists of four domains, three of which are assumed to be available at training time in the illustrated example. Therefore, each training batch consists of three domain batches, one for each source domain. Figure 3A The multi-source domain alignment layer described above is illustrated. During training, different statistics are updated and applied for each source domain. During inference, target samples are propagated through instance normalization to derive their latent embeddings. Figure 3B As shown, the collected population and instance statistics are used to map the domain and target samples to the same latent space. Finally, the domain membership of the samples can be estimated by calculating the Wasserstein distance between the domain and target centroids.

[0082] This elegant formula allows for optimal navigation within the latent space of batchnorm statistics. Specifically, if the test sample belongs to one of the source domains, the proposed method assigns a high membership value to the corresponding domain. On the other hand, if the test sample does not belong to any source domain, the corresponding target model is expressed as a combination (e.g., a linear combination) of source models cleverly contained within a lightweight ensemble.

[0083] Example Training Strategy

[0084] To better define the potential space of each batch normalization layer, the example implementation replicates the same process described in the preceding sections to compute predictions for samples from known domains during training. In an illustrative example, the training batch consists of K domain batches with the same number of samples. During each training step, (i) the domain batches are first propagated to update the corresponding domain population statistics. Then, (ii) all samples are forward-propagated without assuming hard domain membership to collect their instance statistics, similar to how target samples are interpreted during inference. Finally, (iii) each sample is propagated under K multi-domain assumptions, and the resulting domain-specific predictions are weighted according to Equation 12. Applying this process during training can encourage the creation of well-defined batch-normalized latent spaces.

[0085] In some implementations, because the model is initialized with certain weights (e.g., pre-trained on ImageNet), each domain-specific batch normalization branch needs to be specialized before starting this training process; otherwise, convergence problems may occur. Therefore, in some implementations, domain-specific batch normalization statistics can be pre-computed during a warm-up period, where the model is trained on the entire dataset according to the standard training process, except that the domain batches are propagated through the corresponding batch normalization branches (e.g., to accumulate domain-specific batch normalization statistics).

[0086] Example devices and systems

[0087] Figure 5A A block diagram of an example computing system 100 according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0088] User computing device 102 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0089] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0090] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may include various machine learning models, such as neural networks (e.g., deep neural networks); or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.

[0091] In some implementations, one or more machine learning models 120 may be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120.

[0092] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by server computing system 130, which communicates with user computing device 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 140 as part of a web service (e.g., a domain generalization service). Thus, one or more models 120 may be stored and implemented at user computing device 102, and / or one or more models 140 may be stored and implemented at server computing system 130.

[0093] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices through which a user can provide user input.

[0094] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 that are executed by processor 132 to cause server computing system 130 to perform operations.

[0095] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. When the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0096] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0097] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 through interaction with training computing system 150, which is communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0098] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0099] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques (e.g., error backpropagation) to train machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., gradients based on the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters across multiple training iterations.

[0100] In some implementations, performing error backpropagation may include performing time-truncated backpropagation. The model trainer 160 may perform various generalization techniques (e.g., weight decay, loss of weight, etc.) to improve the generalization ability of the trained model.

[0101] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, data assigned to different source domains. Source domains may include different types of data, different data sources, different data structures, data associated with different entities, data associated with different conditions, etc.

[0102] In some implementations, training examples can be provided by the user computing device 102 if the user has consented. Therefore, in such an implementation, the training computing system 150 can train the model 120 provided to the user computing device 102 based on user-specific data received from the user computing device 102. In some cases, this process can be referred to as a personalized model.

[0103] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or optical or magnetic media.

[0104] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication on Network 180 can be carried over any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0105] Figure 5A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, model 120 can be trained and used locally on user computing device 102. In some such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0106] Figure 5B A block diagram of an example computing device 10 according to an exemplary embodiment of the present disclosure is depicted. The computing device 10 may be a user computing device or a server computing device.

[0107] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning models. For example, each application can contain machine learning models. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0108] like Figure 5B As shown, each application can communicate with multiple other components of the computing device, such as one or more sensors, a field manager, a device status component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In other implementations, the API used by each application is application-specific.

[0109] Figure 5C A block diagram of an example computing device 50 according to an exemplary embodiment of the present disclosure is depicted. The computing device 50 may be a user computing device or a server computing device.

[0110] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0111] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 5C As shown, a corresponding machine learning model (e.g., a model) can be provided for each application, and this model is managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of computing device 50.

[0112] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized storage repository for data from computing device 50. For example... Figure 5C As shown, the central device data layer can communicate with multiple other components of the computing device, such as one or more sensors, a field manager, a device status component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0113] Additional Publication

[0114] The technologies discussed in this article refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitions of tasks and functions among components. For example, the processes discussed here can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can run sequentially or in parallel.

[0115] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation rather than limitation of the disclosure. Changes, variations, and equivalents of these embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter that will be obvious to those of ordinary skill in the art. For example, features shown or described as part of one embodiment may be used in another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computer-implemented method for performing domain generalization via batch normalization statistics, the method comprising: An ensemble model of machine learning is obtained by a computing system including one or more computing devices. The ensemble model of machine learning includes a shared parameter part and multiple different batch normalization layers correspondingly associated with multiple different source domains including multiple image domains, wherein multiple different sets of batch normalization statistics are correspondingly associated with the multiple different source domains. Access the target image associated with the target image domain; The calculation system determines the target set of batch normalized statistics for the target image; The computing system accordingly determines multiple similarity scores between the target set in batch normalized statistics and the multiple different sets in batch normalized statistics correspondingly associated with the multiple different source domains; The computing system uses the ensemble model of the machine learning to process the target image accordingly, so as to generate multiple domain-specific predictions that are correspondingly associated with the multiple different source domains; The computational system interpolates the plurality of domain-specific predictions, at least in part, based on corresponding similarity scores between the target set and the plurality of different sets in batch-normalized statistics, to obtain target predictions for the target image in the target image domain, including: The computing system accordingly determines multiple membership values ​​for the plurality of different source domains, wherein the membership value for each source domain is inversely proportional to the corresponding distance between the target set in the batch normalized statistics and the set of batch normalized statistics associated with such source domains; and The calculation system determines a weighted average of the multiple domain-specific predictions weighted according to the multiple membership values, and the target prediction for the target image in the target image domain is the weighted average of the multiple domain-specific predictions; and The computing system outputs the target prediction for the target image.

2. The computer-implemented method according to claim 1, wherein, The ensemble model of the machine learning includes one or more multi-source domain alignment layers, wherein each multi-source domain alignment layer includes a shared weight portion and two or more of the parallel batch normalization layers correspondingly associated with the plurality of different source domains.

3. The computer-implemented method according to claim 1, wherein, The target image is unavailable during the training of the ensemble model of the machine learning.

4. The computer-implemented method according to claim 1, wherein, The multiple different sets of batch normalized statistics and the target set of batch normalized statistics each include corresponding values ​​of mean statistics and variance statistics.

5. The computer-implemented method according to claim 1, wherein, Accordingly, the multiple similarity scores between the target set in the batch normalized statistics and the multiple different sets in the batch normalized statistics correspondingly associated with the multiple different source domains include multiple Wasserstein distances between the corresponding multivariate Gaussian distributions.

6. The computer-implemented method according to claim 1, wherein, The ensemble model for machine learning includes convolutional neural networks.

7. The computer-implemented method according to claim 1, wherein, The ensemble model of the machine learning includes a shared parameter portion, which is configured to perform feature extraction on all and multiple different prediction heads in the plurality of different source domains, and the plurality of different prediction heads are correspondingly configured to perform prediction on the plurality of different source domains individually.

8. The computer-implemented method according to any one of claims 1 to 7, wherein: The computational system processes the target image accordingly using the ensemble model of the machine learning to generate multiple domain-specific predictions correspondingly associated with the multiple different source domains, including: generating multiple softmax layer outputs correspondingly associated with the multiple different source domains; and The interpolation of the plurality of domain-specific predictions by the computing system includes: The computing system interpolates the outputs of the plurality of softmax layers to generate an aggregated softmax output; and The target prediction of the target image is generated by the computing system based on the aggregated softmax output.

9. A computer-readable storage medium comprising computer-executable code, which, when executed by one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 8.

10. A computer system, comprising: One or more processors; as well as The computer-readable storage medium according to claim 9.

11. A computational system for training an ensemble model to perform domain generalization, the computational system comprising: One or more processors; as well as One or more non-transitory computer-readable media, which collectively store: An ensemble model comprising one or more multi-source domain alignment layers, wherein each multi-source domain alignment layer includes a shared weight portion and multiple distinct batch normalization layers correspondingly associated with multiple distinct source domains comprising multiple image domains; and Instructions, which, when executed by the one or more processors, cause the computing system to perform operations, the operations including: for each of the one or more training iterations, Obtain a training batch, which comprises a plurality of domain-specific sets of training examples correspondingly associated with the plurality of different source domains; Update multiple different sets of batch normalization statistics for the multiple different batch normalization layers that are correspondingly associated with the multiple different source domains; For each training example in the plurality of domain-specific sets of training examples: Determine the training set for the batch normalized statistics of the training examples; Accordingly, multiple similarity scores are determined between the training set in batch normalized statistics and the multiple different sets in batch normalized statistics correspondingly associated with the multiple different source domains; and Interpolating multiple domain-specific predictions to obtain training predictions for the training examples, at least in part based on corresponding similarity scores between the training set in batch normalized statistics and the multiple different sets in batch normalized statistics, includes: For the plurality of different source domains, a plurality of membership values ​​are determined accordingly, wherein the membership value of each source domain is inversely proportional to the corresponding distance between the training set in the batch normalized statistics and the set of batch normalized statistics associated with such source domains; and Determine a weighted average of the plurality of domain-specific predictions weighted by the plurality of membership values, wherein the training prediction for the training example is the weighted average of the plurality of domain-specific predictions; The aggregation loss is determined based on the corresponding training predictions generated for each training example in the plurality of domain-specific sets of training examples; and Update one or more parameter values ​​of at least the shared weight portion of at least one of the one or more multi-source domain alignment layers.

12. The computing system according to claim 11, wherein, The operation further includes: prior to the one or more training iterations. A warm-up period is performed, during which the ensemble model is trained on the entire training dataset, wherein gradients from domain-specific batches are propagated through a corresponding batch normalization layer among the plurality of different batch normalization layers associated with the corresponding source domain.

13. The computing system according to claim 11, wherein, The multiple different sets of batch normalized statistics and the training set of batch normalized statistics each include corresponding values ​​of mean statistics and variance statistics.

14. The computing system according to claim 11, wherein, Accordingly, the multiple similarity scores between the training set in batch normalized statistics and the multiple different sets in batch normalized statistics correspondingly associated with the multiple different source domains include multiple Wasserstein distances between the corresponding multivariate Gaussian distributions.

15. The computing system according to any one of claims 11 to 14, wherein, The operation further includes: after the one or more training iterations. The ensemble model is deployed to perform domain generalization to generate target predictions for target images associated with unseen target domains.

Images