Few-Shot Domain Adaptation in Generative Adversarial Networks

Abstract

The present disclosure provides an improved method for learning a generative model with limited training data by adapting a pre-trained GAN model from a related domain and given a set of target examples from the new or target domain to the new domain.

Description

Technical Field

[0001] The present disclosure generally relates to domain adaptation. More specifically, the present disclosure relates to few-shot domain adaptation in generative adversarial networks. Background Art

[0002] Image synthesis is the task of generating novel images by learning the distribution of a dataset. Besides pure visual synthesis applications, it is useful as a data augmentation tool to improve the performance of other models on rare or hard-to-collect data.

[0003] Generative adversarial models (GANs) have increasingly impressive performance in image synthesis tasks. However, these models are sample inefficient and typically require thousands or millions of images to produce high-quality outputs. Thus, in low-data regimes, GANs suffer from instability and overfitting. Additionally, the generated samples are typically concentrated on the most common patterns in the data, making it challenging to generate images of rare classes. Although importance sampling can be used to sample from rare patterns, the diversity of these images will be severely limited.

[0004] Due to the high cost and difficulty of collecting large datasets, there is a need for models that can synthesize diverse images using only limited training data. However, GAN-based adaptation methods still require target training data in the range of 1k to 10k samples, which may be limited in many practical settings.

[0005] Some prior arts are able to adapt to a smaller number of training data. However, these techniques use undesirable base models, such as GLO (Optimizing the latent space of generative networks by Bojanowski, P., Joulin, A., Lopez Paz, D., Szlam, A., published on arXiv preprint arXiv:1707.05776 in 2017), which may result in blurry samples due to the use of pixel loss. Similarly, models based on invertible flows have been shown to be applicable to new domains with limited samples (see Transflow learning: repurposing flow models without retraining by Gambardella, A., Baydin, A.G., Torr, P.H.S., published on arXiv in 2019). However, invertible flow models require computationally and memory-intensive architectures, whose latent space is the same as the data dimension. SUMMARY OF THE INVENTION

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] One example aspect of the present disclosure relates to a computer-implemented method for performing domain adaptation of a generative model. The method includes: obtaining, by a computing system including one or more computing devices, a pre-trained generative adversarial network that has been trained on a source domain training dataset to generate an output in the source domain, where the pre-trained generative adversarial network includes a generator model having a first plurality of pre-trained parameters and a discriminator model having a second plurality of pre-trained parameters. The method includes: modifying, by the computing system, the pre-trained generative adversarial network to obtain a modified generative adversarial network. Modifying the pre-trained generative adversarial network by the computing system includes one or both of: adding, by the computing system, one or more first additional parameters to the generator model; and adding, by the computing system, one or more second additional parameters to the discriminator model. The method includes: accessing, by the computing system, a target domain training dataset associated with a target domain different from the source domain. The method includes: training, by the computing system, the modified generative adversarial network on the target domain training dataset to generate an output in the target domain, where training the modified generative adversarial network by the computing system includes: modifying, by the computing system, at least one of the one or more first additional parameters or the one or more second additional parameters. The method includes: outputting, by the computing system, the modified generative adversarial network as a trained model.

[0008] Other aspects of the present disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of the various embodiments of the present disclosure will become better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A detailed discussion of embodiments involving those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0011] Figures 1A to 1B An example use of a generative adversarial model that has undergone domain adaptation in accordance with an example embodiment of the present disclosure is depicted.

[0012] Figures 2A to 2B A graphical diagram of a pre-trained generative adversarial network in accordance with an example embodiment of the present disclosure is depicted.

[0013] Figures 2C to 2DDepicts a graphical diagram of a domain - adapted generative adversarial network according to an example embodiment of the present disclosure.

[0014] Figures 3A to 3D Depicts a graphical diagram of an example adaptation block according to an example embodiment of the present disclosure.

[0015] Figures 4A to 4B Depicts a graphical diagram of the domain adaptation process of a generative adversarial network according to an example embodiment of the present disclosure.

[0016] Figures 5A to 5B Shows example experimental results according to an example embodiment of the present disclosure.

[0017] Figure 6A Depicts a block diagram of an example computing system according to an example embodiment of the present disclosure.

[0018] Figure 6B Depicts a block diagram of an example computing device according to an example embodiment of the present disclosure.

[0019] Figure 6C Depicts a block diagram of an example computing device according to an example embodiment of the present disclosure.

[0020] Figure 7 Depicts a flowchart of an example method for performing domain adaptation on a GAN according to an example embodiment of the present disclosure.

[0021] Reference numerals repeated across multiple figures are intended to identify the same features in various embodiments. Detailed Description

[0022] Overview

[0023] The present disclosure presents improved methods for learning generative models with limited training data. Specifically, example embodiments of the present disclosure utilize a pre - trained GAN model from a related domain and adapt it to a new domain given a set of target examples from the new or target domain. As an example, additional parameters can be added to the pre - trained GAN model, and the model can be retrained on the set of target examples from the target domain. Thus, to alleviate data requirements, the present disclosure provides systems and methods for generating new concept images in a few - shot setting (some of which may be referred to as “FewShotGAN”).

[0024] As an example, Figures 1A to 1BDepicts an example use of a domain - adapted generative adversarial model according to an example embodiment of the present disclosure. Specifically, a pre - trained GAN that has been trained on a source - domain training dataset to generate outputs in the source domain can be adapted (e.g., by adding one or more additional parameters to the GAN) and retrained on a target - domain training dataset associated with a target domain that is different from the source domain. After retraining, the modified GAN can generate outputs in the target domain. Thus, some example implementations of the proposed system and method utilize a pre - trained base model on a source domain with rich images and evolve the base model with a residual adapter to generate images in the target domain.

[0025] Compared to earlier GAN - based methods that require a higher order of magnitude of training samples, aspects of the present disclosure enable domain transfer (e.g., what can be referred to as “few - shot” learning) in a more restricted setting of 1 to 25 training images. Specifically, training methods are proposed to prevent overfitting to a small target training set, by leveraging the distribution characteristics induced by the pre - trained generator to produce a model that can generate semantically diverse images in the target domain.

[0026] Additionally, the proposed method also allows controlling the degree of transfer and interpolation between domains. For example, the proposed perceptual path gradient sparsity metric can be used to explicitly measure the smoothness of latent - space interpolation (and thus be well - correlated with the overfitting level). By monitoring the PPGS metric (or a similar metric), model retraining can be stopped when the desired degree of transfer or interpolation between domains is achieved.

[0027] In some embodiments, the proposed method uses a GAN model pre - trained on a source domain that is a related domain or the nearest available domain relative to the target domain. For example, if the goal is to synthesize facial images of under - represented attributes (e.g., glasses - wearing faces, occluded faces), a GAN pre - trained on publicly available facial images (e.g., not necessarily having any large number of images with these under - represented attributes) can be used as an initial pre - trained model. Next, additional parameters can be added to this pre - trained model. These additional parameters can be trained using a limited number of training samples from the new domain (e.g., faces with under - represented attributes). For example, the additional parameters can be trained while keeping the original parameters frozen at their pre - trained values.

[0028] An example of this process is depicted in Figures 2A to 2D where these additional parameters are denoted as α for the generator and β for the discriminator. Specifically, Figure 2A and 2B show the architecture of the pre - trained model: G and D represent the generator and discriminator respectively. Figure 2C and 2DDepicts an example adaptation of a pre-trained model via newly introduced parameters α and β. The blocks with shading depict trainable parameters, and the blocks without shading indicate frozen parameters.

[0029] Figure 2B And 2D Also show example functional forms of the layers in the generators and discriminators of the pre-trained network and the adapted network, respectively. In Figure 2D The illustrated example, additional filters are added to each layer, and the output of each layer is added to the output of the original convolutional filter before passing the output to the non-linearity.

[0030] According to another aspect of the present disclosure, some example embodiments also regularize newly added parameters (such as parameters α and β) by penalizing their norms (e.g., during retraining). Doing so prevents the new parameters from overfitting to the limited training data from the new domain, and the generated mapping and discriminative mapping do not move too far from their pre-trained counterparts. In some embodiments, the regularization penalty for the adaptation parameters can be tuned or adjusted to balance overfitting (small penalty) with mode collapse (large penalty).

[0031] Various norms can be used for regularization, including L1, L2, and / or group norms. Group norms can be used to simultaneously zero out groups of parameters. For example, the parameters corresponding to each layer can be grouped together to form a group corresponding to the layer. From the perspective of interpretability, this also helps to identify which layers contribute to the adaptation. Some example embodiments also use low-rank approximations of the adapter parameters (such as parameters α and β) to further reduce the number of newly added parameters.

[0032] Additional aspects of the present disclosure identify that common metrics in cross-domain image synthesis encourage overfitting, and propose a new metric and evaluation method to balance quality and diversity. For example, model selection criteria are provided that balance the quality and diversity of images. As an example, diversity can be measured using the Perceptual Path Gradient Sparsity (PPGS) metric, and quality can be measured using the Frechet Inception Distance (FID). In some embodiments, the PPGS metric can be added to the loss and directly optimized to encourage diversity. In yet other embodiments, diversity can be encouraged via other techniques, such as regularizing the generator to produce different outputs depending on the latent code, as described in Yang et al.'s Diversity-sensitive Conditional Generative Adversarial Networks on arXiv:1901.09024 in 2019.

[0033] Some example embodiments also apply the concept from learning, without losing sight of the degree of constraint transfer. This enables image generation towards unseen domains while retaining diversity. Thus, the proposed technique allows for retaining different patterns of the source dataset while fitting the characteristics of the target dataset.

[0034] Example embodiments of the present disclosure achieve high-quality diverse image synthesis in the few-shot setting. Specifically, example embodiments of the present disclosure have been verified when transferred to the Char75K, LSUN, and Animeface datasets, showing transfers from English characters → Kannada characters, churches → towers, cats → dogs, and FFHQ → anime. The sample quality is measured using the FID score and diversity using Learned Perceptual Image Patch Similarity (LPISP), and it is observed that the proposed method has a significant improvement over the baseline in these aspects. The proposed method is the first to demonstrate GAN domain transfer in the few-shot setting.

[0035] The systems and methods of the present disclosure can be used in many different applications or use cases. As an example, domain adaptation can be performed to personalize a GAN. For example, a GAN trained to generate general handwritings, facial images, and / or other user-agnostic outputs can be retrained on a small user-specific dataset to generate the same type of personalized or user-specific outputs (e.g., user-specific handwritings, facial images, etc. that match the user).

[0036] Another example application is to generate new synthetic data for underrepresented attributes. For example, a GAN trained to generate general outputs can be retrained on a small attribute-specific dataset to produce personalized outputs exhibiting specific attributes. As an example, this process can be used to extend facial attributes to underrepresented attributes, such as generating more facial images with attributes that are underrepresented in the original larger training set (e.g., red hair color). This example usage also has an impact on fairness or addressing biases in the training set. For example, if a particular attribute is underrepresented in the training set, this method can be used to generate more synthetic images for that underrepresented attribute, thereby reducing the bias in the training set. This method can also be used to create a balanced or fair evaluation set to provide a measure that well covers the desired distribution.

[0037] Other example use cases include modifying a GAN to perform style transfer, cross-modal generation, changing a face to anime or the like.

[0038] The systems and methods of the present disclosure provide several technical effects and benefits. As an example technical effect, the proposed domain adaptation technique reduces the number of training epochs required to obtain a GAN for a desired target domain. Specifically, instead of training a new GAN completely from scratch, a GAN pre-trained on a related domain can be reused or adapted to provide an output in the desired target domain, which requires significantly fewer overall training epochs compared to training a new GAN from scratch. In this way, the computational resources that would be spent on model training or training data collection can be saved, thereby reducing the consumption of computational resources such as processor usage, memory usage, and / or network bandwidth.

[0039] Similarly, the proposed domain adaptation technique allows a GAN to be learned in a few-shot setting, such as when only a relatively small number of training examples are available for the target domain. Previous methods that require training a new GAN from scratch would not provide any meaningful model capabilities in this setting. However, by enabling adaptation from a related domain, the present technique enables a high-performance model to be learned in this setting, which represents improved performance and functionality of a computational system in a few-shot setting.

[0040] Accordingly, the present disclosure presents a training method, architecture, and evaluation metric for few-shot domain adaptation in a GAN setting. We demonstrate the problems of existing metrics for the GAN domain transfer setting and show improved performance in various transfer settings using our early stopping evaluation metric.

[0041] Example techniques for domain adaptation

[0042] This section first describes the construction of the baseline and then details the architecture and training method for few-shot image synthesis.

[0043] Example baseline

[0044] Transfer GAN: An example baseline is the fine-tuning method used in the StyleGAN2 architecture in the following paper: Transferring gans: generating images from limited data by Wang et al. published at ECCV in 2018. Note that the authors did not validate the results in a few-shot setting. Example experiments found that repeating their training procedure led to overfitting on few-shot datasets.

[0045] Scaling and shifting: Another example baseline is the scaling and shifting method used in the StyleGAN2 architecture in the following paper: Image generation from small datasets via batch statistics adaptations by Noguchi et al. published in ICCV in 2019. This paper reports low-quality results for scaled and shifted GANs. Example experiments also found that even with refined training methods, this method performs poorly compared to other models.

[0046] Figures 3A to 3D An example adaptation block is shown, which represents multiple different example options for the convolutional layer design for adapting to a new domain. Figure 3A : To learn a generative model on a new domain, the model can be trained from scratch with randomly initialized convolutional layer weights. Figure 3B : The transfer GAN fine-tunes a pre-trained model using the available samples in the target domain. Figure 3C : Scaling and shifting adapts the batch statistics by scaling and shifting the feature channels while freezing the weights of the pre-trained model. Figure 3D : The residual adapter adds one or more parallel convolutions to one or more layers of the network, such as the 1x1 convolutions shown. Some example embodiments of the present disclosure use Figure 3D the residual adapter shown to operate.

[0047] Example FewShotGAN

[0048] Example aspects of the present disclosure provide systems and methods for few-shot domain transfer in a GAN setting, which have not been addressed in prior work on GAN domain transfer. In some embodiments, the capacity of the network can be limited by freezing the pre-trained weights and training a limited number of adaptive domain-specific weights. Additionally, some example embodiments use early stopping based on a trade-off between diversity and quality metrics to preserve the diversity of the pre-trained network. The example pre-trained model provides a prior representing a guess about the dataset it wishes to transfer to. For this reason, selecting a high-quality pre-trained model whose distribution is relatively similar to the target dataset can improve the performance of domain transfer.

[0049] Figure 4A and 4B provides an overview of an example adaptation process. Figure 4APre-training is shown, where the generator G and discriminator D are pre-trained on a dataset where a large number of examples are available (e.g., LSUN churches in the illustrated examples). Some embodiments follow the training process in StyleGAN (A style-based generator architecture for generative adversarial networks by Karras et al. in CVPR 2019), and update the weights in the blocks shown in shadow. Figure 4B The adaptation phase is shown. In some examples, by obtaining a pre-trained model, freezing its model weights, and inserting additional parameters - such as parallel (learnable) convolutional layers for one or more convolutional layers in the model, the pre-trained model is adapted to a new domain. In some embodiments, 1×1 convolutional layers are inserted for some or all convolutional layers. These additional new convolutional layers can then be adapted using a few target examples (e.g., LSUN towers in the illustrated examples). Some example embodiments also use a new metric to determine an optimal stopping criterion for preventing mode collapse.

[0050] Example techniques for restricting network capacity.

[0051] Some example embodiments of the present disclosure limit the capacity of the network. For example, some example embodiments of the present disclosure use a residual adapter, which has additional parameters and can perform instance-specific spatially-varying transformations. Example experiments show that this increased expressive power results in higher-quality images compared to scaling and shifting baselines. It has also been found that, compared to the transfer GAN baseline, the limited capacity of the adaptive parameters prevents overfitting in the few-shot regime.

[0052] Example early stopping

[0053] Some example embodiments of the present disclosure use an early stopping method based on a trade-off between diversity and quality. For the evaluation of image quality, FID can be used, which measures the distribution distance between the generated images and the real images (see Ganstrained by a two time-scale update rule converge to a local nash equilibrium by Martin Heusel, Hubert Ramsauer, T.U.B.N.S.H. in NeurIPS 2017).

[0054] However, FID cannot adequately penalize overfitting. Although the optimal FID score appears at t > 100 (seeFigure 5A ) However, interpolation in the latent space exhibits a discontinuous mode shift. Comparison with the ground truth verifies that the model closely corresponds to the training data.

[0055] In view of the above, the present disclosure proposes Perceptual Path Gradient Sparsity (PPGS), a new metric for measuring overfitting by quantifying these discontinuous mode shifts. In the case of overfitting, the gradient of the perceptual distance along a continuous path in the latent space will tend towards two modes: close to zero or very large. To quantify this phenomenon, the Gini index can be used, which measures the statistical dispersion or sparsity within a distribution. Hurley, N., Rickard, S., Comparing measures of sparsity, IEEE Transactions on Information Theory 55(10), 4723{4741, 2009. To compute PPGS, a number of perceptual path gradients can be randomly sampled:

[0056]

[0057] where \(Z_1, Z_2 \sim \mathcal{N}(0, 1)\) are random vectors input to the generator, \(P\) is the perceptual distance model from Zhang et al., The unreasonable effectiveness of deep features as a perceptual metric, CVPR 2018, and \(\epsilon\) is a scalar distance in the latent space. PPGS can be computed by measuring the sparsity of the perceptual path gradients using the Gini index:

[0058]

[0059] For non - negative mean \(E[S]\), PPGS is bounded in \([0, 1]\), and for negative \(E[S]\), PPGS can be larger. Larger values are sparser, indicating overfitting. This metric can be used to determine the stopping point of training.

[0060] To provide an illustrative example, Figure 5A and 5B shows overfitting measured by Perceptual Path Gradient Sparsity (PPGS). Figure 5A shows FID and PPGS over time. Figure 5B shows FID / PPGS at selected times. Figure 5A and 5BIt is shown that the commonly used FID score cannot fully reflect the degree of overfitting. The FID score continuously decreases during the training time steps. However, through latent space interpolation, sudden transitions can be observed in the generated images using smoothly interpolated latent features. This indicates that the trained model suffers from mode collapse (failure to capture the space of the sample distribution). On the other hand, the proposed perceptual path gradient sparsity metric explicitly measures the smoothness of latent space interpolation (and thus correlates well with the overfitting level). The lower the values of both metrics, the better.

[0061] Example ideas from learning without forgetting

[0062] Some example embodiments of the present disclosure adapt the concept of learning without forgetting to the few-shot image synthesis setting. For example, a residual adapter module can be used in the GAN setting, such as as Figure 3D illustrated. The residual adapter module calculates the residual bias at each layer of the network, which is added to the frozen pre-trained weights. Applying this idea to the few-shot domain transfer setting can operate to limit overfitting to the target set.

[0063] Example implementation details

[0064] The StyleGAN2 architecture and the corresponding pre-trained checkpoint can be used as a basis. The residual adapter can be implemented as a residual 1x1 convolution in parallel with each existing convolution in the network. The scale and shift of the GAN baseline can be implemented as trainable scale and shift operations after each convolution.

[0065] Hyperparameter selection. The number of training iterations can be selected based on PPSG. If there is a sudden increase in PPSG, it indicates overfitting, and the training can be stopped.

[0066] Training. An s-sample subset of a larger dataset can be created for training. For example, mirror augmentation can be used for the tower, dog, and anime face datasets. The learning rate can be decreased to prevent instability and overfitting, and the maximum number of training images seen can be reduced to 500K. Other training details can be matched with StyleGAN2.

[0067] Testing. Quality can be evaluated using FID, and diversity can be evaluated using the PPSG metric. FID can be measured relative to the large dataset X n from which the few-shot training data X s is sampled.

[0068] Example devices and systems

[0069] Figure 6AFIG. 0 depicts a block diagram of an example computing system 100 in accordance with an example embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0070] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop computer or a desktop computer), a mobile computing device (e.g., a smartphone or a tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0071] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash devices, disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0072] In some embodiments, the user computing device 102 can store or include one or more machine learning models 120. For example, the machine learning model 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including non-linear models and / or linear models. The neural network can include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0073] In some embodiments, one or more machine learning models 120 can be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some embodiments, the user computing device 102 can implement multiple parallel instances of a single machine learning model 120.

[0074] Additionally or alternatively, one or more machine learning models 140 may be included or otherwise stored in and implemented by server computing system 130, which communicates with user computing device 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 140 as part of a web service. Accordingly, one or more models 120 may be stored and implemented at user computing device 102 and / or one or more models 140 may be stored and implemented at server computing system 130.

[0075] User computing device 102 may also include one or more user input components 122 that receive user input. For example, user input component 122 may be a touch-sensitive component (such as a touch-sensitive display screen or a touchpad) that is sensitive to a user input object (such as a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other components by which a user may provide user input.

[0076] Server computing system 130 includes one or more processors 132 and a memory 134. One or more processors 132 may be any suitable processing device (such as a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be a single processor or multiple processors operatively connected. Memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash devices, magnetic disks, etc. and combinations thereof. Memory 134 may store data 136 and instructions 138 that are executed by processor 132 to cause server computing system 130 to perform operations.

[0077] In some embodiments, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0078] As described above, server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0079] The user computing device 102 and / or the server computing system 130 may train the models 120 and / or 140 via interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 and may be part of the server computing system 130.

[0080] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash devices, disks, etc. and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0081] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, backpropagation of error. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.

[0082] In some embodiments, performing backpropagation of error may include performing truncated backpropagation through time. The model trainer 160 may perform a number of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0083] Specifically, the model trainer 160 may train the machine learning models 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, samples from a source domain and a target domain, respectively.

[0084] In some embodiments, if the user has provided consent, training examples may be provided by the user computing device 102. Thus, in such an embodiment, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0085] The model trainer 160 includes computer logic for providing desired functionality. The model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard drive or optical or magnetic media.

[0086] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communication over the network 180 may be performed via any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings, or formats (e.g., HTML, XML) and / or security schemes (e.g., VPN, secure HTTP, SSL).

[0087] Figure 6A An example computing system that may be used to implement the present disclosure is illustrated. Other computing systems may also be used. For example, in some embodiments, the user computing device 102 may include the model trainer 160 and the training data 162. In such an embodiment, the model 120 may be trained and used locally at the user computing device 102. In some embodiments of such an embodiment, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0088] Figure 6B A block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure is depicted. The computing device 10 may be a user computing device or a server computing device.

[0089] The computing device 10 includes a number of applications (e.g., Application 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like.

[0090] As Figure 6B Illustrated, each application can communicate with several other components of a computing device, such as, for example, one or more sensors, a context manager, a device status component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a common API). In some embodiments, the API used by each application is specific to that application.

[0091] Figure 6C A block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure is depicted. The computing device 50 can be a user computing device or a server computing device.

[0092] The computing device 50 includes several applications (e.g., Application 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some embodiments, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0093] The central intelligence layer includes several machine learning models. For example, as Figure 6C Illustrated, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 50.

[0094] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As Figure 6C Illustrated, the central device data can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device status component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0095] Example methods

[0096] Figure 7 A flowchart of an example method 700 for performing domain adaptation on a GAN in accordance with an example embodiment of the present disclosure is depicted. Although for purposes of illustration and discussion, Figure 7depicts steps performed in a particular order, but the methods of the present disclosure are not limited to the order or arrangement specifically illustrated. Without departing from the scope of the present disclosure, the various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways.

[0097] At 702, the method may include: obtaining, by a computing system including one or more computing devices, a pre-trained generative adversarial network that has been trained on a source domain training dataset to generate an output in the source domain. The pre-trained generative adversarial network may include a generator model having a first plurality of pre-trained parameters and a discriminator model having a second plurality of pre-trained parameters.

[0098] At 704, the method may include: modifying, by the computing system, the pre-trained generative adversarial network to obtain a modified generative adversarial network. Modifying the pre-trained generative adversarial network by the computing system may include one or both of: adding, by the computing system, one or more first additional parameters to the generator model; and adding, by the computing system, one or more second additional parameters to the discriminator model.

[0099] In some embodiments, modifying the pre-trained generative adversarial network by the computing system may include both: adding, by the computing system, one or more first additional parameters to the generator model; and adding, by the computing system, one or more second additional parameters to the discriminator model.

[0100] In some embodiments, adding one or more first additional parameters to the generator model by the computing system may include: adding, by the computing system, one or more parallel residual layers to a first convolutional neural network of the generator model. In some embodiments, adding one or more second additional parameters to the discriminator model by the computing system may include: adding, by the computing system, one or more parallel residual layers to a second convolutional neural network of the discriminator model.

[0101] At 706, the method may include: accessing, by the computing system, a target domain training dataset associated with a target domain that is different from the source domain.

[0102] In some embodiments, the target domain training dataset includes 25 or fewer target training examples of the target domain.

[0103] In some embodiments, the source domain includes a first image domain, and the target domain includes a second image domain that is different from the first image domain.

[0104] In some embodiments, the source domain includes a general domain having general population samples, and the target domain includes a personalized domain having user-specific samples.

[0105] In some embodiments, the source domain includes a general domain having general population samples, and the target domain includes an underrepresented domain having samples that exhibit specific characteristics not adequately represented within the general population samples.

[0106] In some embodiments, the source domain includes a general face domain having general face image samples, and the target domain includes a face feature domain having image samples that exhibit specific face features.

[0107] In 708, the method may include: training, by a computing system, a modified generative adversarial network on a target domain training dataset to generate an output in the target domain. Training, by the computing system, the modified generative adversarial network may include: modifying, by the computing system, at least one of one or more first additional parameters or one or more second additional parameters.

[0108] In some embodiments, modifying, by the computing system, at least one of one or more first additional parameters or one or more second additional parameters may include: modifying, by the computing system, at least one of one or more first additional parameters or one or more second additional parameters while keeping a first plurality of pre-trained parameters and a second plurality of pre-trained parameters fixed.

[0109] In some embodiments, training, by the computing system, a modified generative adversarial network on a target domain training dataset may include: applying, by the computing system, a penalty to one or more norms of one or more first additional parameters or one or more second additional parameters to regularize the one or more first additional parameters or the one or more second additional parameters.

[0110] In some embodiments, one or more first additional parameters or one or more second additional parameters may be organized into a plurality of groups that respectively correspond to a plurality of layers of the modified generative adversarial network. In some of such embodiments, applying, by the computing system, a penalty to one or more norms may include: applying, by the computing system, a first penalty to corresponding L2 norms within one or more of the plurality of groups; and / or applying, by the computing system, a second penalty to corresponding L1 norms between two or more of the plurality of groups.

[0111] In some embodiments, training, by the computing system, a modified generative adversarial network on a target domain training dataset may include: applying, by the computing system, an early stopping scheme that measures both the diversity of the generated samples and the quality of the generated samples.

[0112] In some embodiments, training, by the computing system, a modified generative adversarial network on a target domain training dataset may include: optimizing, by the computing system, a min-max objective function.

[0113] In 710, the method may include: outputting, by a computing system, a modified generative adversarial network as a training model.

[0114] Additional disclosure

[0115] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for various possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes discussed herein may be implemented using a single device or component or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0116] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation and not limitation of the disclosure. After achieving an understanding of the foregoing, those skilled in the art can readily generate alterations, variations, and equivalents of such embodiments. Accordingly, the subject matter disclosure does not exclude including such modifications, variations, and / or additions to the subject matter that would be apparent to a person of ordinary skill in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet another embodiment. Accordingly, the disclosure is intended to cover such alterations, variations, and equivalents.

Claims

1. A computer-implemented method for performing domain adaptation on a generative model, the method comprising: obtaining, by a computing system including one or more computing devices, a pre-trained generative adversarial network that has been trained on a source domain training dataset to generate an output in a source domain, wherein the pre-trained generative adversarial network includes a generator model having a first plurality of pre-trained parameters and a discriminator model having a second plurality of pre-trained parameters; modifying, by the computing system, the pre-trained generative adversarial network to obtain a modified generative adversarial network, wherein modifying the pre-trained generative adversarial network by the computing system includes one or both of the following: adding, by the computing system, one or more first additional parameters to the generator model; and adding, by the computing system, one or more second additional parameters to the discriminator model; accessing, by the computing system, a target domain training dataset associated with a target domain different from the source domain, wherein: the source domain includes a first image domain, and the target domain includes a second image domain different from the first image domain, or the source domain includes a general domain having general image samples of a population, and the target domain includes a personalized domain having user-specific image samples, or the source domain includes a general domain having general image samples of a population, and the target domain includes an underrepresented domain having image samples exhibiting specific characteristics not adequately represented within the general image samples of the population, or the source domain includes a general face domain having general face image samples, and the target domain includes a face characteristic domain having image samples exhibiting specific face characteristics; training, by the computing system, the modified generative adversarial network on the target domain training dataset to generate an output in the target domain, wherein training the modified generative adversarial network by the computing system includes: modifying, by the computing system, at least one of the one or more first additional parameters or the one or more second additional parameters; and outputting, by the computing system, the modified generative adversarial network as a trained model.

2. The computer-implemented method according to claim 1, wherein, The target domain training dataset includes 25 or fewer target training examples of the target domain.

3. The computer-implemented method according to claim 1, wherein, Modifying, by the computing system, at least one of the one or more first additional parameters or the one or more second additional parameters includes: modifying, by the computing system, at least one of the one or more first additional parameters or the one or more second additional parameters while keeping the first plurality of pre-trained parameters and the second plurality of pre-trained parameters fixed.

4. The computer-implemented method according to claim 1, wherein, Modifying, by the computing system, the pre-trained generative adversarial network includes both of the following: adding, by the computing system, the one or more first additional parameters to the generator model; and adding, by the computing system, the one or more second additional parameters to the discriminator model.

5. The computer-implemented method according to claim 1, wherein, Training the modified generative adversarial network by the computing system on the target domain training dataset includes: the computing system applying a penalty to one or more norms of the one or more first additional parameters or the one or more second additional parameters to regularize the one or more first additional parameters or the one or more second additional parameters.

6. The computer-implemented method according to claim 5, wherein: The one or more first additional parameters or the one or more second additional parameters are organized into a plurality of groups, and the plurality of groups respectively correspond to a plurality of layers of the modified generative adversarial network; And The computing system applying the penalty to the one or more norms includes: The computing system applying a first penalty to the corresponding L2 norms within one or more of the plurality of groups; And The computing system applying a second penalty to the corresponding L1 norms between two or more of the plurality of groups.

7. The computer-implemented method according to claim 1, wherein: The computing system adding the one or more first additional parameters to the generator model includes: the computing system adding one or more parallel residual layers to the first convolutional neural network of the generator model; or The computing system adding the one or more second additional parameters to the discriminator model includes: the computing system adding one or more parallel residual layers to the second convolutional neural network of the discriminator model.

8. The computer-implemented method according to claim 1, wherein, Training the modified generative adversarial network by the computing system on the target domain training dataset includes: the computing system applying an early stopping scheme that measures both the diversity of the generated samples and the quality of the generated samples.

9. The computer-implemented method according to claim 1, wherein, Training the modified generative adversarial network by the computing system on the target domain training dataset includes: the computing system optimizing the min-max objective function.

10. A computing system configured to perform the method according to any one of claims 1 to 9.

11. One or more non-transitory computer-readable media collectively storing a modified generative adversarial network trained according to the method according to any one of claims 1 to 9.

Images