Method for few-shot unsupervised image-to-image translation

Abstract

A few-shot, unsupervised image-to-image translation ("FUNIT") algorithm is disclosed that accepts as input images of previously unseen target categories. These target categories are specified at inference time by only a few images of objects of the target type, such as a single image or a pair of images. A FUNIT network can be trained using a dataset containing images of many different object categories to translate images from one category to another using a few input images of the target category. By learning to extract appearance patterns for the translation task from a few input images, the network learns a generalizable appearance pattern extractor that can be applied to images of unseen categories at translation time for the few-shot image-to-image translation task.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Application No. 16 / 261,395, filed on January 29, 2019, entitled “METHOD FOR FEW-SHOT UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION,” the entire disclosure of which is incorporated herein by reference for all purposes. Background Art

[0003] Advances in processing power and image processing software have enabled an increasing variety of image creation and processing capabilities. For example, an image of a first type of object may be used to generate an image showing the first type of object having aspects of a second type of object. However, in order to accomplish such generation, a user must either manually generate or manipulate the image, or must provide a large number of input images of the first type of object that are sufficient to generate the target image. This can be complex and time consuming in the case of manual generation, and may be impractical in situations where a user may only have one or two images of the first type of object. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0005] Figure 1 Example image transformations that may be performed according to various embodiments are shown.

[0006] Figure 2 Components of an example image conversion network that can be used in accordance with various embodiments are shown.

[0007] Figure 3 An example system that may be used to perform image conversion in accordance with various embodiments is shown.

[0008] Figure 4 An example process for performing image conversion according to various embodiments is shown.

[0009] Figure 5 An example system for training an image synthesis network that can be used in accordance with various embodiments is shown.

[0010] Figure 6 Shown are the layers of an example statistical model that may be used in accordance with various embodiments.

[0011] Figure 7 Example components of a computing device that may be used to implement aspects of various embodiments are shown. DETAILED DESCRIPTION

[0012] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that these embodiments may be practiced without the specific details. In addition, well-known features may be omitted or simplified to avoid confusing the described embodiments.

[0013] Methods according to various embodiments provide for the generation of images including representations of objects having one or more specific aspects. In particular, various embodiments provide a few-shot, unsupervised image-to-image conversion ("FUNIT") algorithm that works only on previously unseen target categories specified by several example images at inference time, such as a single image of an object of the target type or a pair of input images. A FUNIT network can be trained using a dataset containing images of many different object categories to convert images from one category to another by utilizing a small number of input images of the target category. By learning to extract appearance patterns for conversion tasks from a small number of input images, the network learns a generalizable appearance pattern extractor that can be applied to images of unseen categories at conversion time to complete the few-shot image-to-image conversion task.

[0014] Various other functionalities may be implemented within the various embodiments and are discussed and suggested elsewhere herein.

[0015] Figure 1 An example image transformation 100 is shown that can be performed according to various embodiments. The transformation can accept an input image 106 (or a digital representation of an image) that includes a representation of one type of object, in this case a breed of dog corresponding to a pug. It may be desirable to generate images of pugs in different poses, such as to transform the representation of the pug from the pose of a target image 106 to a target pose, such as the pose in a selected or provided image 104. As shown, the image with the target pose includes representations of different types of objects, in this example, lions. The image transformation can take what it has learned from images of lions (and other such object categories) and use that learning to generate an image 108 of a pug showing the target pose. In this example, of interest is the fact that the image transformation network has been trained using images 102 of many categories of object types, including many images of the lion object type. The image transformation network has not been previously trained or processed on any images that include representations of pugs.

[0016] The method according to various embodiments may use one or more image-to-image conversion methods to perform such image generation or conversion as referred to herein. In various embodiments, an unsupervised image-to-image conversion technique may be used, which learns to map images in a given category to similar images in different categories, drawn on an unstructured (i.e., unregistered) image dataset. Although conventional methods can provide a conversion of an object's pose to an image, these methods require access to many images of the source category (e.g., showing the target pose) and the destination category (including previously unseen objects to obtain the desired pose) during training or at the time of conversion. The need for a large number of input images of the desired object category greatly limits the practicality of these conventional methods. Inspired by the ability of humans to extract the essence of new objects from a small number of examples and generalize from there, the method according to various embodiments utilizes a few-sample, unsupervised image-to-image conversion algorithm that works on a target category that has not been seen before, and the target category that has not been seen before is specified by only a small number of example images at the time of conversion, such as a single image of an object of the target type or a pair of input images. In some embodiments, the algorithm uses three or less, while in other embodiments, the algorithm can work using input images of less than five target categories. In some embodiments, a single image may be used, but a second or third image of the target object class in a different pose may be used to increase accuracy or photorealism. The example model achieves this few-shot imagination capability by coupling an adversarial training scheme with a novel network design.

[0017] As mentioned earlier, the human brain is very good at generalization. For example, when given a picture of an animal of a type never seen before, the human brain can form a vivid mental picture of the never-before-seen animal in a variety of poses, especially when that person is exposed to similar images or views, but with the difference that the animal is in those previous poses. For example, seeing a standing pug (e.g., standing up) for the first time may be a good mental picture of the animal. Figure 1 A person viewing an image of a pug sitting on its hind legs (as shown in example input image 106) will have little difficulty imagining what a pug would look like sitting on its hind legs (as shown in example output image 108) or lying down, given a lifetime of experience with other animals in similar postures and how those animals look in those postures.

[0018] While some conventional unsupervised image-to-image translation algorithms provide reasonable results in transferring complex appearance variations across image categories, the ability to generalize from a small number of samples of new categories based on prior knowledge is not possible using these algorithms. Specifically, these algorithms require large training sets of images of all categories on which the translation will be performed, and these algorithms do not support few-shot generalization.

[0019] The method according to various embodiments attempts to bridge the gap between human and machine imagination using what is referred to herein as a few-shot unsupervised image-to-image conversion ("FUNIT") framework. The FUNIT framework can learn an image-to-image conversion model for mapping an image of a source category to a similar image of a target category by utilizing a small number of images (i.e., two or less) of the target category given at the time of conversion. During training or other processes, the conversion model may have never encountered or processed images of the target category, but is required to generate some of them at the time of testing. The method can be based on the idea that humans' few-shot generation capabilities are developed from their past visual experiences. For example, if a person has seen many different object categories in the past, he can better imagine the view of a new object. Using this method, a dataset containing images of many different object categories can be used to train the FUNIT model to simulate these past visual experiences. Specifically, the model can be trained to convert images from one category to another by utilizing a small number of input images of the target category. By learning to extract appearance patterns from a few input images for conversion tasks, the model learns a generalizable appearance pattern extractor that can be applied to images of unseen categories at the time of conversion for use in few-shot image-to-image conversion tasks. In at least some embodiments, the performance of this few-shot conversion method improves as the number of classes in the training set increases. For the purposes of this discussion, source classes will refer to object classes included in the training data, which includes the expected pose or other aspects of the class for objects of the source class. Target classes will refer to object classes that are not included in the training data and are only visible at test, conversion, or inference time.

[0020] The image conversion framework according to various embodiments can be based on a generative adversarial network (GAN). The use of adversarial training schemes combined with a specific type of network design can achieve the desired few-sample unsupervised image-to-image conversion capabilities. Unsupervised and / or unpaired image-to-image conversion can attempt to learn a conditional image generation function that can map an input image of a source category to a similar image of a target category without paired supervision. This approach may not be enough because it attempts to restore the joint distribution using samples from marginal distributions. To address this problem, conventional methods use additional constraints, such as enforcing the conversion to retain certain properties of the source data, such as pixel values, pixel gradients, semantic features, category labels, or pairwise sample distances. Other methods are also used.

[0021] The method according to various embodiments utilizes a partially shared latent space assumption that is designed for a few-shot unsupervised image-to-image conversion task. Although capable of generating realistic conversion outputs, conventional unsupervised image-to-image conversion models are limited in at least two aspects. First, these models are sample inefficient and generate poor conversion outputs if only a small number of images are given during training. Second, the learned models are limited to converting images between two categories. Despite the similarity between the new task and the original task, a trained model for one conversion task cannot be directly reused for the new task. For example, a husky-to-cat conversion model cannot be reused for husky-to-tiger conversion, even though cats and tigers may have great similarities. While some conventional methods utilize an unsupervised image-to-image conversion framework to partially address the first aspect, the method according to various embodiments assumes that there are many source category images but a small number of target category images. In addition, it can be assumed that a small number of target category images are only available at test or conversion time and can come from many different object categories.

[0022] Multi-class unsupervised image-to-image translation extends the unsupervised image-to-image translation method to multiple classes. The method according to various embodiments utilizes a training dataset consisting of images of multiple classes, but instead of translating images between observed classes, it focuses on translating images of these observed classes to similar images of previously unseen classes.

[0023] The FUNIT framework according to various embodiments can map the image of the source category to a similar image of the unseen target category by utilizing a small amount of target category images available at the time of conversion. The training data can be composed of unpaired images from a set of object categories (e.g., images of dogs and cats of various breeds), which are referred to as source categories in this article. These images can be used to train an unsupervised image-to-image conversion model that converts images between any two source categories. During testing, a small amount of images from a new object category (e.g., cougar category) (referred to as the target category) can be provided to the model as input. The model can utilize a small amount of target images to convert any source category image to a similar image of the target category.

[0024] The FUNIT framework according to various embodiments may include a conditional image generator G and a multi-task adversarial discriminator D. Unlike the conditional image generator in the conventional unsupervised image-to-image conversion framework (which takes one image as input), the generator according to various embodiments may simultaneously accept a content image x belonging to a first category and a set of images (y1, ..., y2, y3, y4, y5, y6, y7, y8, y9, y10, y11, y12, y13, y14, y15, y16, y17, y18, y19, y21, y22, y23, y24, y25, y26, y27, y28, y29, y30, y31, y32, y33, y34, y35, y36, y3 K ) as input. The output image can be generated by the following equation:

[0025]

[0026] The content image may be assumed to belong to the destination object category, while each of the K class images belongs to the source object category. The generator may map an input content image to an output image such that the output image has the appearance of an image belonging to the destination object category but with the pose of the source object category, where the source object and the destination object share at least some structural similarity. During training, the generator may learn to convert images between two randomly sampled source categories. When converting, the generator may accept as input one or a small number of images from an unseen target category as images of the destination category, and map images sampled from any source category to similar images of the target or destination category.

[0027] Figure 2 Components of an example FUNIT framework 200 that can be used in accordance with various embodiments are shown. As previously described, a training set can consist of images of various object categories or source categories. A model (such as a GAN) can be trained to convert images between any or all of these source object categories. In deployment, the trained model can receive as input a few images of the target category, which are sufficient to convert the image of the source category to one or more similar images of the target category. An example FUNIT generator can accept as input a content image and one or more target category images. The generator can generate a transformation of the input content image that resembles or has the appearance of an image of the target category. Figure 2 As shown, the FUNIT generator can accept as input an image of a first object category and a set of images of a second object category, and can generate a transformation output. The transformation output should be similar to the image of the second object category and share the same class-invariant appearance as the input image.

[0028] The example generator G consists of four main components: content encoder, category encoder, adaptive instance norm (AdaIN) decoder, and image decoder. Instance norm and rectified linear unit (ReLU) can be applied to each convolutional layer and fully connected layer of the network. The content encoder can contain several convolutional layers followed by several residual blocks. The content encoder can map the input content image x to a spatially distributed feature map z x, referred to herein as content latent code. The category encoder may include several convolutional layers followed by an average pooling layer. The category encoder may accept a category image as input and output a category latent code. The average pooling layer may first average activations across spatial dimensions (e.g., height and width) and then across a set of images. The image decoder may include several AdaIN residual blocks followed by several upscale convolutional layers. The AdaIN residual block may be a residual block using an AdaIN layer as a normalization layer. For each sample, the AdaIN layer (also referred to as a normalization layer) may first normalize the activations in each channel to a zero mean and unit variance distribution. The normalization layer may then convert the distribution to have a specific mean and variance value through a denormalization process. The main goal of the image decoder is to decode the content latent code and the category latent code to generate a transformation of the input content image. In some embodiments, the AdaIN decoder is a multilayer perceptron. It decodes the category latent code into a set of mean and variance vectors, which are used as new means and variances for each channel in each AdaIN residual block in the image decoder. Using such a generator design, a content encoder can be used to extract a class-invariant latent representation (e.g., object pose), and a class encoder can be used to extract a class-specific latent representation (e.g., object appearance). By feeding the class latent code to the image decoder via the AdaIN layer, the class image controls the spatially invariant mean and variance, while the content image determines the remaining information. During training, the class encoder can learn to extract class-specific latent representations from images of the source class. At test or transfer time, this can generalize to images of previously unseen classes.

[0029] An example multi-task adversarial discriminator D is trained by solving multiple adversarial classification tasks simultaneously. In some embodiments, the discriminator is a patch GAN discriminator that can render an output space map of the input image, where each entry in the map indicates the score of the corresponding patch in the input image. In some embodiments, each task to be solved can be a binary classification task, determining whether the input image of D is a real image of the source category or a conversion output from the generator. Since there are many source categories, the discriminator can be designed to produce a corresponding number of outputs. When updating D for a real image of the source category, D can be penalized if a certain output is negative. For the conversion output that produces a fake image of the source category, D can be penalized if the corresponding output is positive. D may not be penalized for not predicting negative for other category images. When updating the generator G, G will only be penalized if the specified output of D is negative. In some embodiments, the discriminator D can be designed based on a category-conditional discriminator, which consists of several residual blocks followed by a global average pooling layer. The features produced by the global average pooling layer are called discriminator features, from which classification scores can be generated using a linear mapping.

[0030] The example FUNIT framework can be trained by solving the minimax optimization problem given by:

[0031] minmaxL GAN (D,G)+λ R L R (G)+λ FM L FM +λ GP L GP (D)

[0032] DG

[0033] Where L GAN , L R , L FM and L GP They are GAN loss, content reconstruction loss, feature matching loss and gradient penalty loss. R , FM and λ GP The relative importance of the control terms is set to 0.1, 1, and 10. The GAN loss can be a conditional loss. The discriminator can be trained to predict values ​​such as "real" for real images and "fake" for conversion outputs, while the generator can be trained to make the discriminator predict the appropriate conversion output as true.

[0034] In some embodiments, FUNIT model training can be regularized using an image reconstruction loss. Specifically, when using the same image as the input content image and the input category image, the model should generate an output image that is identical to the input. The feature matching loss L FM can be used to further improve the performance of few-shot conversion. Specifically, it is called D f The feature extractor of D can be constructed by removing the last (i.e. prediction) layer from D. f Can be used to extract features from the conversion output and the class image and minimize a given L1 loss. In some embodiments, a zero-centered gradient penalty loss can be applied to real images to obtain more stable GAN training. This may involve penalizing the discriminator with a non-zero gradient field.

[0035] Figure 3An example environment 300 is shown that can be used to implement aspects of various embodiments. In some embodiments, a user can provide an input image using a client device 602, which can be an image including a representation of an object that has not been encountered by the image generator network 328 before, as discussed herein. The user can also use the client device to select an image that indicates a target pose to be transformed for an object in the input image. The client device can be any suitable computing device that enables a user to select and / or provide an image for processing, and can include, for example, a desktop computer, a notebook computer, a smart phone, a tablet computer, a computer workstation, a game console, etc. The user can select, provide or otherwise specify the transformation input of the user interface (UI) of the image editor application 306 (or other image processing or generation software package) running on the client device, but in some embodiments at least some functions can also be operated on a remote device, a networked device or a "cloud". The user can provide input to the UI, such as through a touch-sensitive display 304 or by moving a mouse cursor displayed on a display screen, etc. As described above, the user may be able to provide an input image of a target category, and can select an image indicating a desired pose or other such aspect (which can be exhibited by an object in an image of a source category). The client device may include at least one processor 308 (e.g., a CPU or GPU) for executing applications and / or performing tasks on behalf of applications and a memory 310 including non-transitory computer-readable instructions for execution by the processor. Images provided to or generated via an application may be stored locally in a local memory 312, such as a hard drive or flash memory, etc.

[0036] In some embodiments, an input image received or selected on a client device 302 can be processed on the client device to generate an image with a desired transformation, such as applying the appearance of a target image to a gesture extracted from a set of source images. In other embodiments, the client device 302 can send an input image, data extracted from an image, or data specifying an image through at least one network 314 to be received by a remote computing system, which can be part of a resource provider environment 316. At least one network 314 can include any appropriate network, including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network or combination, and communication on the network can be achieved via wired and / or wireless connections. The provider environment 316 can include any appropriate components for receiving requests and returning information or performing actions in response to those requests. For example, the provider environment can include a Web server and / or an application server for receiving and processing requests and then returning data or other content or information in response to the requests.

[0037] Communications received to the provider environment 316 may be received to an interface layer 318. The interface layer 318 may include an application programming interface (API) or other exposed interface that enables a user to submit requests to the provider environment. The interface layer 318 in this example may also include other components, such as at least one web server, a routing component, a load balancer, etc. The components of the interface layer 318 may determine the type of request or communication and may direct the request to an appropriate system or service. For example, if the communication is for training an image conversion network for a category of image content (such as a bird, an animal, or a person), the communication may be directed to an image manager 320, which may be a system or service provided using various resources of the provider environment 316. The communication or information from the communication may be directed to a training manager 324, which may select an appropriate model or network and then train the model using the relevant training images and / or data 324. Once the network is trained and successfully evaluated, the network may be stored in a model repository 326, which may, for example, store different models or networks for different types of image conversion or processing. If a request including an input for an image transformation is received at the interface layer 318, the information of the request may be directed to an image generator 328 (also referred to herein as part of an image transformation network or service), which may obtain a corresponding trained network, such as a trained generative adversarial network (GAN) discussed herein, from a model repository 326 (if not already stored locally to the generator 328). The image generator 328 may take as input a target image (or a small number of images) and data indicating a target pose, which may be exhibited by a selected source image, as discussed herein. The image generator 328 may then cause the input to be processed to generate an image representing the target transformation. As described above, this may involve processing the input by a content encoder 330 to extract a category-invariant latent representation, such as one that may correspond to a selected pose. The category encoder 332 may extract a category-specific latent representation (e.g., object appearance) from the input target image. The class latent code may be fed to an AdaIN decoder 334, which decodes the class latent code into a set of mean and variance vectors that are used as new means and variances for each channel in each AdaIN residual block in an image decoder 336. The generated image may then be sent to the client device 302 for display on the display element 304, or for other such purposes. If the user wants to modify any aspect of the image, the user may provide additional input to the application 306, which may cause the same process to be used to generate a new or updated image for the new or updated input, such as additional images of the target class or specifications of different poses, etc. In some embodiments, the image generation network may utilize a deep generative model that may learn to sample images given a training dataset.The models used may include, for example, generative adversarial networks (GANs) and variational autoencoder (VAE) networks, while targeting image conversion tasks. In various embodiments, the image conversion network or converter 340 may include a GAN consisting of a generator 328 and a discriminator 338. The generator 328 may be used to produce converted images such that the discriminator cannot distinguish between real and generated.

[0038] In various embodiments, the processor 308 (or the processor of the training manager 322 or the image converter 340) will be a central processing unit (CPU). However, as previously described, resources in such an environment can utilize GPUs to process data for at least certain types of requests. GPUs have thousands of cores and are designed to handle massively parallel workloads, so they have become popular in deep learning for training neural networks and generating predictions. Although using GPUs for offline construction can enable faster training of larger and more complex models, generating predictions offline means that request-time input features cannot be used, or predictions must be generated for all permutations of features and stored in lookup tables to serve real-time requests. If the deep learning framework supports CPU mode and the model is small and simple enough, feedforward can be performed on the CPU with reasonable latency, then the service on the CPU instance can host the model. In this case, training can be done offline on the GPU and reasoning is done in real time on the CPU. If the CPU approach is not a viable option, then the service can run on a GPU instance. However, since GPUs have different performance and cost characteristics compared to CPUs, running a service that offloads runtime algorithms to the GPU may require that it be designed differently from a CPU-based service.

[0039] Figure 4An example process 400 for generating an image that can be utilized according to various embodiments is shown, which represents the transformation of a representation of a previously unseen object to have a pose learned from a known source object category. It should be understood that for this process and other processes discussed herein, unless otherwise stated, there may be additional, alternative or fewer steps performed in a similar or alternative order or in parallel within the scope of various embodiments. In addition, although pose is used as a primary example, there may be other aspects of the source image category for generating a transformation of an input object as discussed and suggested herein. In this example, an input digital image is received 402, or otherwise obtained or specified, which includes a representation of an object of interest. In this example, the object belongs to a source category that the neural network has not encountered for processing the image, or at least belongs to a category that has not been used to train the network. A selection of a source image may also be obtained 404, wherein the source image includes a representation of a source object of an object category for training the neural network. In some embodiments, the user may select the source object, while in other embodiments, the system or network may select the source image or pose, as well as other options discussed and suggested herein.

[0040] Once obtained, the input image and the source image (or an indication of the source image) can be provided 406 as input to the trained neural network. As discussed herein, the network can be trained using many images of multiple object categories (including the source category) to learn to convert between objects of each category. One or more encoders of the generator of the network can extract 408 a category-invariant latent representation corresponding to the target pose from the source image or category. Category-specific latent representations corresponding to the appearance of the target object can also be extracted 410 from the input image. The neural network can then be used to infer 412 a converted image including a representation of an object of the target category with the target pose based at least in part on these latent representations. As previously described, such a process can produce high-quality converted images without requiring a large number of input images of the target category.

[0041] As described above, various embodiments utilize machine learning. For example, deep neural networks (DNNs) developed on processors have been used in a variety of use cases, from self-driving cars to faster drug development, from automatic image annotation in online image databases to intelligent real-time language conversion in video chat applications. Deep learning is a technology that mimics the neural learning process of the human brain, constantly learning, constantly getting smarter, and providing more accurate results faster over time. Children are initially taught by adults how to correctly identify and classify various shapes, and eventually are able to recognize shapes without any guidance. Similarly, a deep learning or neural learning system needs to be trained in object recognition and classification as it becomes smarter and more efficient in recognizing basic objects, occluded objects, etc., while also assigning context to objects.

[0042] At the simplest level, neurons in the human brain look at the various inputs they receive, assign importance levels to each of these inputs, and pass outputs to other neurons for action. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is being trained to recognize and classify, and each of these features is assigned a specific weight based on how important that feature is in defining the shape of the object.

[0043] Deep neural network (DNN) models include multiple layers of connected perceptrons (e.g., nodes) that can be trained with large amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DLL model breaks down an input image of a car into its components and looks for basic patterns such as lines and angles. The second layer assembles the lines to look for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the vehicle class, and the final layers generate labels for the input image, identifying the model of a specific car brand. Once a DNN is trained, the DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on checks deposited into an ATM machine, identifying images of friends in photos, providing movie recommendations to more than 50 million users, identifying and classifying cars, pedestrians, and road hazards in different types of self-driving cars, or translating human speech in real time.

[0044] During training, data flows through the DNN in a forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not label the input correctly, the error between the correct label and the predicted label is analyzed, and the weights of each feature are adjusted in a backpropagation phase until the DNN correctly labels the input and other inputs in the dataset during training. Training complex neural networks requires a large amount of parallel computing performance, including supporting floating-point multiplication and addition. Inference, which is less computationally intensive than training, is a latency-sensitive process in which a trained neural network is applied to new inputs it has never seen before to classify images, convert speech, and generally infer new information.

[0045] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a lot of floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, computing platforms can deliver the performance needed for deep neural network-based artificial intelligence and machine learning applications.

[0046] Figure 5An example system 500 that can be used to classify data or generate inferences according to various embodiments is shown. Based on the teachings and suggestions contained herein, it should be apparent that various predictions, labels, or other outputs can also be generated for input data. In addition, supervised training and unsupervised training can be used in various embodiments discussed herein. In this example, a set of classified data 502 is provided as input to be used as training data. The classified data may include instances of at least one type of object for which a statistical model is to be trained, and information identifying the type of object. For example, the classified data may include a set of images, each image containing a representation of an object type, wherein each image also contains or is associated with a label, metadata, classification, or other information identifying the type of object represented in the respective image. Various other types of data may also be used as training data, and may include text data, audio data, video data, and the like. In this example, the classified data 502 is provided as training input to a training manager 504. The training manager 504 may be a system or service including hardware and software, such as one or more computing devices that execute a training application for training a statistical model. In this example, the training manager 504 receives an instruction or request indicating the type of model to be used for training. The model may be any suitable statistical model, network or algorithm that can be used for such purposes, and may include, for example, an artificial neural network, a deep learning algorithm, a learning classifier, a Bayesian network, etc. The training manager 504 may select a base model or other untrained model from an appropriate repository 506 and train the model using the classified data 502 to generate a trained model 508 that can be used to classify similar types of data. In some embodiments that do not use classified data, the input data may still be trained by selecting an appropriate base model according to the training manager.

[0047] Models can be trained in a number of different ways, which may depend in part on the type of model selected. For example, in one embodiment, a set of training data can be provided to a machine learning algorithm, where the model is a model artifact created by a training process. Each instance of the training data contains a correct answer (e.g., a classification), which can be referred to as a target or target attribute. The learning algorithm finds patterns in the training data that map the input data attributes to the target, the answer to be predicted, and outputs a machine learning model that captures these patterns. The machine learning model can then be used to obtain predictions for new data for which the target is not specified.

[0048] In one example, the training manager can select from a set of machine learning models, including binary classification, multi-classification, and regression models. The type of model to be used can depend at least in part on the type of target to be predicted. Machine learning models for binary classification problems predict binary outcomes, such as one of two possible classes. Learning algorithms such as logistic regression can be used to train binary classification models. Machine learning models for multi-class classification problems allow predictions to be generated for multiple classes, such as predicting one of more than two outcomes. Multinomial logistic regression can be useful for training multi-class models. Machine learning models for regression problems predict numerical values. Linear regression is useful for training regression models.

[0049] In order to train a machine learning model according to one embodiment, a training manager must determine the input training data source and other information, such as the name of the data attribute containing the target to be predicted, the required data transformation instructions, and training parameters to control the learning algorithm. During the training process, in some embodiments, the training manager can automatically select an appropriate learning algorithm based on the target type specified in the training data source. The machine learning algorithm can accept parameters for controlling the training process and certain properties of the resulting machine learning model. These are referred to as training parameters in this article. If no training parameters are specified, the training manager can use known default values ​​to handle a wide range of machine learning tasks well. Examples of training parameters for which values ​​can be specified include the maximum model size, the maximum number of passes on the training data, the type of random play, the type of regularization, the learning rate, and the amount of regularization. Default settings can be specified, with options for adjusting values ​​to fine-tune performance.

[0050] The maximum model size is the total size in bytes of the patterns created during training the model. By default, a model of the specified size is created, for example, a 100MB model. If the training manager cannot determine that there are enough patterns to fill the model size, a smaller model may be created. If the training manager finds that there are more patterns than can fit within the specified size, it may enforce a maximum cutoff by trimming the patterns that have the least impact on the quality of the learned model. Choosing a model size provides control over the tradeoff between the predictive quality of the model and the cost of using it. Smaller models may cause the training manager to remove many patterns to fit within the maximum size limit, affecting the quality of the predictions. On the other hand, larger models may be more expensive to query for real-time predictions. Larger input datasets do not necessarily result in larger models, because the model stores the patterns rather than the input data. If the patterns are few and simple, the resulting model will be small. Input data with a large number of raw attributes (input columns) or derived features (output of data transformations) may find and store more patterns during the training process.

[0051] In some embodiments, the training manager may make multiple passes or iterations over the training data to discover patterns. There may be a default number of passes, such as ten, and in some embodiments, a maximum number of passes may be set, such as up to one hundred passes. In some embodiments, there may be no maximum set, or there may be a convergence criterion or other set of criteria that triggers the end of the training process. In some embodiments, the training manager may monitor the quality of the pattern (i.e., model convergence) during training, and may automatically stop training when there are no more data points or patterns to discover. Data sets with only a small number of observations may require more data passes to achieve higher model quality. Larger data sets may contain many similar data points, which can reduce the need for a large number of passes. The potential impact of choosing more data passes on the data is that model training may take longer and cost more in terms of resources and system utilization.

[0052] In some embodiments, the training data is shuffled before training or between passes of training. In many embodiments, the shuffle is a random or pseudo-random shuffle that is used to generate a truly random ordering, although there may be some constraints to ensure that certain types of data are not grouped, or if such grouping exists, the shuffled data can be reshuffled, etc. Shuffling changes the sequence or arrangement in which the data is used for training so that the training algorithm does not encounter groupings of similar types of data or a single type of data that is observed too many times in a row. For example, a model can be trained to predict product types, where the training data includes movie, toy, and video game product types. Before uploading, the data may be sorted by product type. The algorithm can then process the data in alphabetical order by product type, first seeing only data of one type (such as movies). The model will begin to learn the patterns of movies. Then, the model will only encounter data for different product types (e.g., toys) and will try to adjust the model to fit the toy product type, which may degenerate the pattern suitable for movies. This sudden switch from movie to toy type may produce a model that cannot learn how to accurately predict product types. Before partitioning the training data set into the training subset and the evaluation subset, in some embodiments, shuffling may be performed to utilize a relatively uniform distribution of data types for both phases. In some embodiments, the training manager may automatically shuffle the data using, for example, a pseudo-random shuffling technique.

[0053] In some embodiments, when creating a machine learning model, the training manager can enable the user to specify settings or apply customization options. For example, the user can specify one or more evaluation settings to indicate a portion of the input data to be retained for evaluating the predictive quality of the machine learning model. The user can specify a method that indicates which attributes and attribute transformations can be used for model training. The user can also specify various training parameters that control the training process and certain properties of the resulting model.

[0054] Once the training manager determines that the training of the model is complete, such as by using at least one of the final criteria discussed herein, the trained model 508 can be provided for use by the classifier 514 to classify the unclassified data 512. However, in many embodiments, the trained model 508 will first be passed to an evaluator 510, which can include an application or process executed on at least one computing resource for evaluating the quality (or other aspects) of the trained model. The model is evaluated to determine whether the model will provide at least a minimum acceptable or threshold level of performance when predicting targets for new and future data. Since future data instances will typically have unknown target values, it may be desirable to examine the accuracy metric of machine learning on data for which the target answer is known and use the evaluation as a proxy for the prediction accuracy of future data.

[0055] In some embodiments, a subset of the classified data 502 provided for training is used to evaluate the model. The subset can be determined using the shuffling and splitting method described above. This evaluation data subset will be marked with a target, so it can serve as a resource for evaluating ground truth. It is useless to use the same data used for training to evaluate the prediction accuracy of the machine learning model, because a positive evaluation may be produced for a model that remembers the training data rather than generalizing it. Once training is completed, the evaluation data subset is processed using the trained model 508, and the evaluator 510 can determine the accuracy of the model by comparing the ground truth data with the corresponding output (or prediction / observation) of the model. In some embodiments, the evaluator 510 can provide a summary or performance metric that indicates the degree of match between the predicted value and the true value. If the trained model does not meet at least the minimum performance standard or other such accuracy threshold, the training manager 504 can be instructed to perform further training, or in some cases, try to train a new or different model, etc. If the trained model 508 meets the relevant standards, the trained model can be provided for use by the classifier 514.

[0056] When creating and training a machine learning model, in at least some embodiments, it may be desirable to specify model settings or training parameters that will result in a model that is capable of making the most accurate predictions. Example parameters include the number of passes to be performed (forward and / or backward), regularization, model size, and shuffle type. However, as described above, selecting the model parameter settings that produce the best predictive performance on the evaluation data may result in overfitting the model. Overfitting occurs when the model stores patterns that appear in the training and evaluation data sources, but fails to generalize patterns in the data. Overfitting often occurs when the training data includes all the data used in the evaluation. An overfitted model may perform well during evaluation, but may not make accurate predictions on new data or other classified data. To avoid selecting an overfitted model as the best model, the training manager may retain additional data to verify the performance of the model. For example, the training data set may be divided into 60% for training and 40% for evaluation or validation, which may be divided into two or more stages. After the model parameters that best fit the evaluation data are selected, resulting in convergence to a subset of the validation data (e.g., half of the validation data), a second validation may be performed using the remaining validation data to ensure the performance of the model. If the model meets the expectations on the validation data, then the model is not overfitting the data. Optionally, a test set or holdout set can be used to test parameters. Using a second validation or testing step can help choose appropriate model parameters to prevent overfitting. However, taking more data out of the training process for validation leaves less data available for training. This can be problematic for smaller datasets because there may not be enough data available for training. One approach in this case is to perform cross-validation, as described elsewhere in this article.

[0057] There are many metrics or insights that can be used to review and evaluate the predictive accuracy of a given model. An example evaluation result contains a predictive accuracy metric to report the overall success of the model, as well as visualizations to help explore the accuracy of the model beyond the predictive accuracy metric. The results can also provide the ability to see the impact of setting score thresholds (such as binary classification) and can generate alerts on criteria used to check the effectiveness of the evaluation. The choice of metrics and visualizations can depend at least in part on the type of model being evaluated.

[0058] After satisfactory training and evaluation, the trained machine learning model can be used to build or support machine learning applications. In one embodiment, building a machine learning application is an iterative process involving a series of steps. The core machine learning problem can be constructed based on what is observed and the answers that the model is to predict. Data can then be collected, cleaned, and prepared to make it suitable for use by the machine learning model training algorithm. These data can be visualized and analyzed for integrity checks to verify data quality and understand the data. This may be the case where the original data (such as input variables) and answers (such as targets) are not represented in a way that can be used to train highly predictive models. Therefore, it may be desirable to build more predictive input representations or features from the original variables. The resulting features can be input into a learning algorithm to build a model and evaluate the quality of the model based on the data retained from model building. The model can then be used to generate predictions of the target answer for new data instances.

[0059] exist Figure 5 In the exemplary system 500 of , after providing the evaluation, the trained model 510 is provided to or made available to a classifier 514, which is capable of using the trained model to process unclassified data. For example, this may include data received from a user or unclassified third party, such as query images that are being queried for information about the content represented in these images. The unclassified data can be processed by the classifier using the trained model, and the resulting results 516 (i.e., classifications or predictions) can be sent back to the corresponding source or otherwise processed or stored. In some embodiments, and where such use is permitted, these currently classified data instances can be stored in a classified data repository, which can be used by the training manager for further training of the trained model 508. In some embodiments, the models will be continuously trained as new data becomes available, but in other embodiments, these models will be trained periodically, such as once a day or once a week, depending on factors such as the size of the data set or the complexity of the model.

[0060] The classifier may include appropriate hardware and software for processing unclassified data using a trained model. In some cases, the classifier will include one or more computer servers, each having one or more graphics processing units (GPUs) capable of processing data. The configuration and design of the GPU may make them more suitable for processing machine learning data than a CPU or other such components. In some embodiments, the trained model may be loaded into the GPU memory, and the received data instance may be provided to the GPU for processing. The GPU may have many more cores than a CPU, and the GPU core may be less complex. Therefore, a given GPU may be able to process thousands of data instances simultaneously through different hardware threads. The GPU may also be configured to maximize floating point throughput, which can provide significant additional processing advantages for large data sets.

[0061] Even when GPUs, accelerators, and other such hardware are used to accelerate tasks such as model training or data classification using such models, such tasks may still require significant time, resource allocation, and cost. For example, if a machine learning model is to be trained using 100 passes, and the dataset includes 1,000,000 data instances to be used for training, each pass will require processing all million instances. Different parts of the architecture may also be supported by different types of devices. For example, training may be performed using a set of servers at a logically centralized location, as may be provided as a service, while classification of the raw data may be performed by such a service or on a client device, among other such options. In various embodiments, these devices may also be owned, operated, or controlled by the same entity or multiple entities.

[0062] Figure 6An example neural network 600 or other statistical model that can be utilized according to various embodiments is shown. In this example, the statistical model is an artificial neural network (ANN), which includes multiple node layers, including an input layer 602, an output layer 606, and multiple layers 604 of intermediate nodes, which are generally referred to as "hidden" layers, because the internal layers and nodes are generally not visible or accessible in conventional neural networks. As discussed elsewhere in this article, other types of statistical models can also be used, as well as other types of neural networks including other numbers or selections of nodes and layers, and so on. In this network, all nodes of a given layer are interconnected to all nodes of adjacent layers. As shown in the figure, the nodes of the intermediate layer will then be connected to the nodes of two adjacent layers respectively. In some models, nodes are also called neurons or connected units, and the connections between nodes are called edges. Each node can perform a function for the received input, such as by using a specified function. Nodes and edges can obtain different weights during the training process, and the various layers of nodes can perform specific types of conversions on the received input, and these conversions can also be learned or adjusted during the training process. Learning can be supervised learning or unsupervised learning, which may depend at least in part on the type of information contained in the training data set. Various types of neural networks can be utilized, for example, a convolutional neural network (CNN) can be included, which includes many convolutional layers and a set of pooling layers and has been shown to be beneficial for applications such as image recognition. CNNs are also easier to train than other networks due to the relatively small number of parameters to be determined.

[0063] In some embodiments, various tuning parameters can be used to train such complex machine learning models. Selecting parameters, fitting models, and evaluating models are part of the model tuning process, commonly referred to as hyperparameter optimization. In at least some embodiments, such adjustments can include introspection of the underlying model or data. In training or production settings, a robust workflow is very important to avoid overfitting of hyperparameters, as described elsewhere herein. Cross-validation and adding Gaussian noise to training data sets are useful techniques to avoid overfitting any one data set. For hyperparameter optimization, in some embodiments, it may be necessary to keep the training set and validation set fixed. In some embodiments, hyperparameters can be adjusted in certain categories, such as data preprocessing (in other words, converting words to vectors), CNN architecture definition (e.g., filter size, number of filters), stochastic gradient descent parameters (e.g., learning rate), regularization (e.g., drop probability), and other such options.

[0064] In an example preprocessing step, instances of a data set may be embedded into a lower dimensional space of a specific size. The size of this space is a parameter to be adjusted. The architecture of a CNN contains many adjustable parameters. The parameters of the filter size may represent an interpretation of information that corresponds to the size of the instance to be analyzed. In computational linguistics, this is called the n-gram size. The example CNN uses three different filter sizes, which represent possibly different n-gram sizes. The number of filters for each filter size may correspond to the depth of the filter. Each filter attempts to learn something different from the instance structure, such as the sentence structure of text data. In the convolutional layer, the activation function may be a rectified linear unit, and the pooling type is set to a maximum pool. The results may then be concatenated into a one-dimensional vector, and the last layer is fully connected to the two-dimensional output. This corresponds to a binary classification to which an optimization function may be applied. One such function is an implementation of the root mean square (RMS) propagation method of gradient descent, where example hyperparameters may include learning rate, batch size, maximum gradient normal, and epoch. For neural networks, regularization may be a very important consideration. As described, in some embodiments, the input data may be relatively sparse. In this case, the main hyperparameters can be dropped at the penultimate layer, which means that a certain proportion of nodes will not "fire" at each training cycle. The example training process can suggest different hyperparameter configurations based on feedback on the performance of previous configurations. The model can be trained using the suggested configuration, can be evaluated on a specified validation set, and performance can be reported. This process can be repeated, such as to balance exploration (learn more about different configurations) and development (leverage previous knowledge to get better results).

[0065] Since training CNNs can be parallelized and can take advantage of GPU-powered computing resources, multiple optimization strategies can be tried for different scenarios. Complex scenarios allow tuning of the model architecture and preprocessing and stochastic gradient descent parameters. This expands the model configuration space. In the base case, only preprocessing and stochastic gradient descent parameters are tuned. In complex scenarios, more configuration parameters can be used than in the base scenario. Tuning of the joint space can be performed using linear or exponential steps and iterated through the optimization loop of the model. Such a tuning process can be significantly less expensive than tuning processes such as random search and grid search without any noticeable performance loss.

[0066] Some embodiments may use back propagation to calculate the gradient for determining the weights of a neural network. Back propagation is a form of differentiation, and as described above, a gradient descent optimization algorithm may be used to adjust the weights applied to various nodes or neurons. In some embodiments, the gradient of the associated loss function may be used to determine the weights. Back propagation may utilize the derivative of the output generated by the loss function to the statistical model. As described above, each node may have an associated activation function defining the output of each node. Various activation functions may be appropriately used, such as radial basis functions (RBFs) and sigmoid functions, which may be used by various support vector machines (SVMs) for data conversion. The activation function of the intermediate layer of a node is referred to herein as an inner product core. These functions may include, for example, recognition functions, step functions, sigmoid functions, ramp functions, and the like. The activation function may also be linear or nonlinear, as well as other such options.

[0067] Figure 7 A set of basic components of a computing device 700 is shown, which can be used to implement aspects of various embodiments. In this example, the device includes at least one processor 702 for executing instructions that may be stored in a memory device or element 704. It will be apparent to one of ordinary skill in the art that the device may include many types of memory, data storage, or computer-readable media, such as a first data storage for program instructions executed by the processor 702, the same or separate storage for images or data, removable memory for sharing information with other devices, and any number of communication methods for sharing with other devices. The device will typically include some type of display element 706, such as a touch screen, organic light emitting diode (OLED), or liquid crystal display (LCD), but the device (such as a portable media player) may convey information via other means, such as through audio speakers. As discussed, the device in many embodiments will include at least a communication component 708 and / or a network component 710, such as may support wired or wireless communication over at least one network, such as the Internet, a local area network (LAN), Or cellular network, etc. These components can make the device communicate with remote system or service. The device can also include at least one additional input device 712 that can receive conventional input from the user. This conventional input can include, for example, a button, a touch pad, a touch screen, a roller device, a joystick, a keyboard, a mouse, a tracking ball, a keypad or any other such device or element, through which the user inputs commands to the device. In some embodiments, these I / O devices can even be connected by infrared or Bluetooth or other links. However, in some implementations, such a device may not include any buttons at all, and may only be controlled by a combination of visual and audio commands, so that the user can control the device without having to contact the device.

[0068] Various embodiments can be implemented in a variety of operating environments, which in some cases can include one or more user computers or computing devices that can be used to operate any one of a plurality of applications. User or client devices can include any of a variety of general-purpose personal computers, such as desktops or laptops running standard operating systems, and cellular, wireless and handheld devices running mobile software and capable of supporting a variety of network and messaging protocols. Such a system can also include a plurality of workstations running any one of a variety of commercial operating systems and other known applications for purposes such as development and database management. These devices can also include other electronic devices, such as virtual terminals, thin clients, gaming systems and other devices capable of communicating over a network.

[0069] Most embodiments utilize at least one network familiar to those skilled in the art to support communications using any of a variety of commercially available protocols, such as TCP / IP or FTP. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. In embodiments utilizing a network server, the network server can run any of a variety of server or middle-tier applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. One or more servers can also execute programs or scripts in response to requests from user devices, such as by executing one or more Web applications, which can be implemented in any programming language, such as JavaScript. C, C# or C++, or any scripting language, such as Python, and combinations thereof). One or more servers may also include a database server, including but not limited to and Those servers that are commercially available.

[0070] Environment can include various data storage and other memory and storage media as described above. These can reside in various locations, such as on a storage medium local to one or more computers (and / or reside therein), or away from any or all computers on the network. In a specific set of embodiments, information can reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to computers, servers or other network devices can be appropriately stored locally and / or remotely. In the case where the system includes a computerized device, each such device can include hardware elements that can be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (such as a mouse, keyboard, controller, touch-sensitive display element or keypad) and at least one output device (such as a display device, printer or speaker). Such a system can also include one or more storage devices, such as disk drives, optical storage devices and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), and removable media devices, memory cards, flash memory cards, etc.

[0071] Such a device may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device) and a working memory as described above. A computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium representing a remote, local, fixed and / or removable storage device and a storage medium for temporarily and / or more permanently containing, storing, sending and retrieving computer-readable information. The system and various devices will generally also include a plurality of software applications, modules, services or other elements located in at least one working memory device, including an operating system and an application, such as a client application or a web browser. It should be understood that alternative embodiments may have many variations different from the above-described embodiments. For example, customized hardware and / or specific elements that may be implemented in hardware, software (including portable software, such as applets) or both may also be used. In addition, connections to other computing devices (such as network input / output devices) may be employed.

[0072] Storage media and other non-transitory computer-readable media for containing code or code portions may include any appropriate media known or used in the art, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by system devices. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will understand other ways and / or methods of implementing various embodiments.

[0073] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

Claims

1. A computer-implemented method comprising: One or more images of the one or more first objects are generated using one or more neural networks based at least in part on the one or more pose-independent features of the one or more first objects and the one or more poses of the one or more second objects.

2. The computer-implemented method of claim 1 , wherein the one or more neural networks comprises a generative adversarial network (GAN), the generative adversarial network comprising a conditional image generator and a multi-task adversarial discriminator.

3. The computer-implemented method of claim 2, further comprising: extracting a class-invariant representation of the one or more poses using a content encoder of the conditional image generator; as well as A category encoder of the conditional image generator is used to extract a category-specific latent representation of the appearance of the one or more first objects.

4. The computer-implemented method of claim 2, further comprising: The layer activations are normalized to zero mean and unit variance distribution by a normalization layer of the multi-task adversarial discriminator; as well as Denormalize the normalized layer activations using an affine transformation.

5. The computer-implemented method of claim 1 , further comprising: Adversarial training of the one or more neural networks is performed to transfer poses between multiple object categories.

6. The computer-implemented method of claim 1, comprising: receiving digital representations of one or more additional images of the one or more first objects; as well as The one or more images of the one or more first objects are reasoned about using the one or more neural networks, wherein the one or more neural networks include a conditional generator and an adversarial discriminator to extract category-invariant representations of the one or more poses and category-specific representations of appearances of the one or more first objects.

7. The computer-implemented method of claim 1, wherein the one or more neural networks are not trained using images of an object category of the one or more first objects.

8. The computer-implemented method of claim 1 , further comprising: Unsupervised training of the one or more neural networks is performed using a training dataset comprising a plurality of images of a plurality of object categories, wherein the one or more neural networks learn to convert poses between the plurality of object categories.

9. The computer-implemented method of claim 4, further comprising: The class latent code is fed to an adaptive instance norm (AdaIN) decoder to decode the class latent code into a set of mean and variance vectors, which are used as new means and variances for each channel in each AdaIN residual block of the image decoder of the one or more neural networks.

10. The computer-implemented method of claim 2, wherein The conditional image generator comprises a content encoder comprising a plurality of first convolutional layers followed by a plurality of residual blocks, and Wherein the conditional image generator comprises a category encoder comprising a plurality of second convolutional layers followed by an average pooling layer.

11. The computer-implemented method of claim 10, further comprising: The average pooling layer averages the activations over the spatial dimensions before averaging the activations over the set of images.

12. A system comprising: at least one processor; as well as A memory comprising instructions that, when executed by the at least one processor, cause the system to: generate one or more images of the one or more first objects using one or more neural networks based at least in part on one or more pose-independent features of the one or more first objects and one or more poses of one or more second objects.

13. The system of claim 12, wherein the one or more neural networks are not trained using images of an object category of the one or more first objects.

14. The system of claim 12, wherein the instructions, when executed by the at least one processor, further cause the system to: Unsupervised training of the one or more neural networks is performed using a training dataset comprising a plurality of images of different object categories, wherein the one or more neural networks learn to convert poses between the different object categories.

15. The system of claim 12, wherein the one or more neural networks comprise a generative adversarial network (GAN).

16. The system of claim 12, wherein the instructions, when executed by the at least one processor, further cause the system to: extracting class-invariant representations of the one or more gestures using a content encoder of a conditional generator included in the one or more neural networks; and A category encoder of the conditional generator is used to extract a category-specific latent representation of the appearance of the one or more first objects.

17. The system of claim 12, wherein the instructions, when executed by the at least one processor, further cause the system to: normalizing layer activations to zero mean and unit variance distributions by a normalization layer of an adversarial discriminator included in the one or more neural networks; and Denormalize the normalized layer activations using an affine transformation.

18. A processor, comprising: One or more circuits that generate one or more images of the one or more first objects using one or more neural networks based at least in part on one or more pose-independent features of the one or more first objects and one or more poses of one or more second objects.

19. The processor of claim 18, wherein The one or more neural networks are trained using one or more additional images depicting the one or more second objects; and The one or more neural networks are not trained using images depicting the one or more first objects.

20. The processor of claim 18, wherein The one or more circuits extract, using the one or more neural networks, representations of the one or more poses from one or more images of the one or more second objects; and The one or more circuits generate the one or more images of the one or more first objects using the one or more neural networks based at least in part on the extracted representations of the one or more poses.

21. The processor of claim 18, wherein the one or more circuits generate the one or more images of the one or more first objects using the one or more neural networks based at least in part on a set of spatially invariant mean and variance vectors generated from the one or more pose-independent features.

Images