Method for few-sample unsupervised image-to-image conversion

Abstract

A low sample, unsupervised image-to-image conversion ("FUNIT") algorithm is disclosed that accepts as input an image of a target category that was not previously seen. These target categories are specified only by a few images of the object of the target type, such as a single image or a pair of images, when reasoning. The FUNIT network may be trained using a dataset of images containing a number of different object categories to convert images from one category to another by utilizing a small number of input images of the target category. The network learns a generalizable appearance pattern extractor that can be applied to images that do not see categories during conversion by learning the appearance patterns for the conversion tasks extracted from a small number of input images for fewer sample image-to-image conversion tasks.

Description

[0001] The application date of this invention patent is January 27, 2020, and the application number is

[0002] 202080011323.4, and a divisional application of the invention patent application entitled “Method for Few-Sample Unsupervised Image-to-Image Conversion”.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of priority to U.S. application No. 16 / 261,395, filed on January 29, 2019, entitled “METHOD FOR FEW-SHOT UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION,” the entire disclosure of which is incorporated herein by reference for all purposes. Background Art

[0005] Advances in processing power and image processing software have enabled an increasing variety of image creation and processing capabilities. For example, an image of a first type of object can be used to generate an image showing the first type of object with aspects of a second type of object. However, to accomplish this generation, the user must either manually generate or manipulate the image or provide a large number of input images of the first type of object sufficient to generate the target image. This can be complex and time-consuming in manual generation and may be impractical in situations where the user may only have one or two images of the first type of object. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0007] Figure 1 Example image transformations that may be performed according to various embodiments are shown.

[0008] Figure 2 Components of an example image conversion network that can be used in accordance with various embodiments are shown.

[0009] Figure 3 An example system that may be used to perform image conversion in accordance with various embodiments is shown.

[0010] Figure 4 An example process for performing image conversion according to various embodiments is shown.

[0011] Figure 5 An example system for training an image synthesis network that can be used in accordance with various embodiments is shown.

[0012] Figure 6Shown are layers of an example statistical model that may be used in accordance with various embodiments.

[0013] Figure 7 Illustrated are example components of a computing device that may be used to implement aspects of various embodiments. DETAILED DESCRIPTION

[0014] In the following description, various embodiments are described. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that these embodiments can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments.

[0015] Methods according to various embodiments provide for the generation of images comprising representations of objects having one or more specific aspects. In particular, various embodiments provide a few-shot, unsupervised image-to-image translation ("FUNIT") algorithm that works at inference time only on previously unseen target categories specified by a number of example images, such as a single image of an object of the target type or a pair of input images. A FUNIT network can be trained using a dataset containing images of many different object categories to translate images from one category to another using a small number of input images of the target category. By learning to extract appearance patterns for the translation task from a small number of input images, the network learns a generalizable appearance pattern extractor that can be applied to images of unseen categories at translation time to complete the few-shot image-to-image translation task.

[0016] Various other functionalities may be implemented within the various embodiments and are discussed and suggested elsewhere herein.

[0017] Figure 1An example image transformation 100 that can be performed according to various embodiments is shown. The transformation can accept an input image 106 (or a digital representation of an image) that includes a representation of one type of object, in this case, a dog breed corresponding to a pug. It may be desirable to generate images of the pug in different poses, such as to transform the representation of the pug from the pose of a target image 106 to a target pose, such as the pose in a selected or provided image 104. As shown, the image with the target pose includes a representation of a different type of object, in this example, a lion. The image transformation can take what it has learned from images of lions (and other such object categories) and use this learning to generate an image 108 of a pug exhibiting the target pose. Of interest in this example is the fact that the image transformation network has been trained using images 102 of many categories of object types, including many images of the lion object type. The image transformation network has not been previously trained or processed on any images that include representations of pugs.

[0018] Methods according to various embodiments may use one or more image-to-image conversion methods to perform such image generation or conversion as referred to herein. In various embodiments, unsupervised image-to-image conversion techniques may be used that learn to map images in a given category to similar images in a different category, drawn from an unstructured (i.e., unregistered) image dataset. While conventional methods can provide object pose conversions to images, these methods require access to many images of both the source category (e.g., exhibiting the target pose) and the destination category (including previously unseen objects to obtain the desired pose) during training or at the time of conversion. The need for a large number of input images of the desired object category significantly limits the practicality of these conventional methods. Drawing inspiration from the human ability to extract the essence of a new object from a small number of examples and generalize from there, methods according to various embodiments utilize a few-shot, unsupervised image-to-image conversion algorithm that operates on previously unseen target categories that are specified at the time of conversion by only a small number of example images, such as a single image of an object of the target type or a pair of input images. In some embodiments, the algorithm uses three or fewer examples, while in other embodiments, the algorithm can operate using fewer than five input images of the target category. In some embodiments, a single image may be used, but a second or third image of the target object class at a different pose may be used to increase accuracy or photorealism. The example model achieves this few-shot imagining capability by coupling an adversarial training scheme with a novel network design.

[0019] As mentioned earlier, the human brain is very good at generalization. For example, when given a picture of an animal of a type never seen before, the human brain can form a vivid mental picture of the never-before-seen animal in a variety of poses, especially when that person is exposed to similar images or views, but with the animal in those poses. For example, seeing a standing pug (e.g., Figure 1 A person viewing the example input image 106 will have little difficulty imagining what a pug would look like sitting on its hind legs (as shown in the example output image 108) or lying down, given a lifetime of experience with other animals in similar postures and how those animals look in those postures.

[0020] While some conventional unsupervised image-to-image translation algorithms provide reasonable results in transferring complex appearance variations across image categories, the ability to generalize from a small number of samples of new categories based on prior knowledge is not possible using these algorithms. Specifically, these algorithms require a large training set of images from all categories on which the translation is to be performed, and they do not support few-shot generalization.

[0021] Methods according to various embodiments attempt to bridge the gap between human and machine imagination using what is referred to herein as the Few-Shot Unsupervised Image-to-Image Translation ("FUNIT") framework. The FUNIT framework can learn an image-to-image translation model that maps an image of a source category to a similar image of a target category using a small number of images (i.e., two or fewer) of the target category given at translation time. During training or other processes, the translation model may have never encountered or processed images of the target category but is asked to generate some of these images at test time. This approach can be based on the idea that humans' ability to generate images of the target category develops from their past visual experience. For example, if a person has seen many different object categories in the past, they can better imagine the view of a new object. Using this approach, a FUNIT model can be trained using a dataset containing images of many different object categories to simulate these past visual experiences. Specifically, the model can be trained to translate images from one category to another using a small number of input images of the target category. By learning to extract appearance patterns from a small number of input images for the translation task, the model learns a generalizable appearance pattern extractor that can be applied to images of unseen categories at translation time for the few-shot image-to-image translation task. In at least some embodiments, the performance of this few-shot conversion method improves as the number of classes in the training set increases. For the purposes of this discussion, source classes will refer to object classes included in the training data, which includes the expected pose or other aspects of the class for objects of the source class. Target classes will refer to object classes that are not included in the training data and are only visible at test, conversion, or inference time.

[0022] The image conversion framework according to various embodiments can be based on a generative adversarial network (GAN). The use of an adversarial training scheme combined with a specific type of network design can achieve the desired few-sample unsupervised image-to-image conversion capability. Unsupervised and / or unpaired image-to-image conversion can attempt to learn a conditional image generation function that can map an input image of a source category to a similar image of a target category without paired supervision. This approach may not be enough because it attempts to restore the joint distribution using samples from marginal distributions. To solve this problem, conventional methods use additional constraints, such as enforcing the conversion to retain certain properties of the source data, such as pixel values, pixel gradients, semantic features, category labels, or pairwise sample distances. Other methods are also used.

[0023] Methods according to various embodiments utilize a partially shared latent space assumption designed for few-shot unsupervised image-to-image translation tasks. While capable of generating realistic translation output, conventional unsupervised image-to-image translation models are limited in at least two respects. First, these models suffer from poor sample efficiency, producing poor translation output when trained on only a small number of images. Second, the learned models are limited to translating images between two categories. A model trained for one translation task cannot be directly reused for a new task, despite similarities between the new and original tasks. For example, a model trained for husky-to-cat translation cannot be reused for husky-to-tiger translation, even though cats and tigers may share significant similarities. While some conventional methods utilize an unsupervised image-to-image translation framework to partially address the first aspect, methods according to various embodiments assume that there are many source category images but a small number of target category images. Furthermore, it can be assumed that the small number of target category images is only available at test or translation time and can come from many different object categories.

[0024] Multi-class unsupervised image-to-image translation extends unsupervised image-to-image translation methods to multiple categories. Methods according to various embodiments utilize a training dataset consisting of images from multiple categories, but instead of translating images between observed categories, they focus on translating images from these observed categories into similar images from previously unseen categories.

[0025] The FUNIT framework according to various embodiments can map the image of the source category to a similar image of the unseen target category by utilizing a small amount of target category images available when converting. Training data can be composed of unpaired images from a group of object categories (e.g., images of dogs and cats of various breeds), which are referred to as source categories in this article. These images can be used to train an unsupervised image-to-image conversion model that converts images between any two source categories. During testing, a small amount of images from a new object category (e.g., a mountain lion category) (referred to as the target category) can be provided to the model as input. The model can utilize a small amount of target images to convert any source category image into a similar image of the target category.

[0026] The FUNIT framework according to various embodiments may include a conditional image generator G and a multi-task adversarial discriminator D. Unlike the conditional image generator in the conventional unsupervised image-to-image translation framework (which takes one image as input), the generator according to various embodiments may simultaneously accept a content image x belonging to a first category and a set of images (y1, ..., y K ) as input. The output image can be generated by the following equation:

[0027]

[0028] The content image can be assumed to belong to the destination object category, while each of the K class images belongs to the source object category. The generator can map the input content image to an output image such that the output image has the appearance of an image belonging to the destination object category but with the pose of the source object category, where the source and destination objects share at least some structural similarity. During training, the generator can learn to translate images between two randomly sampled source categories. During translation, the generator can accept as input one or a small number of unseen images from the target category as images of the destination category and map images sampled from any source category to similar images of the target or destination category.

[0029] Figure 2 Components of an example FUNIT framework 200 that can be used in accordance with various embodiments are shown. As previously described, a training set can consist of images of various object categories or source categories. A model (such as a GAN) can be trained to convert images between any or all of these source object categories. In deployment, the trained model can receive as input a few images of the target category that are sufficient to convert the image of the source category into one or more similar images of the target category. An example FUNIT generator can accept as input a content image and one or more target category images. The generator can generate a transformation of the input content image that resembles or has the appearance of an image of the target category. Figure 2 As shown, the FUNIT generator can accept as input an image of a first object category and a set of images of a second object category, and can generate a transformed output. The transformed output should be similar to the images of the second object category and share the same class-invariant appearance as the input images.

[0030] The example generator G consists of four main components: a content encoder, a category encoder, an adaptive instance norm (AdaIN) decoder, and an image decoder. The instance norm and rectified linear unit (ReLU) can be applied to each convolutional layer and fully connected layer of the network. The content encoder can contain several convolutional layers followed by several residual blocks. The content encoder can map the input content image x to a spatially distributed feature map z x, referred to herein as the content latent code. The category encoder may include several convolutional layers followed by an average pooling layer. The category encoder may accept a category image as input and output a category latent code. The average pooling layer may first average activations across spatial dimensions (e.g., height and width) and then across the image set. The image decoder may include several AdaIN residual blocks followed by several upscaled convolutional layers. The AdaIN residual block may be a residual block that uses the AdaIN layer as a normalization layer. For each sample, the AdaIN layer (also called the normalization layer) may first normalize the activations in each channel to a distribution with zero mean and unit variance. The normalization layer may then transform the distribution to have specific mean and variance values ​​through a denormalization process. The primary goal of the image decoder is to decode the content latent code and the category latent code to generate a transformation of the input content image. In some embodiments, the AdaIN decoder is a multilayer perceptron. It decodes the category latent code into a set of mean and variance vectors, which are used as the new mean and variance for each channel in each AdaIN residual block in the image decoder. Using such a generator design, a content encoder can be used to extract a category-invariant latent representation (e.g., object pose), and a category encoder can be used to extract a category-specific latent representation (e.g., object appearance). By feeding the category latent code to the image decoder via the AdaIN layer, the category image can control the spatially invariant mean and variance, while the content image determines the remaining information. During training, the category encoder can learn to extract category-specific latent representations from images of the source category. During testing or transfer, this can generalize to images of previously unseen categories.

[0031] An example multi-task adversarial discriminator D is trained by solving multiple adversarial classification tasks simultaneously. In some embodiments, the discriminator is a patch-GAN discriminator that renders an output-space map of the input image, where each entry in the map indicates the score of the corresponding patch in the input image. In some embodiments, each task to be solved can be a binary classification task, determining whether the input image to D is a real image of the source category or a transformed output from the generator. Since there are many source categories, the discriminator can be designed to produce a corresponding number of outputs. When updating D for real images of the source category, D can be penalized if a certain output is negative. For transformed outputs that produce fake images of the source category, D can be penalized if the corresponding output is positive. D can be not penalized for not predicting a negative value for images of other categories. When updating the generator G, G is penalized only if a specific output of D is negative. In some embodiments, the discriminator D can be designed based on a class-conditional discriminator consisting of several residual blocks followed by a global average pooling layer. The features produced by the global average pooling layer are called discriminator features, from which a classification score can be generated using a linear mapping.

[0032] The example FUNIT framework can be trained by solving the minimax optimization problem given by:

[0033] minmaxL GAN (D,G)+λ R L R (G)+λ FM L FM +λ GP L GP (D)

[0034] DG

[0035] Among them L GAN , L R , L FM and L GP They are GAN loss, content reconstruction loss, feature matching loss and gradient penalty loss. R ,λ FM and λ GP The relative importance of the control terms is set to 0.1, 1, and 10. The GAN loss can be a conditional loss. The discriminator can be trained to predict values ​​such as "real" for real images and "fake" for conversion outputs, while the generator can be trained to make the discriminator predict the appropriate conversion output as true.

[0036] In some embodiments, FUNIT model training can be regularized using an image reconstruction loss. Specifically, when using the same image as the input content image and the input category image, the model should generate the same output image as the input. The feature matching loss L FM It can be used to further improve the performance of few-shot conversion. Specifically, it is called D f The feature extractor of D can be constructed by removing the last (i.e. prediction) layer from D. f It can be used to extract features from the conversion output and the class image and minimize a given L1 loss. In some embodiments, a zero-centered gradient penalty loss can be applied to real images to obtain more stable GAN training. This can involve penalizing the discriminator with a non-zero gradient field.

[0037] Figure 3An example environment 300 is shown that can be used to implement aspects of various embodiments. In some embodiments, a user can utilize a client device 602 to provide an input image, which can be an image including representations of objects not previously encountered by the image generator network 328, as discussed herein. The user can also utilize the client device to select an image that indicates a target pose for which a transformation is to be performed on an object in the input image. The client device can be any suitable computing device that enables a user to select and / or provide an image for processing, and can include, for example, a desktop computer, a laptop computer, a smartphone, a tablet computer, a computer workstation, a gaming console, etc. The user can select, provide, or otherwise specify transformation inputs to a user interface (UI) of an image editor application 306 (or other image processing or generation software package) running on the client device, although in some embodiments, at least some functionality can also be operated on a remote device, a networked device, or the "cloud." The user can provide input to the UI, for example, via a touch-sensitive display 304 or by moving a mouse cursor displayed on a display screen, etc. As described above, the user can provide an input image of a target category and can select an image that indicates a desired pose or other such aspect (which can be exhibited by an object in an image of a source category). The client device may include at least one processor 308 (e.g., a CPU or GPU) for executing applications and / or performing tasks on behalf of applications, and memory 310 including non-transitory computer-readable instructions for execution by the processor. Images provided to or generated by an application may be stored locally in local memory 312, such as a hard drive or flash memory, etc.

[0038] In some embodiments, an input image received or selected on client device 302 can be processed on the client device to generate an image having a desired transformation, such as applying the appearance of a target image to a gesture extracted from a set of source images. In other embodiments, client device 302 can send an input image, data extracted from an image, or data specifying an image via at least one network 314 to be received by a remote computing system, which can be part of a resource provider environment 316. At least one network 314 can include any suitable network, including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network or combination, and communication over the network can be achieved via wired and / or wireless connections. Provider environment 316 can include any suitable components for receiving requests and returning information or performing actions in response to those requests. For example, the provider environment can include a web server and / or application server for receiving and processing requests and then returning data or other content or information in response to the requests.

[0039] Communications received into provider environment 316 may be received into interface layer 318. Interface layer 318 may include an application programming interface (API) or other exposed interface that enables users to submit requests to the provider environment. Interface layer 318 in this example may also include other components, such as at least one web server, a routing component, a load balancer, etc. Components of interface layer 318 may determine the type of request or communication and direct the request to the appropriate system or service. For example, if the communication is for training an image conversion network for a category of image content (such as bird, animal, or person), the communication may be directed to image manager 320, which may be a system or service provided using various resources of provider environment 316. The communication or information from the communication may be directed to training manager 324, which may select an appropriate model or network and then train the model using the relevant training images and / or data 324. Once the network is trained and successfully evaluated, the network may be stored in model repository 326, which may, for example, store different models or networks for different types of image conversion or processing. If a request including input for an image transformation is received at the interface layer 318, the information of the request can be directed to the image generator 328 (also referred to herein as part of an image transformation network or service), which can obtain a corresponding trained network, such as a trained generative adversarial network (GAN) discussed herein, from the model repository 326 (if not already stored locally to the generator 328). The image generator 328 can take as input a target image (or a small number of images) and data indicating a target pose, which, as discussed herein, can be exhibited by a selected source image. The image generator 328 can then cause the input to be processed to generate an image representing the target transformation. As described above, this can involve processing the input by a content encoder 330 to extract a category-invariant latent representation, such as which can correspond to the selected pose. The category encoder 332 can extract a category-specific latent representation (e.g., object appearance) from the input target image. The class latent code can be fed to the AdaIN decoder 334, which decodes the class latent code into a set of mean and variance vectors, which are used as new means and variances for each channel in each AdaIN residual block in the image decoder 336. The generated image can then be sent to the client device 302 for display on the display element 304, or for other such uses. If the user wants to modify any aspect of the image, the user can provide additional input to the application 306, which can cause the same process to generate a new or updated image for the new or updated input, such as additional images of the target class or specifications of different poses, etc. In some embodiments, the image generation network can utilize a deep generative model that can learn to sample images given a training dataset.The models used may include, for example, generative adversarial networks (GANs) and variational autoencoder (VAE) networks, while targeting image conversion tasks. In various embodiments, the image conversion network or converter 340 may include a GAN consisting of a generator 328 and a discriminator 338. The generator 328 may be used to produce converted images such that the discriminator cannot distinguish between real and generated images.

[0040] In various embodiments, processor 308 (or the processor of training manager 322 or image converter 340) will be a central processing unit (CPU). However, as previously mentioned, resources in such environments can utilize GPUs to process data for at least some types of requests. GPUs, with their thousands of cores and designed to handle massively parallel workloads, have become popular in deep learning for training neural networks and generating predictions. While using GPUs for offline building enables faster training of larger and more complex models, generating predictions offline means that request-time input features cannot be used, or predictions must be generated for all permutations of features and stored in lookup tables to service real-time requests. If the deep learning framework supports CPU mode, and the model is small and simple enough that feedforward can be executed on the CPU with reasonable latency, a service on a CPU instance can host the model. In this case, training can be done offline on the GPU, and inference is done in real time on the CPU. If a CPU approach is not a viable option, the service can run on a GPU instance. However, because GPUs have different performance and cost characteristics than CPUs, running a service that offloads runtime algorithms to GPUs may require designing them differently than a CPU-based service.

[0041] Figure 4An example process 400 is shown for generating an image that can be utilized according to various embodiments, representing the transformation of a representation of a previously unseen object to have a pose learned from a known source object category. It should be understood that for this and other processes discussed herein, unless otherwise noted, additional, alternative, or fewer steps may be present, performed in a similar or alternative order, or in parallel, within the scope of various embodiments. Furthermore, while pose is used as the primary example, other aspects of source image categories may be present for generating the transformation of an input object as discussed and suggested herein. In this example, an input digital image is received 402, or otherwise obtained or specified, that includes a representation of an object of interest. In this example, the object belongs to a source category that a neural network has not encountered for processing images, or at least to a category that was not used to train the network. A selection of a source image may also be obtained 404, where the source image includes a representation of a source object of the object category used to train the neural network. In some embodiments, a user may select the source object, while in other embodiments, the system or network may select the source image or pose, as well as other options discussed and suggested herein.

[0042] Once obtained, the input image and the source image (or an indication of the source image) can be provided 406 as input to a trained neural network. As discussed herein, the network can be trained using many images of multiple object categories (including the source category) to learn to transform between objects of each category. One or more encoders of the network's generator can extract 408 a category-invariant latent representation corresponding to the target pose from the source image or category. Category-specific latent representations corresponding to the appearance of the target object can also be extracted 410 from the input image. The neural network can then be used to infer 412 a transformed image including a representation of an object of the target category with the target pose based at least in part on these latent representations. As previously discussed, such a process can produce high-quality transformed images without requiring a large number of input images of the target category.

[0043] As described above, various embodiments utilize machine learning. For example, deep neural networks (DNNs) developed on processors have been used in a variety of use cases, from self-driving cars to faster drug development, from automatic image annotation in online image databases to intelligent real-time speech conversion in video chat applications. Deep learning is a technology that mimics the neural learning process of the human brain, constantly learning, getting smarter, and providing more accurate results faster over time. Children are initially taught by adults how to correctly identify and classify various shapes, and eventually are able to recognize shapes without any guidance. Similarly, a deep learning or neural learning system needs to be trained in object recognition and classification as it becomes smarter and more efficient at recognizing basic objects, occluded objects, etc., while also assigning context to objects.

[0044] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for action. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a specific weight based on its importance in defining the object's shape.

[0045] Deep neural network (DNN) models consist of multiple layers of connected perceptrons (e.g., nodes) that can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns such as lines and angles. The second layer assembles the lines to look for higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the vehicle class, and the final layers generate labels for the input image, identifying the model of a specific car brand. Once trained, the DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten digits on checks deposited at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying cars, pedestrians, and road hazards in different types of self-driving cars, or translating human speech in real time.

[0046] During training, data flows through the DNN in a forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct and predicted labels is analyzed, and the weights of each feature are adjusted in a backpropagation phase until the DNN correctly labels the input and other inputs in the dataset during training. Training complex neural networks requires a large amount of parallel computing performance, including support for floating-point multiplication and addition. Inference, which is less computationally intensive than training, is a latency-sensitive process in which a trained neural network is applied to new inputs it has never seen before to classify images, convert speech, and generally infer new information.

[0047] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, computing platforms can deliver the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0048] Figure 5An example system 500 is shown that can be used to classify data or generate inferences according to various embodiments. Based on the teachings and suggestions contained herein, it should be apparent that various predictions, labels, or other outputs can also be generated for input data. Furthermore, both supervised and unsupervised training can be used in various embodiments discussed herein. In this example, a set of classified data 502 is provided as input to serve as training data. The classified data can include instances of at least one type of object for which a statistical model is to be trained, as well as information identifying objects of that type. For example, the classified data might include a set of images, each containing a representation of an object type, where each image also contains or is associated with a label, metadata, classification, or other information identifying the type of object represented in the respective image. Various other types of data can also be used as training data, including text data, audio data, video data, and the like. In this example, the classified data 502 is provided as training input to a training manager 504. The training manager 504 can be a system or service comprising hardware and software, such as one or more computing devices executing a training application, for training a statistical model. In this example, the training manager 504 receives an instruction or request indicating the type of model to be trained. The model can be any suitable statistical model, network, or algorithm that can be used for such purposes, and may include, for example, artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, etc. The training manager 504 can select a base model or other untrained model from an appropriate repository 506 and train the model using the classified data 502 to generate a trained model 508 that can be used to classify similar types of data. In some embodiments that do not use classified data, the training manager can still select an appropriate base model to train the input data.

[0049] Models can be trained in a variety of different ways, which may depend in part on the type of model selected. For example, in one embodiment, a set of training data can be provided to a machine learning algorithm, where the model is a model artifact created by the training process. Each instance of the training data contains a correct answer (e.g., a classification), which can be referred to as a target or target attribute. The learning algorithm finds patterns in the training data that map the input data attributes to the target, the answer to be predicted, and outputs a machine learning model that captures these patterns. The machine learning model can then be used to obtain predictions for new data for which the target is not specified.

[0050] In one example, the training manager can select from a set of machine learning models, including binary classification, multi-class classification, and regression models. The type of model to use can depend at least in part on the type of target to be predicted. Machine learning models for binary classification problems predict a binary outcome, such as one of two possible classes. Learning algorithms (such as logistic regression) can be used to train binary classification models. Machine learning models for multi-class classification problems allow predictions to be generated for multiple classes, such as predicting one of more than two outcomes. Multinomial logistic regression can be useful for training multi-class models. Machine learning models for regression problems predict numerical values. Linear regression is useful for training regression models.

[0051] In order to train a machine learning model according to one embodiment, a training manager must determine the input training data source and other information, such as the name of the data attribute containing the target to be predicted, the required data transformation instructions, and training parameters to control the learning algorithm. During the training process, in some embodiments the training manager can automatically select an appropriate learning algorithm based on the target type specified in the training data source. The machine learning algorithm can accept parameters that are used to control certain properties of the training process and the resulting machine learning model. These are referred to as training parameters in this article. If no training parameters are specified, the training manager can utilize known default values ​​that work well for a wide range of machine learning tasks. Examples of training parameters for which values ​​can be specified include the maximum model size, the maximum number of passes over the training data, the shuffle type, the regularization type, the learning rate, and the amount of regularization. Default settings can be specified, with options for adjusting the values ​​to fine-tune performance.

[0052] The maximum model size is the total size (in bytes) of the patterns created during model training. By default, a model of a specified size, such as a 100MB model, is created. If the training manager cannot determine that there are enough patterns to fill the model size, a smaller model may be created. If the training manager finds that there are more patterns than can be accommodated within the specified size, it may enforce a maximum cutoff by trimming the patterns that have the least impact on the quality of the learned model. Choosing the model size allows you to control the tradeoff between the model's predictive quality and cost. Smaller models may cause the training manager to remove many patterns to fit within the maximum size limit, affecting prediction quality. On the other hand, larger models may be more expensive to query for real-time predictions. Larger input datasets do not necessarily result in larger models because the model stores the patterns, not the input data. If the patterns are few and simple, the resulting model will be smaller. Input data with a large number of raw attributes (input columns) or derived features (output of data transformations) may find and store more patterns during training.

[0053] In some embodiments, the training manager may make multiple passes or iterations over the training data to discover patterns. There may be a default number of passes, such as ten, and in some embodiments, a maximum number of passes may be set, such as up to one hundred passes. In some embodiments, there may not be a maximum set, or there may be a convergence criterion or other set of criteria that triggers the end of the training process. In some embodiments, the training manager may monitor the quality of the pattern (i.e., model convergence) during training and may automatically stop training when there are no more data points or patterns to discover. Data sets with only a small number of observations may require more data passes to achieve higher model quality. Larger data sets may contain many similar data points, which may reduce the need for a large number of passes. The potential impact of choosing more data passes over the data is that model training may take longer and cost more in terms of resources and system utilization.

[0054] In some embodiments, the training data is shuffled before training or between training passes. In many embodiments, the shuffling is a random or pseudo-random shuffle to produce a truly random ordering, although there may be constraints to ensure that certain types of data are not grouped, or if such grouping occurs, the shuffled data can be reshuffled. Shuffling changes the sequence or arrangement of the data used for training so that the training algorithm does not encounter groupings of similar types of data or too many consecutive observations of a single type of data. For example, a model may be trained to predict product type, where the training data includes movie, toy, and video game product types. Before uploading, the data may be sorted by product type. The algorithm can then process the data in alphabetical order by product type, initially seeing only data for one type (such as movies). The model will begin to learn patterns for movies. Later, the model will only encounter data for a different product type (e.g., toys) and will attempt to adjust the model to fit the toy product type, potentially degrading patterns that are appropriate for movies. This abrupt switch from movie to toy type may result in a model that fails to learn how to accurately predict product type. Before partitioning the training data set into the training subset and the evaluation subset, in some embodiments, shuffling can be performed to utilize a relatively uniform distribution of data types for both phases. In some embodiments, the training manager can automatically shuffle the data using, for example, a pseudo-random shuffling technique.

[0055] In some embodiments, when creating a machine learning model, a training manager can enable a user to specify settings or apply customization options. For example, a user can specify one or more evaluation settings to indicate a portion of the input data to be retained for evaluating the predictive quality of the machine learning model. A user can specify a method that indicates which attributes and attribute transformations to use for model training. The user can also specify various training parameters that control the training process and certain properties of the resulting model.

[0056] Once the training manager determines that the model training is complete, for example by using at least one of the final criteria discussed herein, the trained model 508 can be provided to the classifier 514 for use in classifying unclassified data 512. However, in many embodiments, the trained model 508 will first be passed to an evaluator 510, which can include an application or process executed on at least one computing resource for evaluating the quality (or other aspects) of the trained model. The model is evaluated to determine whether it will provide at least a minimum acceptable or threshold level of performance when predicting targets for new and future data. Since future data instances will typically have unknown target values, it may be desirable to examine the accuracy metric of the machine learning on data for which the target answer is known and use this evaluation as a proxy for the prediction accuracy of future data.

[0057] In some embodiments, a subset of the classified data 502 provided for training is used to evaluate the model. This subset can be determined using the shuffling and splitting methods described above. This evaluation data subset will be labeled with a target and can therefore serve as a resource for evaluating ground truth. It is useless to use the same data used for training to evaluate the prediction accuracy of a machine learning model because a model that memorizes the training data rather than generalizing it may produce a positive evaluation. Once training is complete, the trained model 508 is used to process the evaluation data subset, and the evaluator 510 can determine the accuracy of the model by comparing the ground truth data with the corresponding output (or prediction / observation) of the model. In some embodiments, the evaluator 510 can provide a summary or performance metric that indicates the degree of match between the predicted value and the true value. If the trained model does not meet at least the minimum performance standard or other such accuracy threshold, the training manager 504 can be instructed to perform further training, or in some cases, try training a new or different model, etc. If the trained model 508 meets the relevant standards, the trained model can be provided for use by the classifier 514.

[0058] When creating and training a machine learning model, in at least some embodiments, it may be desirable to specify model settings or training parameters that will result in a model that makes the most accurate predictions. Example parameters include the number of forward and / or backward passes to be performed, regularization, model size, and shuffling type. However, as described above, selecting the model parameter settings that produce the best predictive performance on the evaluation data may result in overfitting the model. Overfitting occurs when the model memorizes patterns present in both the training and evaluation data sources but fails to generalize to patterns in the data. Overfitting often occurs when the training data includes all the data used in the evaluation. An overfitted model may perform well during evaluation but may not make accurate predictions on new or otherwise classified data. To avoid selecting an overfitted model as the best model, the training manager may retain additional data to validate the model's performance. For example, the training dataset may be divided into 60% for training and 40% for evaluation or validation, which may be divided into two or more phases. After selecting the model parameters that best fit the evaluation data, resulting in convergence on a subset of the validation data (e.g., half of the validation data), a second validation run can be performed using the remaining validation data to ensure the model's performance. If the model meets expectations on the validation data, then the model is not overfitting the data. Optionally, a test or holdout set can be used to test parameters. Using a second validation or testing step can help choose appropriate model parameters to prevent overfitting. However, taking more data out of the training process for validation results in less data available for training. This can be problematic for smaller datasets, as there may not be enough data available for training. One approach in this situation is to perform cross-validation, as described elsewhere in this article.

[0059] There are many metrics or insights that can be used to review and evaluate the predictive accuracy of a given model. A sample evaluation result includes a predictive accuracy metric to report the overall success of the model, as well as visualizations that help explore the model's accuracy beyond the predictive accuracy metric. The results can also provide the ability to view the impact of setting score thresholds (such as for binary classification) and can generate alerts based on the criteria used to check the effectiveness of the evaluation. The choice of metric and visualization may depend at least in part on the type of model being evaluated.

[0060] After satisfactory training and evaluation, the trained machine learning model can be used to build or support machine learning applications. In one embodiment, building a machine learning application is an iterative process involving a series of steps. The core machine learning problem can be constructed based on what is observed and the answer that the model is to predict. Data can then be collected, cleaned, and prepared to make it suitable for use by the machine learning model training algorithm. This data can be visualized and analyzed for integrity checks to verify data quality and understand the data. This may be the case where the original data (such as input variables) and answers (such as targets) are not represented in a way that can be used to train a highly predictive model. Therefore, it may be desirable to build more predictive input representations or features from the original variables. The resulting features can be input into the learning algorithm to build the model and evaluate the quality of the model based on the data retained from model construction. The model can then be used to generate predictions of the target answer for new data instances.

[0061] exist Figure 5 In the exemplary system 500 of , after providing the evaluation, the trained model 510 is provided to or made available to a classifier 514, which is capable of using the trained model to process unclassified data. For example, this may include data received from a user or unclassified third party, such as query images that are being queried for information about the content represented in those images. The unclassified data can be processed by the classifier using the trained model, and the generated results 516 (i.e., classifications or predictions) can be sent back to the corresponding source or otherwise processed or stored. In some embodiments, and where such use is permitted, these now-classified data instances can be stored in a classified data repository, which can be used by a training manager for further training of the trained model 508. In some embodiments, the models are continuously trained as new data becomes available, but in other embodiments, the models are trained periodically, such as daily or weekly, depending on factors such as the size of the dataset or the complexity of the model.

[0062] The classifier may include appropriate hardware and software for processing unclassified data using a trained model. In some cases, the classifier will include one or more computer servers, each having one or more graphics processing units (GPUs) capable of processing data. The configuration and design of GPUs may make them more suitable for processing machine learning data than CPUs or other such components. In some embodiments, the trained model can be loaded into GPU memory and the received data instances can be provided to the GPU for processing. A GPU can have many more cores than a CPU, and a GPU core can be less complex. Therefore, a given GPU may be able to process thousands of data instances simultaneously through different hardware threads. The GPU can also be configured to maximize floating-point throughput, which can provide significant additional processing advantages for large data sets.

[0063] Even when GPUs, accelerators, and other such hardware are used to accelerate tasks such as model training or data classification using such models, such tasks can still require significant time, resource allocation, and cost. For example, if a machine learning model is to be trained using 100 passes, and the dataset includes 1,000,000 data instances to be used for training, each pass will need to process all million instances. Different parts of the architecture can also be supported by different types of devices. For example, training can be performed using a set of servers at a logically centralized location, such as can be provided as a service, while classification of the raw data can be performed by such a service or on client devices, among other such options. In various embodiments, these devices can also be owned, operated, or controlled by the same entity or multiple entities.

[0064] Figure 6An example neural network 600 or other statistical model that can be utilized according to various embodiments is shown. In this example, the statistical model is an artificial neural network (ANN) that includes multiple layers of nodes, including an input layer 602, an output layer 606, and multiple layers 604 of intermediate nodes, commonly referred to as "hidden" layers because the internal layers and nodes are typically not visible or accessible in conventional neural networks. As discussed elsewhere herein, other types of statistical models, as well as other types of neural networks including other numbers or selections of nodes and layers, can also be used. In this network, all nodes in a given layer are interconnected to all nodes in adjacent layers. As shown, the nodes in the intermediate layers are then connected to the nodes of the two adjacent layers respectively. In some models, nodes are also called neurons or connected units, and the connections between nodes are called edges. Each node can perform a function for the input it receives, for example, by using a specified function. Nodes and edges can be given different weights during the training process, and each layer of nodes can perform specific types of transformations on the input it receives, and these transformations can also be learned or adjusted during the training process. Learning can be supervised or unsupervised, which may depend at least in part on the type of information contained in the training data set. Various types of neural networks can be utilized, including, for example, convolutional neural networks (CNNs), which include many convolutional layers and a set of pooling layers and have proven beneficial for applications such as image recognition. CNNs are also easier to train than other networks due to the relatively small number of parameters to be determined.

[0065] In some embodiments, various tuning parameters can be used to train such complex machine learning models. Selecting parameters, fitting the model, and evaluating the model are part of the model tuning process, often referred to as hyperparameter optimization. In at least some embodiments, such tuning can include introspection of the underlying model or data. In training or production settings, a robust workflow is important to avoid overfitting of hyperparameters, as described elsewhere herein. Cross-validation and adding Gaussian noise to the training dataset are useful techniques to avoid overfitting to any one dataset. For hyperparameter optimization, in some embodiments, it may be necessary to keep the training set and validation set fixed. In some embodiments, hyperparameters can be tuned in certain categories, which can include, for example, data preprocessing (in other words, converting words to vectors), CNN architecture definition (e.g., filter size, number of filters), stochastic gradient descent parameters (e.g., learning rate), regularization (e.g., dropout probability), and other such options.

[0066] In the example preprocessing step, instances in the dataset can be embedded into a lower-dimensional space of a specific size. The size of this space is a parameter to be adjusted. The CNN architecture contains many adjustable parameters. The filter size parameter can represent the interpretation of information corresponding to the size of the instance to be analyzed. In computational linguistics, this is called the n-gram size. The example CNN uses three different filter sizes, which represent different possible n-gram sizes. The number of filters for each filter size can correspond to the depth of the filter. Each filter attempts to learn something different from the instance structure, such as the sentence structure of text data. In the convolutional layer, the activation function can be a rectified linear unit, and the pooling type is set to max pooling. The results can then be concatenated into a one-dimensional vector, and the final layer is fully concatenated to the two-dimensional output. This corresponds to binary classification, to which an optimization function can be applied. One such function is an implementation of the root mean square (RMS) propagation method of gradient descent, where example hyperparameters can include the learning rate, batch size, maximum gradient normal, and epochs. For neural networks, regularization can be a very important consideration. As mentioned, in some embodiments, the input data can be relatively sparse. In this case, the main hyperparameters can be dropped at the penultimate layer, which means that a certain proportion of nodes will not "fire" in each training cycle. The example training process can recommend different hyperparameter configurations based on feedback on the performance of the previous configuration. The model can be trained using the recommended configuration, evaluated on a specified validation set, and performance reported. This process can be repeated, such as to balance exploration (learning more about different configurations) and development (leveraging previous knowledge to achieve better results).

[0067] Because CNN training can be parallelized and utilize GPU-powered computing resources, multiple optimization strategies can be tried for different scenarios. Complex scenarios allow for tuning of the model architecture, preprocessing, and stochastic gradient descent parameters. This expands the model configuration space. In the basic scenario, only preprocessing and stochastic gradient descent parameters are tuned. Compared to the basic scenario, complex scenarios allow for more configuration parameters. Tuning of the joint space can be performed using linear or exponential steps and iterated through the model's optimization loop. This tuning process can be significantly less expensive than tuning procedures such as random search and grid search, without any noticeable performance loss.

[0068] Some embodiments may use back propagation to calculate the gradients for determining the weights of a neural network. Back propagation is a form of differentiation, and as described above, a gradient descent optimization algorithm may be used to adjust the weights applied to various nodes or neurons. In some embodiments, the gradient of the associated loss function may be used to determine the weights. Back propagation may utilize the derivative of the output generated by the loss function to the statistical model. As described above, each node may have an associated activation function that defines the output of each node. Various activation functions may be used appropriately, such as radial basis functions (RBFs) and sigmoid functions, which may be used by various support vector machines (SVMs) for data conversion. The activation function of the intermediate layer of a node is referred to herein as an inner product core. These functions may include, for example, a recognition function, a step function, a sigmoid function, a ramp function, and the like. The activation function may also be linear or nonlinear, as well as other such options.

[0069] Figure 7 A basic set of components of a computing device 700 is shown that can be used to implement aspects of various embodiments. In this example, the device includes at least one processor 702 for executing instructions that may be stored in a memory device or element 704. It will be apparent to one of ordinary skill in the art that the device may include many types of memory, data storage, or computer-readable media, such as a first data storage for program instructions executed by the processor 702, the same or separate storage for images or data, removable memory for sharing information with other devices, and any number of communication methods for sharing with other devices. The device will typically include some type of display element 706, such as a touch screen, organic light emitting diode (OLED), or liquid crystal display (LCD), but the device (such as a portable media player) may convey information via other means, such as through audio speakers. As discussed, the device in many embodiments will include at least a communication component 708 and / or a network component 710, such as may support wired or wireless communication over at least one network, such as the Internet, a local area network (LAN), Or cellular network, etc. These components can enable the device to communicate with remote systems or services. The device may also include at least one additional input device 712 that can receive conventional input from the user. The conventional input may include, for example, buttons, touch pads, touch screens, roller devices, joysticks, keyboards, mice, tracking balls, keypads or any other such devices or elements, through which the user enters commands to the device. In some embodiments, these I / O devices may even be connected via infrared or Bluetooth or other links. However, in some implementations, such devices may not include any buttons at all and may only be controlled by a combination of visual and audio commands, so that the user can control the device without having to make contact with the device.

[0070] Various embodiments can be implemented in a variety of operating environments, which in some cases can include one or more user computers or computing devices that can be used to operate any of a plurality of applications. User or client devices can include any of a variety of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting a variety of network and messaging protocols. Such systems can also include multiple workstations running any of a variety of commercial operating systems and other known applications for purposes such as development and database management. These devices can also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and other devices capable of communicating over a network.

[0071] Most embodiments utilize at least one network to support communications using any of a variety of commercially available protocols familiar to those skilled in the art, such as TCP / IP or FTP. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. In embodiments utilizing a network server, the network server can run any of a variety of server or middle-tier applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. One or more servers can also execute programs or scripts in response to requests from user devices, such as by executing one or more web applications, which can be implemented in any programming language, such as JavaScript. C, C# or C++, or any scripting language such as Python, and combinations thereof). The one or more servers may also include a database server, including but not limited to and Those servers that are commercially available.

[0072] The environment can include various data storage and other memories and storage media as described above. These can reside in various locations, such as on a storage medium local to one or more computers (and / or residing therein), or away from any or all computers on the network. In a specific set of embodiments, information can reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to a computer, server or other network device can be appropriately stored locally and / or remotely. In the case where the system includes a computerized device, each such device can include hardware elements that can be electrically coupled via a bus, including at least one central processing unit (CPU), at least one input device (such as a mouse, keyboard, controller, touch-sensitive display element or keypad) and at least one output device (such as a display device, printer or loudspeaker). Such a system can also include one or more storage devices, such as disk drives, optical storage devices and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), and removable media devices, memory cards, flash memory cards, etc.

[0073] Such a device may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device), and a working memory as described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium representing a remote, local, fixed, and / or removable storage device and a storage medium for temporarily and / or more permanently containing, storing, sending, and retrieving computer-readable information. The system and various devices will typically also include a plurality of software applications, modules, services, or other elements located within at least one working memory device, including an operating system and applications, such as a client application or a web browser. It should be understood that alternative embodiments may have many variations different from the above-described embodiments. For example, customized hardware and / or specific elements that can be implemented in hardware, software (including portable software, such as applets), or both may also be used. In addition, connections to other computing devices (such as network input / output devices) may be employed.

[0074] Storage media and other non-transitory computer-readable media for containing code or code portions may include any appropriate media known or used in the art, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by system devices. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will understand other ways and / or methods of implementing various embodiments.

[0075] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

Claims

1. A computer-implemented method comprising: receiving an input image including a representation of a target object of unknown type and having a first pose; receiving an indication of a source image, the source image comprising a representation of a source object of a source category, the source object of the source image having a second pose; providing the input image and the source image to a neural network trained using images of a plurality of object categories including the source category to extract a category-invariant representation of the second pose and a category-specific latent representation of the appearance of the target object; as well as An output image is inferred representing the target object in the second pose.

2. The computer-implemented method of claim 1 , wherein the neural network is a generative adversarial network (GAN) comprising a conditional image generator and a multi-task adversarial discriminator.

3. The computer-implemented method of claim 2, further comprising: extracting a class-invariant representation of the second pose using a content encoder of the conditional image generator; as well as A category encoder of the conditional image generator is used to extract a category-specific latent representation of the appearance of the target object.

4. The computer-implemented method of claim 2, further comprising: The layer activations are normalized to zero mean and unit variance distribution by a normalization layer of the multi-task adversarial discriminator; as well as Denormalize the normalized layer activations using an affine transformation.

5. The computer-implemented method of claim 1 , further comprising: Adversarial training of the neural network is performed to transfer poses between the plurality of object categories.

6. A computer-implemented method comprising: receiving a digital representation of an image including an object in a first pose; as well as An output image representing the object in a second pose is inferred using a neural network, wherein the neural network includes a conditional generator and an adversarial discriminator for extracting a class-invariant representation of the second pose and a class-specific representation of the appearance of the object.

7. The computer-implemented method of claim 6, wherein the neural network is not trained using images of an object class of the object.

8. The computer-implemented method of claim 6, further comprising: Unsupervised training of the neural network is performed using a training dataset comprising a plurality of images of a plurality of object categories, wherein the neural network learns to convert poses between the object categories.

9. The computer-implemented method of claim 6, wherein the neural network is a generative adversarial network (GAN).

10. A system comprising: at least one processor; as well as a memory comprising instructions that, when executed by the at least one processor, cause the system to: receiving a digital representation of an image including an object in a first pose; and An output image representing the object at a second pose is inferred using a neural network, wherein the neural network includes a conditional generator and an adversarial discriminator for extracting a class-invariant representation of the second pose and a class-specific representation of the appearance of the object.

Images