Latent variable generative model with noise contrastive prior

Abstract

Latent variable generative models with noise contrastive priors are disclosed, one embodiment sets forth techniques for generating images (or other generative outputs). The techniques include determining one or more first values of a set of visual attributes included in a plurality of training images, where the set of visual attributes are encoded via a prior network. The techniques also include applying a reweighting factor to the first values to generate one or more second values of the set of visual attributes, where the second values represent the first values shifted toward one or more third values of the set of visual attributes, where the one or more third values have been generated via an encoder network. The techniques also include performing one or more decoding operations on the second values via a decoder network to generate new images not included in the plurality of training images.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 083,635, filed September 25, 2020, entitled “VARIATIONAL AUTOENCODERS WITH NOISE CONTRASTIVE PRIORS.” The subject matter of this related application is hereby incorporated by reference herein. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate generally to machine learning and computer science, and more specifically, to latent variable generative models with noise contrastive priors. BACKGROUND

[0004] In machine learning, generative models generally include deep neural networks and / or other types of machine learning models that are trained to generate new data instances. For example, a generative model can be trained on a training dataset that includes a large number of images of cats. During training, the generative model “learns” visual attributes of the various cats depicted in the images. The generative model can then use these learned visual attributes to produce new images of cats that are not found in the training dataset.

[0005] A variational autoencoder (VAE) is a type of generative model. A VAE generally includes an encoder network that is trained to convert data points in a training dataset into values for “latent variables,” where each latent variable represents an attribute of the data points in the training dataset. The VAE also includes a prior network that is trained to learn a distribution of the latent variables associated with the training dataset, where the distribution of the latent variables represents variations and occurrences of different attributes in the training dataset. The VAE also includes a decoder network that is trained to convert the latent variable values generated by the encoder network back into data points that are substantially the same as the data points in the training dataset. After training is complete, a trained VAE can be used to generate new data that is similar to the data in the original training dataset by sampling latent variable values from the distribution learned by the prior network during training and converting these sampled values into new data points via the decoder network. Each new data point generated in this way can include attributes that are similar (but not identical) to one or more attributes of the data points in the training dataset.

[0006] For example, a VAE can be trained on a training dataset that includes images of cats, where each image includes tens of thousands to millions of pixels. The trained VAE will include an encoder network that converts each image to hundreds or thousands of numerical latent variable values. Each latent variable will represent a respective visual attribute found in one or more images used to train the VAE (e.g., the appearance of a cat’s face, fur, body, expression, pose, etc. in the image). The prior network will capture the variations and occurrences of visual attributes across all images in the training dataset as respective distributions of the latent variables (e.g., as mean, standard deviation, and / or other summary statistics associated with the numerical latent variable values). After training is complete, the trained VAE can be used to generate additional images of cats by sampling latent variable values that fall within the latent variable distributions learned by the prior network and converting those sampled latent variable values to new pixel values in additional cat images via the decoder network.

[0007] One drawback of using VAEs to generate new data is known as the “prior hole problem,” in which high probabilities are assigned by the prior network to regions of latent variable values that do not correspond to any actual data in the given training dataset. These regions of false high probabilities often arise from limitations in the complexity or “expressiveness” of the distributions of latent variable values that can be learned by the decoder in the VAE. Moreover, because these regions do not reflect the attributes of any actual data points in the training dataset, when the decoder network in the VAE converts samples from these regions to new data points, these new data points are often dissimilar to the data in the training dataset.

[0008] Continuing the above example, during training, the encoder in the VAE can convert the training dataset of cat images to latent variable values that occupy a first set of regions. In turn, the distribution of latent variables learned by the prior network from the training dataset can include high probabilities for this first set of regions, which reflects the fact that the latent variable values within the first set of regions correspond to actual training data. However, the distribution learned by the prior network can also include high probabilities for a second set of regions that do not include any latent variable values generated by the encoder from the training dataset. In this case, the high probabilities for this second set of regions are false and falsely suggest that the second set of regions includes latent variable values that correspond to attributes of actual training data. As noted above, in such cases, the distribution learned by the prior network does not match the actual distribution of latent variables produced by the encoder network from the training dataset because the distribution learned by the prior network is simpler or less “expressive” than the actual distribution produced by the encoder network. Thus, if a latent variable value that falls within the second set of regions in the distribution of latent variables learned by the prior network is sampled and converted to new pixel values by the decoder network in the VAE, the resulting image will not resemble a cat.

[0009] One approach to address the mismatch between the latent variable distribution learned by the prior network and the actual distribution of latent variables generated by the encoder network from the training dataset is to train the machine learning model using energy-based model training that employs iterative Markov Chain Monte Carlo (MCMC) sampling techniques to learn a more complex, or more "expressive," latent variable distribution to represent the training dataset. However, each MCMC sampling step depends on the result of the previous sampling step, which prevents parallel execution of the MCMC sampling. Sequentially executing different MCMC steps is both computationally inefficient and time consuming.

[0010] As noted above, there is a need in the art for more efficient techniques for generating new data using variational autoencoders. SUMMARY

[0011] One embodiment of the present invention sets forth techniques for improving generative outputs produced by a generative model. The techniques include sampling one or more first values from a distribution of a set of latent variables learned by a prior network included in the generative model. The techniques also include applying a reweighting factor to the one or more first values in order to generate one or more second values of the set of latent variables, where the reweighting factor is determined based on one or more classifiers that are used to distinguish between values sampled from the prior distribution and values of the set of latent variables generated via an encoder network included in the generative model. The techniques further include performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to produce the generative output

[0012] At least one technical advantage of the disclosed techniques, as compared to the prior art, is that the generative outputs produced by the disclosed techniques look more realistic, more similar to the data in the training dataset, as compared to outputs typically produced using conventional variational autoencoders. Another technical advantage is that, with the disclosed techniques, a complex distribution of latent variables produced by an encoder from a training dataset can be approximated by a machine learning model that is trained and executed in a more computationally efficient manner relative to the prior art. These technical advantages provide one or more technical improvements over the prior art approaches. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order that the foregoing aspects and features of various embodiments can be understood in detail, a more particular description will be rendered by reference to various embodiments (some of which are illustrated in the appended drawings) wherein like reference characters will be used to represent like features. It should be noted that the drawings are not necessarily to scale and that the embodiments can be used for a variety of purposes other than the specific applications illustrated.

[0014] Figure 1A computing device configured to implement one or more aspects of the various embodiments is shown.

[0015] Figure 2 An example residual unit included in an encoder of a hierarchical version of the VAE according to various embodiments. Figure 1 More detailed descriptions of the training engine and the execution engine of the

[0016] Figure 3A An example residual unit included in a generative model of a hierarchical version of the VAE according to various embodiments. Figure 2 An example architecture of an encoder included in a hierarchical version of the VAE according to various embodiments.

[0017] Figure 3B An example architecture of a generative model included in a hierarchical version of the VAE according to various embodiments. Figure 2 An example residual unit included in a generative model of a hierarchical version of the VAE according to various embodiments.

[0018] Figure 4A An example residual unit included in an encoder of a hierarchical version of the VAE according to various embodiments. Figure 2 An example residual unit included in a generative model of a hierarchical version of the VAE according to various embodiments.

[0019] Figure 4B An example residual unit included in a generative model of a hierarchical version of the VAE according to various embodiments. Figure 2 An example residual unit included in a classifier usable with a hierarchical version of the VAE according to various embodiments.

[0020] Figure 5A An example residual block included in a classifier usable with a hierarchical version of the VAE according to other various embodiments. Figure 2 An example architecture of a classifier usable with a hierarchical version of the VAE according to other various embodiments.

[0021] Figure 5B An example architecture of a classifier usable with a hierarchical version of the VAE according to other various embodiments. Figure 2 An example architecture of a classifier usable with a hierarchical version of the VAE according to other various embodiments.

[0022] Figure 5C An example architecture of a classifier usable with a hierarchical version of the VAE according to other various embodiments. Figure 2 An example architecture of a classifier usable with a hierarchical version of the VAE according to other various embodiments.

[0023] Figure 6 A flowchart of method steps for training a generative model according to various embodiments.

[0024] Figure 7 A flowchart of method steps for producing a generative output according to various embodiments.

[0025] Figure 8 A game streaming system configured to implement one or more aspects of the various embodiments is shown. DETAILED DESCRIPTION

[0026] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one of skill in the art that the inventive concept can practice without one or more of these specific details.

[0027] SUMMARY

[0028] A variational autoencoder (VAE) is a machine learning model trained to generate new instances of data after“learning” properties of data found in a training dataset. For example, a VAE can be trained on a dataset including images of cats. During training of the VAE, the VAE learns patterns of faces, fur, bodies, expressions, poses, and / or other visual properties of cats in the images. These learned patterns allow the VAE to generate new images of cats not found in the training dataset.

[0029] A VAE includes many neural networks. These neural networks can include an encoder network trained to convert data points in the training dataset to values of“latent variables,” where each latent variable represents a property of data points in the training dataset. These neural networks can also include a prior network trained to learn a distribution of latent variables associated with the training dataset, where the distribution of latent variables represents variations and occurrences of different properties in the training dataset. These neural networks can also include a decoder network trained to convert latent variable values generated by the encoder network back to data points substantially identical to data points in the training dataset. After training is complete, a trained VAE can be used to generate new data similar to data in the original training dataset by sampling latent variable values from the distribution learned by the prior network during training and converting these sampled values to new data points via the decoder network. Each new data point generated in this way can include properties similar (but not identical) to one or more properties of data points in the training dataset.

[0030] For example, a VAE can be trained on a training dataset that includes images of cats, where each image includes tens of thousands to millions of pixels. The trained VAE will include an encoder network that converts each image to hundreds or thousands of numerical latent variable values. Each latent variable will represent a respective visual attribute (e.g., the appearance of a cat’s face, fur, body, expression, pose, etc. in the image) found in one or more images used to train the VAE. The prior network will capture the variation and occurrence of visual attributes across all images in the training dataset as respective distributions of the latent variables (e.g., as mean, standard deviation, and / or other summary statistics associated with the numerical latent variable values). After training is complete, the trained VAE can be used to generate additional images of cats not included in the training dataset by sampling latent variable values that fall into the distributions of the latent variables learned by the prior network and converting those sampled latent variable values into new pixel values in additional images of cats via the decoder network.

[0031] VAEs can be used in a variety of practical applications. First, VAEs can be used to produce images, text, music, and / or other content that can be used in advertisements, publications, games, videos, and / or other types of media. Second, VAEs can be used in computer graphics applications. For example, VAEs can be used to render two-dimensional (2D) or three-dimensional (3D) characters, objects, and / or scenes without requiring a user to explicitly draw or create the 2D or 3D content. Third, VAEs can be used to generate or augment data. For example, the appearance of a person in an image (e.g., facial expression, gender, facial features, hair, skin, clothing, accessories, etc.) can be changed by adjusting the latent variable values output by the encoder network in a VAE from the image and converting the adjusted values into a new image using the decoder network from the same VAE. In another example, the prior and decoder networks in a trained VAE can be used to generate new images that are included in the training data for another machine learning model. Fourth, VAEs can be used to analyze or aggregate attributes of a given training dataset. For example, the visual attributes of faces, animals, and / or objects learned by a VAE from a set of images can be analyzed to better understand the visual attributes and / or improve the performance of a machine learning model that distinguishes different types of objects in images.

[0032] To help the VAE generate new data that accurately captures attributes found in the training dataset, the VAE is first trained on the training dataset. After the training of the VAE is complete, a separate machine learning model, referred to as a “classifier,” is trained to distinguish between values sampled from the latent variable distribution learned by the prior network from the training dataset and values sampled from the actual distribution of latent variables generated by the encoder network from the training dataset. For example, the VAE can first be trained to learn a latent variable distribution that represents visual attributes of faces in images included in the training dataset. The classifier can then be trained to determine whether a set of latent variable values was produced by the encoder network in the VAE from an image in the training dataset or sampled from the latent variable distribution learned by the prior network.

[0033] The trained VAE and classifier can then be used together to produce generative output that resembles the data in the training dataset. First, a set of latent variable values is sampled from the latent variable distribution learned by the prior network, and the sampled latent variable values are input to the classifier to generate a “reweighting factor” that captures the difference between the sampled latent variable values and actual latent variable values generated by the encoder network from data in the training dataset. Next, the reweighting factor is combined with the sampled latent variable values to shift the sampled latent variable values toward the latent variable values generated by the encoder network from actual data in the training dataset. The shifted latent variable values are then input to the decoder network to produce new “generative output” that is not in the training dataset.

[0034] For example, the prior network can store statistical data related to latent variable values that represent gender, facial expression, facial features, hair color and style, skin tone, clothing, accessories, and / or other attributes of faces included in the training dataset. A set of latent variable values can be sampled using the statistical data stored in the prior network, and the classifier can be applied to the sampled latent variable values to generate one or more values between 0 and 1 that represent the probability that the sampled latent variable values were produced from the training dataset by the encoder network and / or sampled using the prior network. The output of the classifier can then be converted to a reweighting factor that can be used to shift the sampled latent variable values away from regions that do not represent attributes of actual images in the training dataset. The decoder network can then be applied to the shifted latent variable values to generate new facial images with identifiable and / or realistic facial features.

[0035] System Overview

[0036] Figure 1A computing device 100 configured to implement aspects of the various embodiments is shown. In one embodiment, the computing device 100 includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. The computing device 100 is configured to run a training engine 122 and an execution engine 124 that reside in memory 116. Note that the computing device described herein is illustrative, and any other technically feasible configuration falls within the scope of the present disclosure. For example, multiple instances of the training engine 122 and the execution engine 124 can be executed on a set of nodes in a distributed and / or cloud computing system to implement the functionality of the computing device 100.

[0037] In one embodiment, the computing device 100 includes, without limitation, an interconnect (bus) 112 that connects one or more processors 102, an input / output (I / O) device interface 104 that couples to one or more input / output (I / O) devices 108, a memory 116, a storage 114, and a network interface 106. The processor 102 can be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units (e.g., a CPU configured to run with a GPU). In general, the processor 102 can be any technically feasible hardware unit capable of processing data and / or executing software applications. Moreover, in the context of the present disclosure, the computing elements shown in the computing device 100 can correspond to a physical computing system (e.g., a system in a data center), or can be a virtual computing instance executing within a computing cloud.

[0038] In one embodiment, the I / O devices 108 include devices capable of receiving input, such as a keyboard, a mouse, a touchpad, and / or a microphone, and devices capable of providing output, such as a display device and / or a speaker. Moreover, the I / O devices 108 can include devices capable of receiving input and providing output, such as a touchscreen, a universal serial bus (USB) port, etc. The I / O devices 108 can be configured to receive various types of input from an end user (e.g., a designer) of the computing device 100, and also provide various types of output to the end user of the computing device 100, such as a displayed digital image or digital video or text. In some embodiments, one or more I / O devices 108 are configured to couple the computing device 100 to a network 110.

[0039] In one embodiment, the network 110 is any technically feasible type of communication network that allows for the exchange of data between the computing device 100 and external entities or devices, such as a web server or another networked computing device. For example, the network 110 can include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet, among others.

[0040] In one embodiment, the storage 114 includes non-volatile storage for application programs and data, and can include fixed or removable magnetic, optical, or solid-state storage devices, as well as CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. The training engine 122 and the execution engine 124 can be stored in the storage 114 and loaded into the memory 116 at execution time.

[0041] In one embodiment, the memory 116 includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof. The processor 102, the I / O device interface 104, and the network interface 106 are configured to read data from and write data to the memory 116. The memory 116 includes various software programs executable by the processor 102, as well as application data associated with the software programs, including the training engine 122 and the execution engine 124.

[0042] The training engine 122 includes functionality to train a variational autoencoder (VAE) on a training data set, and the execution engine 124 includes functionality to execute one or more portions of the VAE to generate additional data not found in the training data set. For example, the training engine 122 can train an encoder network, a prior network, and / or a decoder network in the VAE on a set of training images, and the execution engine 124 can execute a generative model including the trained prior network and decoder network to produce additional images not found in the training images.

[0043] In some embodiments, the training engine 122 and the execution engine 124 use techniques to mitigate a mismatch between a latent variable distribution learned by the prior network from the training data set and an actual distribution of latent variables output by the encoder network from the training data set. More specifically, the training engine 122 and the execution engine 124 learn to identify and avoid regions in the latent variable space of the VAE that do not encode actual properties of data in the training data set. As described in further detail below, this improves the generative performance of the VAE by increasing the likelihood that generative outputs produced by the VAE capture properties of data in the training data set.

[0044] Variational autoencoder with noise-contrastive prior

[0045] Figure 2are in accordance with various embodiments Figure 1 A more detailed description of the training engine 122 and the execution engine 124 in accordance with various embodiments. The training engine 122 trains the VAE 200 that learns a distribution of a dataset of training data 208, and the execution engine 124 executes one or more portions of the VAE 200 to produce generative output 250 that includes additional data points in the distribution that are not found in the training data 208.

[0046] As shown, the VAE 200 includes a number of neural networks: an encoder 202, a prior 252, and a decoder 206. The encoder 202 “encodes” a set of training data 208 into latent variable values, the prior 252 learns a distribution of the latent variables output by the encoder 202, and the decoder 206 “decodes” latent variable values sampled from the distribution into reconstruction data 210 that substantially reproduces the training data 208. For example, the training data 208 can include images of human faces, animals, vehicles, and / or other types of objects; speech, music, and / or other audio; articles, posts, written documents, and / or other text; three-dimensional point clouds, meshes, and / or models; and / or other types of content or data. When the training data 208 includes images of human faces, the encoder 202 can convert pixel values in each image into a smaller number of latent variables that represent inferred visual attributes of the object and / or image (e.g., skin color, hair color and style, shapes and sizes of facial features, gender, facial expressions, and / or other features of the human face in the image), the prior 252 can learn a mean and variance of the distribution of the latent variables over multiple images in the training data 208, and the decoder 206 can convert latent variables sampled from the distribution of latent variables and / or output by the encoder 202 into image reconstructions in the training data 208.

[0047] The generative operation of the VAE 200 can be expressed using the following probability model:

[0048] p(x, z) = p(z)p(x|z), (1)

[0049] where p(z) is a prior distribution learned by the prior 252 for the latent variable z, and p(x|z) is a likelihood function or the decoder 206 that generates data x given the latent variable z. In other words, a latent variable is sampled from the prior 252 p(z), and the data x has a likelihood conditioned on the sampled latent variable z. The probability model includes a posterior p(z|x) that is used to infer the value of the latent variable z. Because p(z|x) is difficult to work with, another distribution q(z|x) learned by the encoder 202 is used to approximate p(z|x).

[0050] As shown, the training engine 122 performs a VAE training phase 220 that updates the parameters of the encoder 202, the prior 252, and the decoder 206 based on an objective 232 computed based on an error between the probabilistic model representing the VAE 200 and the training data 208 (e.g., a set of images, text, audio, video, etc.) and the reconstructed data 210. In particular, the objective 232 includes a variational lower bound on the log-likelihood p(x) to be maximized:

[0051]

[0052] where q(z\ x) is the approximate posterior learned by the encoder 202, and KL is the Kullback-Leibler (KL) divergence. The final training objective is expressed as where p d (x) is the distribution of the training data 208.

[0053] Those skilled in the art will appreciate that the prior 252 can not match the aggregate approximate posterior distribution output by the encoder 202 after the VAE training phase 220 is complete from the training data 208. In particular, the aggregate approximate posterior can be expressed as During the VAE training phase 220, the parameters of the prior 252 are maximized with respect to corresponding to making the prior 252 as close as possible to the aggregate approximate posterior by minimizing KL(q(z) \ p(z)) with respect to p(z). However, at the end of the VAE training phase 220, the prior 252 p(z) does not perfectly match the aggregate approximate posterior q(z) (e.g., because the expressiveness of the prior 252 is insufficient to capture the aggregate approximate posterior). As a result of this mismatch, the latent variable distribution learned by the prior 252 from the training data 208 will assign high probabilities to regions in the latent space that are not occupied by any of the samples in the training data 208. In turn, the decoder 206 is unable to transform samples from these regions into data that meaningfully resembles or reflects the properties of any of the training data 208.

[0054] In one or more embodiments, the training engine 122 alleviates the mismatch between the aggregate approximate posterior distribution learned by the encoder 202 and the prior distribution encoded by the prior 252 at the end of the VAE training phase 220 by creating a noise-contrast prior (NCP) 226 that adjusts samples from the prior 252 to avoid regions in the latent space that are not occupied by samples in the training data 208. In some embodiments, the NCP 226 includes the following form:

[0055] p NCP (z) = (r(z)p(z)) / Z, (3)

[0056] where p(z) is the base prior 252 distribution (e.g., a Gaussian distribution), r(z) is a reweighting factor (e.g., reweighting factor 218), Z = J r(z)p(z) dz is a normalization constant. The function r maps an n-dimensional real-valued latent variable z to a positive scalar.

[0057] As shown, the NCP 226 is created using a combination of the prior 252 and a classifier 212 that outputs probabilities 214 related to the outputs of the encoder 202 and the prior 252. More specifically, the classifier 212 includes a binary classifier that analyzes a latent variable sample from the VAE 200 and determines whether the sample is from the encoder 202 (e.g., after a corresponding sample of the training data 208 is input to the encoder 202) or from the prior 252.

[0058] To create the NCP 226, the training engine 122 freezes the parameters of the encoder 202, the prior 252, and the decoder 206 in the VAE 200 after the VAE training phase 220 is complete and performs a classifier training phase 222 that trains the classifier 212 to distinguish between a first set of latent variable values generated by the encoder 202 from the training data 208 and a second set of latent variable values sampled from the prior 252. For example, the classifier 212 can include a residual neural network, a tree-based model, a logistic regression model, a support vector machine, and / or other types of machine learning models. The input to the classifier 212 can include a set of latent variable values from the latent space of the VAE 200, and the output from the classifier 212 can include two probabilities 214 that sum to one: a first probability representing the likelihood that the set of latent variable values was generated by the encoder 202, and a second probability representing the likelihood that the set of latent variable values was sampled from the prior 252.

[0059] In one or more embodiments, the classifier 212 is trained using an objective 234 that includes a binary cross-entropy loss:

[0060]

[0061] In the above equation, is the binary classifier 212 that generates the classification probabilities 214 that distinguish between samples from the encoder 202 and samples from the prior 252. When Equation 4 is minimized. Denoting the optimal classifier 212 as D*(z), the reweighting factor is estimated as:

[0062]

[0063] After both the VAE training phase 220 and the classifier training phase 222 are complete, the training engine 122 and / or another component of the system creates an NCP 226 from the prior 252 and the classifier 212. For example, the training engine 122 can use Equation 3 to create the NCP 226 in the form of p(z) = r(z)p(z) where p(z) is the prior 252 and r(z) is a reweighting factor. NCP (z)∝r(z)p(z) is a more expressive distributional substitute for the base prior 252. In turn, the NCP 226 uses the reweighting factor r(z) to match the prior 252 to the aggregate approximate posterior, thereby avoiding regions of the latent space associated with z that do not correspond to the encoded properties of the training data 208.

[0064] After the VAE training phase 220 and the classifier training phase 222 are complete and the NCP 226 is created, the execution engine 124 uses the NCP 226 and / or one or more portions of the VAE 200 to produce generative output 250 that is not found in the training data set 208. In particular, the execution engine 124 uses latent variable samples 236 from the latent variable distribution learned by the prior 252 and corresponding reweighting factors 218 generated using the classifier 212 to generate NCP samples 224 from the NCP 226. The execution engine 124 then uses the NCP samples 224 to generate a data distribution 238 as output of the decoder 206, and subsequently samples from the data distribution 238 to produce the generative output 250.

[0065] For example, the execution engine 124 can obtain a set of latent variable samples 236 as values of latent variables sampled from a distribution described by parameters (e.g., mean and variance) output by the prior 252 after the VAE 200 is trained on the training data 208 that includes images of human faces. The execution engine 124 can apply the classifier 212 to the latent variable samples 236 to generate probabilities 214 as one or more values between 0 and 1, and convert the probabilities 214 to one or more reweighting factors 218 using Equation 5. The execution engine 124 can use the reweighting factors 218 to convert the latent variable samples 236 to NCP samples 224 from the NCP 226, and apply the decoder 206 to the NCP samples 224 to obtain parameters of a data distribution 238 corresponding to the likelihood p(x|z) (e.g., pixel value distributions for individual pixels in an image given an NCP sample 224). The execution engine 124 can then sample from the likelihood parameterized by the decoder 206 to produce generative output 250 that includes images of human faces. Because the latent variable samples 236 and the NCP samples 224 are obtained from a continuous latent space representation, the execution engine 124 can further interpolate between visual attributes represented by the latent variables (e.g., produce smooth transitions between facial expressions of anger and happiness represented by one or more latent variables) to generate human face images that are not in the training data 208.

[0066] Execution engine 124 includes the capability to generate NCP samples 224 from NCP 226 using various techniques. First, execution engine 124 can use the sampling-importance-resampling (SIR) technique to generate NCP samples 224 based on latent variable samples 236 and reweighting factors 218. In the SIR technique, execution engine 124 generates NCP samples 224 from prior information. M samples are generated. Then, the execution engine 124 resamples one of the M proposed samples using importance weights calculated with a reweighting factor 218.

[0067] w (m) =p NCP (z (m) ) / p(z (m) )=r(z (m) (6)

[0068] Each importance weight w (m) The probability is proportional to the probability of resampling the corresponding sample from the M original samples. Because SIR is non-iterative, the execution engine 124 can generate sample proposals from the prior 252 and evaluate r on the sample proposals in parallel.

[0069] Second, execution engine 124 can use a Langevin Dynamics (LD)-based sampling technique to generate NCP samples 224 from latent variable samples 236 and reweighting factors 218. This LD-based sampling technique is performed using the following energy function:

[0070] E(z)=-log r(z)-log p(z) (7)

[0071] During the LD-based sampling process, the execution engine 124 initializes the sample z0 by extracting it from the prior 252p(z) and iteratively updates the sample using the following method:

[0072]

[0073] Where ε t ~N(0,1), where λ is the step size. After a finite number of iterations of LD, the initial samples are converted into corresponding NCP samples.

[0074] In some embodiments, the VAE 200 is a hierarchical VAE that uses deep neural networks for the encoder 202, the prior 252, and the decoder 206. The hierarchical VAE includes a hierarchy of latent variables 204 that partitions the latent variables into a sequence of disjoint groups. Within the hierarchy of latent variables 204, samples from a given group of latent variables are combined with a feature map and passed to a subsequent group of latent variables in the hierarchy for generating samples from the subsequent group.

[0075] Continuing with the probabilistic model represented by Equation 1, the partitioning of the latent variables can be represented by z = {z1, z2,..., zK}, where K is the number of groups. Within the hierarchy of latent variables 204, the prior 252 is represented by p(z) = Πk=1K p(zk), and the approximate posterior is represented by p(z | x) = Πk=1K q(zk | z1, z2,..., zk-1, x). K θ k l ＜k k l ＜k l ＜k k ＜k

[0076] In some embodiments, the encoder 202 includes a bottom-up model and a top-down model that perform bidirectional inference on the groups of latent variables based on the training data 208. The top-down model is then reused as the prior 252 to infer latent variable values that are input to the decoder 206 to produce the reconstructed data 210 and / or the generative output 250. The architecture of the encoder 202 and the decoder 206 will be described in more detail below with reference to FIGS. 3-5. Figures 3A-3B

[0077] When the VAE 200 is a hierarchical VAE that includes the hierarchy of latent variables 204, the objective 232 includes maximizing the lower bound on the evidence in the form:

[0078]

[0079] ​​​​​​​​​​​​​​​​is the approximate posterior to the (k - 1)th group. Further, log p(x | z) is the log likelihood of the observed data x given the sampled latent variable z; this term is maximized when p(x | z) assigns high probability to the original data x (i.e., when the decoder 206 attempts to reconstruct a data point x in the training data 208 given the latent variable z generated by the encoder 202 from that data point). The "KL" term in the equation represents the KL divergence between the posterior and the corresponding prior (e.g., represented by the prior 252) for different levels of the latent variable hierarchy 204. Each KL(q(z k | z ＜k , x) || p(z k | z ＜k )) can be considered as the amount of information encoded in the kth group. A reparameterization trick can be used to backpropagate the parameters of the encoder 202 through the objective 232.

[0080] In one or more embodiments, the NCP 226 for the hierarchical VAE 200 with the latent variable hierarchy 204 includes a hierarchical NCP 226, which is defined as:

[0081]

[0082] where each factor is an energy-based model (EBM). During the training of the hierarchical VAE 200, the training engine 122 initially performs the VAE training phase 220 using the prior and the objective 232 represented by Equation 8. Then, the training engine 122 performs the classifier training phase 222 with the parameters of the hierarchical VAE 200 frozen.

[0083] During the classifier training phase 222 of the hierarchical NCP 226, the training engine 122 trains a plurality of classifiers, where the number of classifiers is equal to the number of groups of latent variables in the latent variable hierarchy 204, and each classifier is assigned to a corresponding group of latent variables in the latent variable hierarchy 204. Each classifier is additionally trained to distinguish between values from the assigned group of latent variables generated by the encoder 202 and values sampled from the prior 252 for the assigned group of latent variables. Each classifier can further use one or more residual neural networks to predict whether a sample from the corresponding group of latent variables is from the prior 252 or from the encoder 202, as described in further detail below with respect to Figures 5A-5C .

[0084] More specifically, each classifier is trained using the following objective 234.

[0085]

[0086] The external expected samples from each group up to the (k-1)th group and the internal expected samples from encoder 202 and the basic prior 252, respectively, are approximate posteriors of the output of the kth group, conditioned on the same z < k. Each classifier D k For sample z k Classification is performed, while the predictions are adjusted on z<k using a shared contextual feature c(z<k). This shared contextual feature may include one or more previous latent variable groups of samples z<k and / or representations extracted from z<k.

[0087] In the following case, objective 234 represented by Equation 13 is minimized.

[0088] Use D*(z), c(z) ＜k To represent the optimal classifier 212, the reweighting factor of hierarchical NCP 226 can be obtained as:

[0089]

[0090] After training level VAE 200 and the corresponding level NCP 226, execution engine 124 generates latent variable samples 236 from prior 252 using ancestor sampling and generates NCP samples 224 from each group in latent variable level 204 using SIR or LD. Then, one or more latent variable groups from NCP samples 224 are input to decoder 206 to produce data distribution 238 and corresponding generative output 250, as discussed above.

[0091] Figure 3A According to various embodiments Figure 2 The example architecture of encoder 202 in the layered version of VAE 200 is shown in the figure. As shown, the example architecture forms a bidirectional inference model, which includes a bottom-up model 302 and a top-down model 304.

[0092] The bottom-up model 302 includes multiple residual networks 308-312, and the top-down model 304 includes multiple additional residual networks 314-316 and trainable parameters 326. Each of the residual networks 308-316 includes one or more residual units, which will be discussed below. Figure 4A and Figure 4B It is described in detail.

[0093] The residual networks 308-312 in the bottom-up model 302 deterministically extract features from the input 324 (e.g., an image) to infer the latent variables in the approximate posterior (e.g., q(z\x) in the probabilistic model of the VAE 200). In turn, the components of the top-down model 304 are used to generate the parameters of each conditional distribution in the hierarchy of latent variables 204. After sampling latent variables from a given group in the hierarchy of latent variables 204, these samples are combined with the feature maps in the bottom-up model 302 and passed as input to the next group.

[0094] More specifically, a given data input 324 is processed in sequence by the residual networks 308, 310, and 312 in the bottom-up model 302. The residual network 308 generates a first feature map from the input 324, the residual network 310 generates a second feature map from the first feature map, and the residual network 312 generates a third feature map from the second feature map. The third feature map is used to generate the parameters of a first group 318 of latent variables in the hierarchy of latent variables 204, and samples are taken from the group 318 and combined (e.g., added) with the parameters 326 to produce input to the residual network 314 in the top-down model 304. The output of the residual network 314 in the top-down model 304 is combined with the feature map produced by the residual network 310 in the bottom-up model 302 and used to generate the parameters of a second group 320 of latent variables in the hierarchy of latent variables 204. Samples are taken from the group 320 and combined with the output of the residual network 314 to generate input to the residual network 316. Finally, the output of the residual network 316 in the top-down model 304 is combined with the output of the residual network 308 in the bottom-up model 302 to generate the parameters of a third group 322 of latent variables, and samples can be taken from the group 322 to produce a full set of latent variables representing the input 324.

[0095] While Figure 3AThe example architecture of the encoder 202 is illustrated with three latent variable tiers of latent variable groups 318-322, but one of skill in the art will appreciate that the encoder 202 can utilize a different number of latent variable tiers, a different number of latent variables in each group of the tier, and / or a different number of residual units in the residual network. For example, a latent variable tier 204 of an encoder trained using 28x28 pixel images of handwritten characters can include 15 groups of latent variables at two different “scales” (i.e., spatial dimensions) and one residual unit per group of latent variables. The first five groups have 4x4x20-dimensional latent variables (in the form of height x width x channels), and the next ten groups have 8x8x20-dimensional latent variables. In another example, a latent variable tier 204 of an encoder trained using 256x256 pixel images of human faces can include 36 groups of latent variables at five different scales and two residual units per group of latent variables. The scales include 8x8x20, 16x16x20, 32x32x20, 64x64x20, and 128x128x20, and 4, 4, 4, 8, and 16 groups of spatial dimensions, respectively.

[0096] Figure 3B is an example encoder architecture according to various embodiments Figure 2 An example architecture of the generative model in a hierarchical version of the VAE 200. As shown, the generative model includes a top-down model 304 from the example encoder architecture of Figure 3A and an additional residual network 328 that implements the decoder 206.

[0097] In the example generative model architecture of Figure 3B representing the groups 318-322 of latent variables in the inference hierarchy are extracted by the residual networks 314-316 of the top-down model 304. Then, a sample from the last group 322 of latent variables is combined with the output of the residual network 316 and provided as input to the residual network 328. Accordingly, the residual network 328 generates a data output 330 that is a reconstruction of the corresponding input 324 to the encoder and / or a new data point sampled from the distribution of training data for the VAE 200.

[0098] In some embodiments, the top-down model 304 is used to learn a prior of the latent variables during training of the VAE 200 (e.g., the prior is learned by the encoder 202). In other embodiments, the top-down model 304 is used to learn a posterior of the latent variables during training of the VAE 200 (e.g., the posterior is learned by the decoder 206). Figure 2The prior (252) is then reused in the generative model and / or NCP 226 to sample from groups 318-322 of the latent variables before the decoder 206 converts some or all of the samples into the generative output. This sharing of the top-down model 304 between the encoder 202 and the generative model reduces the computational and / or resource overhead associated with a separate top-down model for learning the prior 252 and using a separate top-down model in the generative model. Alternatively, the VAE 200 can be configured such that the encoder 202 uses a first top-down model to generate latent representations of the training data 208, and the generative model uses a second, separate top-down model as the prior 252.

[0099] Figure 4A According to various embodiments Figure 2 Example residual units in encoder 202 of the layered version of VAE 200. More specifically, Figure 4A It shows Figure 3A The residual units used by one or more residual networks 308-312 in the bottom-up model 302. As shown in the figure, the residual unit includes multiple blocks 402-410 and residual links 430 that add the input of the residual unit to the output of the residual unit.

[0100] Block 402 is a batch-normalized block with the Swish activation function, block 404 is a 3x3 convolutional block, block 406 is a batch-normalized block with the Swish activation function, block 408 is a 3x3 convolutional block, and block 410 is a squeeze and activation block that performs channel-by-channel gating (e.g., a squeeze operation, such as averaging, which obtains a single value for each channel, followed by an activation operation that applies a nonlinear transformation to the output of the squeeze operation to produce per-channel weights) in the residual unit. Furthermore, the same number of channels are maintained on blocks 402-410. This differs from a conventional residual unit with a convolution-batch-normalization-activation order. Figure 4A The residual units include a batch normalization-activation-convolution sequence, which can improve the performance of the bottom-up model 302 and / or encoder 202.

[0101] Figure 4B According to various embodiments Figure 2 Example residual cells from the generative section of the hierarchical version of VAE 200. More specifically, Figure 4B It shows Figure 3A and Figure 3B The residual units used in one or more residual networks 314-316 in the top-down model 304. As shown in the figure, the residual unit includes multiple blocks 412-426 and residual links 432 that add the input of the residual unit to the output of the residual unit.

[0102] Block 412 is a batch normalization block, block 414 is a 1x1 convolution block, block 416 is a batch normalization block with a Swish activation function, block 418 is a 5x5 depthwise separable convolution block, block 420 is a batch normalization block with a Swish activation function, block 422 is a 1x1 convolution block, block 424 is a batch normalization block, and block 426 is a squeeze-and-excitation block. Blocks 414-420 labeled with “EC” indicate that the number of channels is expanded by “E” times, while blocks labeled with “C” include the original “C” number of channels. Specifically, block 414 performs a 1x1 convolution that expands the number of channels to improve the expressiveness of the depthwise separable convolution performed by block 418, and block 422 performs a 1x1 convolution that maps back to the “C” number of channels. At the same time, compared to regular convolution, depthwise separable convolution reduces the parameter size and computational complexity with increased kernel size without negatively impacting the performance of the generative model.

[0103] Further, using batch normalization with a Swish activation function in the residual units of Figure 4A and Figure 4B can improve the training of the encoder 202 and / or the generative model on regular residual units or networks. For example, compared to using weight normalization and exponential linear unit activations in the same residual unit, Figure 4A the combination of batch normalization and Swish activation in the residual units of

[0104] Figure 5A is an exemplary residual block in a classifier (e.g., classifier 212) according to various embodiments, which can be used with a hierarchical version of the VAE 200 of Figure 2 . More specifically, Figure 5A shows a residual block named “Residual Block A” that is included in a classifier that is trained to predict the probability that the corresponding latent variable group in the hierarchy of latent variables 204 is generated by the encoder 202 from the training data 208 and / or sampled from the prior 252. As shown, the residual block includes several blocks 502-510 (which can also be referred to as layers) and a residual link 512 that adds the input of the residual block to the output of the residual block.

[0105] Block 502 is a batch normalization block with a Swish activation function, block 504 is a 3x3 convolution block, block 506 is a batch normalization block with a Swish activation function, block 508 is a 3x3 convolution block, and block 510 is a squeeze-and-excitation block. Figure 5AAll blocks 502-510 are labeled "C", indicating that the original number of "C" channels is preserved in the feature maps of all blocks 502-510 and the residual block output. Furthermore, the values ​​of "s1" and "p1" associated with the 3x3 convolutional blocks 504 and 508 represent the stride and padding parameters, respectively, and are both set to 1.

[0106] Figure 5B These are exemplary residual blocks in classifiers (e.g., classifier 212) according to various embodiments, which can be used with... Figure 2 It is used in conjunction with the layered version of VAE 200. More specifically, Figure 5B A residual block named "Residual Block B" is shown, which is included in a classifier trained to predict the probabilities of the corresponding latent variable groups in latent variable hierarchy 204 generated by encoder 202 from training data 208 and / or sampled from prior 252. As shown, the residual block includes several blocks 522-530 (also referred to as layers), a residual link 532 that adds the input of the residual block to the output of the residual block, and several additional blocks 534-536 (also referred to as layers) along the residual link 532.

[0107] Block 522 is a batch-normalized block with the Swish activation function, block 524 is a 3x3 convolutional block, block 526 is a batch-normalized block with the Swish activation function, block 528 is a 3x3 convolutional block, and block 530 is a squeeze and excitation block. Block 534 is the Swish activation function, and block 536 includes a sequence of 1x1 convolutional kernels, which are part of a larger convolutional kernel (factorization).

[0108] The value "C" following blocks 522 and 534 indicates that blocks 522 and 534 output the original number of channels "C", while the value "2C" following blocks 524-530 and 536 indicates that the feature maps of blocks 524-530 and 536, as well as the residual blocks, output twice the original number of channels "C". The values ​​"s2" and "p1" associated with the first 3x3 convolutional block 524 indicate that the stride of block 524 is 2 and the padding is 1. The values ​​"s1" and "p1" associated with the second 3x3 convolutional block 528 indicate that the stride of block 528 is 1 and the padding is 1. The values ​​"s2" and "p0" associated with the series of 1x1 convolutional kernels in block 536 indicate that the stride of each 1x1 convolution is 2 and the padding is 0.

[0109] Figure 5C This is an exemplary architecture 542 for a classifier according to various embodiments, which can be used with... Figure 2The hierarchical version of the VAE 200 is used together. As shown, the architecture 542 starts with a 3x3 convolutional kernel with a rectified linear unit (ReL) activation function, a stride of 1, and padding of 1. The 3x3 convolution is followed by 8 residual blocks: three instances of residual block A, followed by one instance of residual block B, followed by three instances of residual block A, followed by one instance of residual block B. The structure of residual block A is described above with respect to Figure 5A The structure of residual block B is described above with respect to Figure 5B The structure of residual block B is described above with respect to

[0110] While the classifier 212 and the NCP 226 have been described above with respect to the VAE 200, it will be appreciated that the classifier 212 and the NCP 226 can also be used to improve generative outputs of other types of generative models that include a prior distribution of latent variables in a latent space, a decoder that converts samples of the latent variables to samples in a data space of a training dataset, and a component or method that maps samples in the training dataset to samples in the latent space of the latent variables. In the context of the VAE 200, the prior distribution is learned by the prior 252, the encoder 202 converts samples of the training data in the data space to latent variables in the latent space associated with the latent variable hierarchy 204, and the decoder 206 is a neural network separate from the encoder 202 and converts latent variable values from the latent space back to a likelihood in the data space.

[0111] A generative adversarial network (GAN) is another type of generative model that can be used with the classifier 212 and the NCP 226. The prior distribution in the GAN is represented by a Gaussian distribution and / or another simple distribution, the decoder in the GAN is a generator network that converts samples from the prior distribution to samples in the data space of the training dataset, and numerical inversion of the generator network can be performed to map samples in the training dataset to samples in the latent space of the latent variables.

[0112] A normalizing flow is another type of generative model that can be used with the classifier 212 and the NCP 226. Like the GAN, the prior distribution in the normalizing flow is implemented using a Gaussian distribution and / or another simple distribution. The decoder in the normalizing flow is represented by a decoder network that associates the latent space with the data space using a deterministic and invertible transformation from observed variables in the data space to latent variables in the latent space. Inversion of the decoder network in the normalizing flow can be used to map samples in the training dataset to samples in the latent space.

[0113] For each of these types of generative models, a first training phase is used to train the generative model, and a second training phase is used to train the classifier 212 to distinguish between latent variable values sampled from the prior distribution in the generative model and latent variable values mapped to data points in the training dataset. The NCP 226 is then created by combining the prior distribution with a reweighting factor computed from the output of the classifier 212. Samples from the prior distribution of the generative model can then be converted to samples from the NCP 226 using SIR and / or LD techniques. Decoders in the generative model can then be used to convert samples from the NCP 226 to new data in the data space of the training dataset.

[0114] Figure 6 is a flowchart of method steps for training a generative model according to various embodiments. Although the method steps are described in connection with the system of FIG. 5, one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the application. Figure 1 -5, one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the application.

[0115] As shown, the training engine 122 performs 602 a first training phase that trains a prior network, an encoder network, and a decoder network included in a generative model based on a training dataset. For example, the training engine 122 can input a set of training images scaled to a certain resolution into a hierarchical VAE (or another type of generative model that includes a distribution of latent variables). The training images can include human faces, animals, vehicles, and / or other types of objects. The training engine 122 can also perform one or more operations that update parameters of the hierarchical VAE based on outputs of the prior, encoder, and decoder networks and respective objective functions.

[0116] Next, the training engine 122 performs 604 a classifier training phase that trains one or more classifiers to distinguish between a first set of latent variable values sampled from a prior distribution learned by the prior network and a second set of latent variable values generated by the encoder network from the training dataset. Continuing the example above, the training engine 122 can freeze parameters of the hierarchical VAE at the end of the first training phase. The training engine 122 can then obtain the first set of latent variable values by sampling from the prior network and the second set of latent variable values by applying the encoder network to a subset of the training images. The training engine 122 can then perform one or more operations that iteratively update weights of each classifier using a training technique (e.g., gradient descent and backpropagation), an objective function (e.g., binary cross-entropy loss), and / or one or more hyperparameters so that probabilities output by the classifiers (e.g., probabilities that respective sets of latent variable values in the latent variable hierarchy were generated by the prior or encoder) better match respective labels.

[0117] At the end of the classifier training phase, the training engine 122 produces a series of classifiers, where each classifier in the series distinguishes between latent variable values sampled from a respective group in the latent variable hierarchy by the prior network and latent variable values generated for the same group in the latent variable hierarchy by the encoder network. The predictions of the classifiers can additionally be based on a feature map that includes and / or is representative of latent variable values of a previous group in the latent variable hierarchy.

[0118] The training engine 122 then creates 606 an NCP based on the prior distribution and the reweighting factors associated with the predictions of the classifiers. For example, the training engine 122 can incorporate the prior network and the classifiers into the NCP. The NCP converts the one or more probabilities of the classifier outputs into reweighting factors that are used to adjust the latent variable samples for each group of latent variables produced using the prior network, as described above.

[0119] Figure 7 is a flowchart of method steps for producing a generative output according to various embodiments. Although this method is described in connection with the system of Figure 1 -5, one skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

[0120] As shown, the execution engine 124 samples 702 one or more values from a prior distribution of latent variables learned by a prior network included in a generative model (e.g., VAE, normalized flow, GAN, etc.). For example, the execution engine 124 can use an ancestral sampling technique to sample from multiple groups of latent variables in a latent variable hierarchy learned by the prior network from a hierarchical VAE. After sampling a first value from a given group in the latent variable hierarchy, a second value is sampled from a next group in the latent variable hierarchy based on the first value and / or a feature map generated from the first value. In other words, the sampling of a given group of latent variables in the hierarchy is conditioned on the sampling of a previous group in the hierarchy.

[0121] Next, the execution engine 124 adjusts 704 the one or more values based on reweighting factors associated with an NCP of the generative model. For example, the execution engine 124 can apply the reweighting factors to the one or more values sampled in operation 702 to shift the values toward one or more other values of the latent variables that have been generated from a training dataset by an encoder network in the generative model.

[0122] As described above, the reweighting factors can be computed using one or more probabilities output by one or more classifiers that learn to distinguish between a first set of values sampled from a prior distribution and a second set of latent variable values generated by an encoder network included in the generative model. Each classifier can include a residual neural network (e.g., the residual blocks described above with respect to Figures 5A-5B the residual blocks) having one or more residual blocks. Each classifier can also distinguish between latent variable values sampled from a respective set of latent variable levels encoded in the prior network and latent variable values generated by the encoder network from a training dataset. The reweighting factors can be computed based on a quotient of a probability output by the classifier and a difference between the probability and 1. The adjustment in operation 704 can be performed by resampling the latent variable values based on importance weights proportional to the reweighting factors for multiple samples of the prior distribution and / or iteratively updating the latent variable values based on gradients of an energy function associated with the prior distribution and the reweighting factors.

[0123] The execution engine 124 then applies 706 a decoder network included in the generative model to the adjusted values to produce a generative output. For example, the decoder network can output parameters of a likelihood function based on the adjusted values, and a sample can be obtained from the likelihood function to produce the generative output (e.g., as pixel values for pixels in an image).

[0124] Example game streaming system

[0125] Figure 8 is an example system diagram of a game streaming system 800 in accordance with various embodiments. Figure 8 includes a game server 802 (which can include components, features and / or functionality similar to those of the example computing device 100 described above with respect to Figure 1 includes a game server 802 (which can include components, features and / or functionality similar to those of the example computing device 100 described above with respect to Figure 1 includes a game server 802 (which can include components, features and / or functionality similar to those of the example computing device 100 described above with respect to

[0126] In the system 800, for a game session, the client device 804 can receive input data only in response to input to an input device, send the input data to the game server 802, receive encoded display data from the game server 802, and display the display data on a display 824. Again, more computationally intensive computations and processing are offloaded to the game server 802 (e.g., rendering of graphical output for the game session - particularly ray or path tracing - performed by a GPU of the game server 802). In other words, the game session is streamed from the game server 802 to the client device 804, thereby reducing the requirements of the client device 804 for graphics processing and rendering.

[0127] For example, with respect to instantiation of a game session, the client device 804 can display frames of the game session on the display 824 based on receiving display data from the game server 802. The client device 804 can receive input to the one or more input devices 826 and generate input data in response. The client device 804 can send the input data to the game server 802 via the communication interface 820 and over the network 806 (e.g., the Internet), and the game server 802 can receive the input data via the communication interface 818. The CPU 808 can receive the input data, process the input data, and send the data to the GPU 810, which causes the GPU 810 to generate a rendering of the game session. For example, the input data can represent movement of a user character in the game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 812 can render the game session (e.g., representative of the results of the input data), and the rendering capture component 814 can capture the rendering of the game session as display data (e.g., capturing image data of a rendered frame of the game session). The rendering of the game session can include ray or path tracing lighting and / or shadow effects, computed using one or more parallel processing units (such as the GPU 810), which can further use one or more specialized hardware accelerators or processing cores to perform ray or path tracing techniques of the game server 802. The encoder 816 can then encode the display data to generate encoded display data, and the encoded display data can be sent to the client device 804 via the communication interface 818 over the network 806. The client device 804 can receive the encoded display data via the communication interface 820, and the decoder 822 can decode the encoded display data to generate display data. The client device 804 can then display the display data via the display 824.

[0128] In some embodiments, the system 800 includes a game server 802 that is configured to implement a game session, and a client device 804 that is configured to display the game session. The game server 802 can include a CPU 808, a GPU 810, a rendering component 812, a rendering capture component 814, an encoder 816, and a communication interface 818. The client device 804 can include a communication interface 820, a decoder 822, and a display 824. Figures 1-2the training engine 122 and / or the inference engine 124. For example, one or more components of the game server 802 and / or the client device 804 can execute the training engine 122 to train a VAE and / or another generative model that includes an encoder network, a prior network, and / or a decoder network based on a training dataset (e.g., a set of images or models of characters or objects in a game). The executed training engine 122 can then train one or more classifiers to distinguish between a first set of values sampled from a prior distribution learned by the prior network and a second set of values of a set of latent variables generated by the encoder network from the training dataset. One or more components of the game server 802 and / or the client device 804 can then execute the inference engine 124 to produce a generative output (e.g., an additional image or model of a character or object not found in the training dataset) by sampling from the prior distribution, adjusting the sampled values based on a reweighting factor associated with the output of the classifier, and applying the decoder network to the adjusted sampled values. The generative output can then be displayed in the display 824 on the client device 804 during one or more game sessions.

[0129] In summary, the disclosed techniques improve generative outputs produced by VAEs and / or other types of generative models with a distribution of latent variables. After training a generative model on a training dataset, a classifier is trained to distinguish between a first set of samples from a prior distribution of latent variables (e.g., visual attributes of faces or other objects in images) learned by the generative model and a second set of samples from an approximate total posterior distribution of the latent variables associated with the training dataset (e.g., samples generated by an encoder portion of the generative model from a set of training images). The output of the classifier is used to compute a reweighting factor for the prior distribution that is combined with the prior distribution into a noise contrast prior (NCP) of the generative model. The NCP makes the prior distribution closer to the approximate total posterior, which makes samples from the NCP (e.g., samples from the prior distribution adjusted or selected based on the reweighting factor) able to avoid “holes” in the prior distribution that do not correspond to data samples in the training dataset. A given sample from the NCP is then input into a decoder portion of the generative model to produce a generative output that contains attributes extracted from the training dataset but not found in the training dataset.

[0130] At least one technical advantage of the disclosed technology over the prior art is that the generative output produced by the disclosed technology looks more realistic and similar to the data in the training dataset compared to the results typically produced using conventional variational autoencoders (or other types of generative models that learn a distribution of latent variables). Another technical advantage is that with the disclosed technology, the complex distribution of latent variables produced by the encoder from the training dataset can be approximated by a machine learning model that is trained and executed in a more efficient computational manner relative to the prior art. These technical advantages provide one or more technical improvements over prior art approaches.

[0131] 1. In some embodiments, a computer-implemented method for generating images using a variational autoencoder, the method comprising: determining one or more first values of a set of visual attributes included in a plurality of training images, wherein the set of visual attributes has been encoded via a prior network; applying a reweighting factor to the one or more first values in order to generate one or more second values for the set of visual attributes, wherein the one or more second values represent the one or more first values shifted toward one or more third values of the set of visual attributes, wherein the one or more third values have been generated via an encoder network; and performing one or more decoding operations on the one or more second values via a decoder network to generate a new image that is not included in the plurality of training images.

[0132] 2. The computer-implemented method of clause 1, wherein applying the reweighting factor to the one or more first values comprises generating the reweighting factor based on a classifier that distinguishes between values sampled from the set of visual attributes and values of the latent variables generated by the encoder network from the plurality of training images.

[0133] 3. The computer-implemented method of clause 1 or 2, wherein the new image includes at least one face.

[0134] 4. In some embodiments, a computer-implemented method for generating data using a generative model, the method comprising: sampling one or more first values from a distribution of latent variables learned by a prior network included in the generative model; applying a reweighting factor to the one or more first values in order to generate one or more second values for the latent variables, wherein the reweighting factor is generated based on one or more classifiers used to distinguish between values sampled from the distribution and values of the latent variables generated via an encoder network included in the generative model; and performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to produce a generative output.

[0135] 5. The computer-implemented method of clause 4, further comprising training the one or more classifiers based on binary cross-entropy loss.

[0136] 6. The computer-implemented method of clause 4 or 5, wherein prior to training the one or more classifiers, the prior network, the encoder network, and the decoder network are trained using a training dataset.

[0137] 7. The computer-implemented method of any of clauses 4-6, wherein the distribution of latent variables learned by the prior network comprises a hierarchy of latent variables, and sampling the one or more first values comprises: sampling a first value from a first set of latent variables included in the hierarchy of latent variables; and sampling a second value from a second set of latent variables included in the hierarchy of latent variables based on the first value and a feature map.

[0138] 8. The computer-implemented method of any of clauses 4-7, wherein the one or more classifiers comprise a first classifier and a second classifier, the first classifier distinguishing between a third value sampled from the first set of latent variables using the prior network and a fourth value of the first set of latent variables generated by the encoder network, the second classifier distinguishing between a fifth value sampled from the second set of latent variables using the prior network and a sixth value of the second set of latent variables generated by the encoder network.

[0139] 9. The computer-implemented method of any of clauses 4-8, wherein applying the reweighting factor to the one or more first values comprises resampling the one or more first values based on an importance weight proportional to the reweighting factor.

[0140] 10. The computer-implemented method of any of clauses 4-9, wherein applying the reweighting factor to the one or more first values comprises iteratively updating the one or more first values based on a gradient of an energy function associated with the distribution and the reweighting factor.

[0141] 11. The computer-implemented method of any of clauses 4-10, wherein the energy function comprises a difference between the distribution and the reweighting factor.

[0142] 12. The computer-implemented method of any of clauses 4-11, wherein the reweighting factor is generated by computing a quotient of a difference between a probability output by the one or more classifiers and one.

[0143] 13. The computer-implemented method of any of clauses 4-12, wherein at least one of the one or more classifiers comprises a residual neural network.

[0144] 14. In some embodiments, a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: sampling one or more first values from a distribution of latent variables learned by a previous component included in a generative model; applying a reweighting factor to the one or more first values in order to generate one or more second values for the latent variables, wherein the reweighting factor is generated based on one or more classifiers that are used to distinguish between values sampled from the distribution and values of the latent variables generated via an encoder network included in the generative model; and performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to produce a generative output.

[0145] 15. The non-transitory computer-readable medium of clause 14, wherein the instructions further cause the processor to perform the steps of: training, during a first training phase, the generative model based on a training data set; and after completion of the first training phase, training, during a second training phase, the one or more classifiers to distinguish between values sampled from the distribution and values of the latent variables generated via an encoder network.

[0146] 16. The non-transitory computer-readable medium of clause 14 or 15, wherein the one or more classifiers are trained based on binary cross-entropy loss.

[0147] 17. The non-transitory computer-readable medium of any of clauses 14-16, wherein sampling the one or more first values comprises: sampling a first value from a first group in a hierarchy of latent variables learned by a prior network that implements the previous component; and sampling a second value from a second group in the hierarchy of latent variables based on the first value and a feature map.

[0148] 18. The non-transitory computer-readable medium of any of clauses 14-17, wherein the one or more classifiers comprise a first classifier and a second classifier, the first classifier distinguishing between a third value sampled from the first group and a fourth value of the first group generated by the encoder network, the second classifier distinguishing between a fifth value sampled from the second group and a sixth value of the second group generated by the encoder network.

[0149] 19. The non-transitory computer-readable medium of any of clauses 14-18, wherein the one or more classifiers comprise a convolutional layer and one or more residual blocks.

[0150] 20. The non-transitory computer-readable medium of any of clauses 14-19, wherein the one or more residual blocks comprise: a first batch normalization layer with a first Swish activation function, a first convolutional layer after the first batch normalization layer with the first Swish activation function, a second batch normalization layer with a second Swish activation function, a second convolutional layer after the second batch normalization layer with the second Swish activation function, and a squeeze and excitation layer.

[0151] 21. The non-transitory computer-readable medium of any of clauses 14-20, wherein the prior component is implemented by at least one of a prior network or a Gaussian distribution.

[0152] 22. The non-transitory computer-readable medium of any of clauses 14-21, wherein the decoder network is implemented by at least one of a generator network included in a generative adversarial network, a decoder portion of a variational autoencoder, or an invertible decoder represented by one or more normalization flows.

[0153] 23. The non-transitory computer-readable medium of any of clauses 14-22, wherein the encoder network is implemented by at least one of an encoder portion of a variational autoencoder, an inversion of numerical values applied to a generator network included in a generative adversarial network, or an inversion of a decoder included in a normalization flow network.

[0154] 1a. A computer-implemented method for creating a generative model, the method comprising:

[0155] performing one or more operations based on a plurality of training images to generate a trained encoder network and a trained prior network, wherein the trained encoder network converts each image included in the plurality of training images to a set of visual attributes, and the trained prior network learns a distribution of the set of visual attributes over the plurality of training images;

[0156] performing one or more operations to train one or more classifiers to distinguish between values of the set of visual attributes generated by the trained encoder network and values of the set of visual attributes selected from the distribution learned by the trained prior network; and

[0157] combining the trained prior network with the one or more classifiers to produce a trained prior component included in the generative model,

[0158] wherein in operation, the trained prior component generates one or more values for the set of visual attributes in order to generate a new image that is not included in the plurality of training images.

[0159] 2a. The computer-implemented method of clause la, wherein combining the trained prior network and the one or more classifiers comprises combining one or more first values selected from a distribution learned from the trained prior network with a reweighting factor based on the one or more first values.

[0160] 3a. The computer-implemented method of clause la, wherein the new image includes at least one face.

[0161] 4a. A computer-implemented method for creating a generative model, the method comprising:

[0162] performing one or more operations based on a training dataset to generate a trained encoder network and a trained prior network, wherein the trained encoder network converts a plurality of data points included in the training dataset to a set of latent variables, and the trained prior network learns a distribution of the set of latent variables over the training dataset;

[0163] performing one or more operations to train one or more classifiers to distinguish between values of the set of latent variables generated via the trained encoder network and values sampled from the distribution learned from the trained prior network; and

[0164] creating a trained prior component based on the trained prior network and the one or more classifiers, wherein the trained prior component applies a reweighting factor to one or more first values sampled from the distribution learned from the trained prior network to generate one or more second values of the set of latent variables,

[0165] wherein in operation, the trained prior component generates one or more second values in order to generate a new data point that is not included in the training dataset.

[0166] 5a. The computer-implemented method of clause 4a, wherein the distribution learned from the trained prior network includes a hierarchy of latent variables, and wherein the one or more first values are sampled from the distribution learned from the trained prior network by:

[0167] sampling one first value from a first set of latent variables included in the hierarchy of latent variables; and

[0168] based on the first value and a feature map, sampling another first value from a second set of latent variables included in the hierarchy of latent variables.

[0169] 6a. The computer-implemented method of clause 5a, wherein the one or more classifiers include a first classifier that distinguishes a third value sampled from the first set of latent variables using the trained prior network from a fourth value of the first set of latent variables generated by the trained encoder network and a second classifier that distinguishes a fifth value sampled from the second set of latent variables using the trained prior network from a sixth value of the second set of latent variables generated by the trained encoder network.

[0170] 7a. The computer-implemented method of clause 4a, wherein the reweighting factor is applied to the one or more first values by resampling the one or more first values based on importance weights that are proportional to the reweighting factor.

[0171] 8a. The computer-implemented method of clause 4a, wherein the reweighting factor is applied to the one or more first values by iteratively updating the one or more first values based on a gradient of an energy function associated with a distribution learned by the trained prior network and the reweighting factor.

[0172] 9a. The computer-implemented method of clause 4a, wherein at least one of the one or more classifiers includes a residual neural network.

[0173] 10a. The computer-implemented method of clause 9a, wherein the residual neural network includes: a first batch normalization layer having a first Swish activation function, a first convolutional layer, a second batch normalization layer having a second Swish activation function, a second convolutional layer, and a squeeze and excitation layer.

[0174] 11a. The computer-implemented method of clause 9a, wherein the residual neural network includes a Swish activation function and a sequence of convolutional kernels.

[0175] 12a. The computer-implemented method of clause 4a, further comprising: computing the reweighting factor based on an output generated by the one or more classifiers from the one or more first values.

[0176] 13a. The computer-implemented method of clause 4a, wherein performing the one or more operations to train the one or more classifiers includes iteratively updating weights of the one or more classifiers based on a binary cross-entropy loss.

[0177] 14a. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:

[0178] performing one or more operations based on the training data set to train a generative model, wherein the generative model comprises a first component that transforms a plurality of data points included in the training data set into a set of latent variables and a second component that generates a prior distribution of the set of latent variables over the training data set;

[0179] performing one or more operations to train one or more classifiers to distinguish between values of the set of latent variables generated via the first component and values sampled from the prior distribution; and

[0180] creating a trained prior component based on the second component and the one or more classifiers, wherein the trained prior component applies a reweighting factor to one or more first values sampled from the prior distribution to generate one or more second values of the set of latent variables, wherein the reweighting factor is determined based on an output of the one or more classifiers generated from the one or more first values,

[0181] wherein in operation, the trained prior component produces the one or more second values in order to generate a new data point that is not included in the training data set.

[0182] 15a. The non-transitory computer-readable medium of clause 14a, wherein the instructions further cause the processor to perform the step of performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to generate the new data point.

[0183] 16a. The non-transitory computer-readable medium of clause 15a, wherein the decoder network is implemented by at least one of a generator network included in a generative adversarial network, a decoder portion of a variational autoencoder, or an invertible decoder represented by one or more normalizing flows.

[0184] 17a. The non-transitory computer-readable medium of clause 14a, wherein the reweighting factor is applied to the one or more first values by resampling the one or more first values based on an importance weight that is proportional to the reweighting factor.

[0185] 18a. The non-transitory computer-readable medium of clause 14a, wherein the reweighting factor is applied to the one or more first values by iteratively updating the one or more first values based on a gradient of an energy function associated with the distribution and the reweighting factor.

[0186] 19a. The non-transitory computer-readable medium of clause 18a, wherein the energy function comprises a difference between the prior distribution and the reweighting factor.

[0187] 20a. The non-transitory computer-readable medium of clause 14a, wherein at least one of the one or more classifiers comprises a residual block sequence, and at least one residual block of the residual block sequence comprises a first batch normalization layer with a first Swish activation function, a first convolutional layer after the first batch normalization layer with the first Swish activation function, a second batch normalization layer with a second Swish activation function, a second convolutional layer after the second batch normalization layer with the second Swish activation function, and a squeeze and excitation layer.

[0188] 21a. The non-transitory computer-readable medium of clause 14a, wherein the instructions further cause the processor to perform the step of generating the reweighting factor by computing a quotient of a difference between a probability output by the one or more classifiers and 1.

[0189] 22a. The non-transitory computer-readable medium of clause 14a, wherein the second component is implemented by at least one of a prior network or a Gaussian distribution.

[0190] 23a. The non-transitory computer-readable medium of clause 14a, wherein the first component is implemented by at least one of an encoder portion of a variational autoencoder, a numerical inversion applied to a generator network included in a generative adversarial network, or an inversion of a decoder included in a normalization flow network.

[0191] Any and all combinations of any of the claimed elements and / or any of the elements described in this application fall within the intended scope of the application and protective scope.

[0192] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0193] Aspects of the embodiments can be implemented as a system, a method, or a computer program product. Accordingly, aspects of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "module", "system" or "computer", whichever is appropriate. Furthermore, any of the hardware and / or software technology, processes, functions, components, engines, modules or systems described in this disclosure can be implemented as circuitry or a set of circuits. Also, aspects of the disclosure can take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0194] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include the following: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that may include or store a program for use with or in combination with an instruction execution system, apparatus, or device.

[0195] The foregoing description of various aspects of this disclosure refers to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When the instructions are executed via a processor of a computer or other programmable data processing apparatus, the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams can be implemented. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, dedicated processors, or field-programmable gate arrays.

[0196] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or code, including one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions indicated in the blocks may not appear in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function or action.

[0197] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure can be devised without departing from the basic scope thereof, the scope thereof being determined by the claims that follow.

Claims

1. A computer-implemented method for generating images using a variational autoencoder, the method comprising: determining one or more first values of a set of visual attributes included in a plurality of training images, wherein the set of visual attributes has been encoded via a prior network; applying a reweighting factor to the one or more first values in order to generate one or more second values for the set of visual attributes, wherein the one or more second values represent the one or more first values shifted towards one or more third values of the set of visual attributes, wherein the one or more third values have been generated via an encoder network, the reweighting factor being generated based on one or more classifiers for distinguishing between values sampled from the set of visual attributes and values of a latent variable generated via the encoder network; and performing one or more decoding operations on the one or more second values via a decoder network to generate a new image not included in the plurality of training images.

2. The computer-implemented method of claim 1, wherein the new image includes at least one face.

3. A computer-implemented method for generating images using a generative model, the method comprising: sampling one or more first values from a distribution of a latent variable learned by a prior network included in the generative model, wherein the latent variable is transformed from a plurality of training images; applying a reweighting factor to the one or more first values in order to generate one or more second values for the latent variable, wherein the reweighting factor is generated based on one or more classifiers for distinguishing between values sampled from the distribution and values of the latent variable generated via an encoder network included in the generative model; and performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to generate a new image not included in the plurality of training images.

4. The computer-implemented method of claim 3, further comprising: training the one or more classifiers based on a binary cross-entropy loss.

5. The computer-implemented method of claim 3, wherein prior to training the one or more classifiers, the prior network, the encoder network, and the decoder network are trained using the plurality of training images.

6. The computer-implemented method of claim 3, wherein the distribution of a latent variable learned by the prior network includes a hierarchy of latent variables, and sampling the one or more first values comprises: sampling a first value from a first set of latent variables included in the hierarchy of latent variables; and sampling a second value from a second set of latent variables included in the hierarchy of latent variables based on the first value and a feature map.

7. The computer-implemented method of claim 6, wherein the one or more classifiers include a first classifier that distinguishes between a third value sampled from the first set of latent variables using the prior network and a fourth value of the first set of latent variables generated by the encoder network, and a second classifier that distinguishes between a fifth value sampled from the second set of latent variables using the prior network and a sixth value of the second set of latent variables generated by the encoder network.

8. The computer-implemented method of claim 3, wherein applying the reweighting factor to the one or more first values comprises: re-sampling the one or more first values based on an importance weight that is proportional to the re-weighting factor.

9. The computer-implemented method of claim 3, wherein applying the reweighting factor to the one or more first values comprises: iteratively updating the one or more first values based on a gradient of an energy function associated with the distribution and the re-weighting factor.

10. The computer-implemented method of claim 9, wherein the energy function includes a difference between the distribution and the re-weighting factor.

11. The computer-implemented method of claim 3, wherein the re-weighting factor is generated by computing a quotient of a difference between a probability output by the one or more classifiers and one.

12. The computer-implemented method of claim 3, wherein at least one of the one or more classifiers includes a residual neural network.

13. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: sampling one or more first values from a distribution of latent variables learned by a previous component included in a generative model, wherein the latent variables are transformed from a plurality of training images; applying a re-weighting factor to the one or more first values in order to generate one or more second values for the latent variables, wherein the re-weighting factor is generated based on one or more classifiers that distinguish between values sampled from the distribution and values of the latent variables generated via an encoder network included in the generative model; and performing one or more decoding operations on the one or more second values via a decoder network included in the generative model to generate a new image that is not included in the plurality of training images.

14. The non-transitory computer-readable medium of claim 13, wherein the instructions further cause the processor to perform the steps of: training the generative model based on the plurality of training images during a first training phase; and after completion of the first training phase, training the one or more classifiers to distinguish between values sampled from the distribution and values of the latent variables generated via an encoder network during a second training phase.

15. The non-transitory computer-readable medium of claim 14, wherein the one or more classifiers are trained based on a binary cross-entropy loss.

16. The non-transitory computer-readable medium of claim 13, wherein sampling the one or more first values includes: sampling a first value from a first set in a hierarchy of latent variables learned by a prior network that implements the previous component; and sampling a second value from a second set in the hierarchy of latent variables learned by the prior network. sampling a second value from a second group in the hierarchy of latent variables based on the first value and a characteristic map.

17. The non-transitory computer-readable medium of claim 16, wherein the one or more classifiers comprise a first classifier and a second classifier, the first classifier distinguishing between a third value sampled from the first group and a fourth value of the first group generated by the encoder network, the second classifier distinguishing between a fifth value sampled from the second group and a sixth value of the second group generated by the encoder network.

18. The non-transitory computer-readable medium of claim 13, wherein the one or more classifiers comprise a convolutional layer and one or more residual blocks.

19. The non-transitory computer-readable medium of claim 18, wherein the one or more residual blocks comprise: a first batch normalization layer having a first Swish activation function, a first convolutional layer after the first batch normalization layer having the first Swish activation function, a second batch normalization layer having a second Swish activation function, a second convolutional layer after the second batch normalization layer having the second Swish activation function, and a squeeze and excitation layer.

20. The non-transitory computer-readable medium of claim 13, wherein the previous component is implemented by at least one of a prior network or a Gaussian distribution.

21. The non-transitory computer-readable medium of claim 13, wherein the decoder network is implemented by at least one of a generator network included in a generative adversarial network, a decoder portion of a variational autoencoder, or an invertible decoder represented by one or more normalization flows.

22. The non-transitory computer-readable medium of claim 13, wherein the encoder network is implemented by at least one of an encoder portion of a variational autoencoder, a numerical inversion applied to a generator network included in a generative adversarial network, or an inversion of a decoder included in a normalization flow network.

Images