Committed Information Rate Variational Autoencoder

Abstract

A variational autoencoder (VAE) neural network system, comprising: an encoder neural network for encoding input data items to define a posterior distribution of a set of latent variables; and a decoder neural network for generating output data items representing values ​​of the set of latent variables sampled from the posterior distribution. The system is configured for training with an objective function that includes a term that depends on the difference between the posterior distribution and the prior distribution. The prior distribution and the posterior distribution are arranged so that they cannot match each other. The VAE system can be used to compress and decompress data.

Description

Background Art

[0001] This specification relates to variational autoencoder neural network systems and their components.

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0003] In a broad sense, a variational autoencoder (VAE) determines the distribution of a set of latent variables representing an input data item x. The encoder therefore determines the parameters of a posterior distribution q(z|x) over the latent variable z. The VAE is trained with the goal of encouraging the system to keep the posterior distribution close to the prior p(z), which is usually a standard Gaussian. Samples can be taken from this distribution to generate output data items. The VAE can be trained using unlabeled data. The goal can include a term that measures the difference between the posterior distribution and the prior distribution, such as the KL (Kullback-Leibler) divergence. However, a problem with some VAE implementations, particularly those with a powerful decoder, is that the decoder may be able to generate output data items without relying on latent variables.

[0004] In one innovative aspect, a variational autoencoder neural network system is provided. The system can be implemented as a computer program on one or more computers in one or more locations.

[0005] The system may include an input for receiving an input data item such as an image, digitized sound, or text from a camera or LIDAR system, for example. The system may also include an encoder neural network configured to encode the input data item to determine a set of parameters defining a first, posterior distribution of a set of latent variables. The system may also include a subsystem for sampling from the posterior distribution to determine the value of the set of latent variables. The system may also include a decoder neural network configured to receive the value of the set of latent variables and generate an output data item representing the value of the set of latent variables. The variational autoencoder neural network system may be configured to train according to an objective function. This may have a first term, such as a cross entropy term, that depends on the difference between the input data item and the output data item, and a second term, such as a KL divergence term, that depends on the difference between the posterior distribution of the set of latent variables and the second, prior distribution. The prior distribution may be different from the posterior distribution; more particularly, the structure of the prior distribution may be different from the structure of the posterior distribution, so that the posterior distribution cannot match the prior distribution.

[0006] In an embodiment, the information transferred from the encoder to the decoder can be represented as a non-zero divergence between the posterior distribution and the prior distribution. The divergence can be the KL divergence, the Jensen-Shannon divergence, or some other difference measure.

[0007] By imposing different structures on the posterior and prior distributions, the second term of the objective function can be guaranteed to be non-zero, and thus the decoder can be forced to rely on the latent variables when generating output data items.

[0008] The structure of a distribution may be determined by its intrinsic shape, which may be determined, for example, by its mathematical form, and / or it may be determined by parameters of the distribution. Thus, by fixing one or more parameters of the distributions to be different from each other, the posterior distribution and the prior distribution may be defined to have different structures. For example, where the distributions have the same or similar mathematical form, the distributions may be constrained to be structurally different from each other by constraining one or more parameters of each distribution to have different relative or absolute values.

[0009] For example, the posterior distribution and the prior distribution may each comprise a multivariate Gaussian distribution. The variance of the posterior distribution may then differ from the variance of the prior distribution by a factor α (where α≠1). This structure for the posterior distribution and the prior distribution has the advantage of being simple; it also facilitates determining the KL divergence term in a closed form, for example during training. In some embodiments, the parameter α can be determined (in closed form) based on the desired commitment, i.e., the minimum, information rate transferred from the encoder to the decoder via the latent variable.

[0010] It is desirable to be able to use autoregressive neural networks as decoders, because such decoders can be very powerful, i.e., they can be able to generate very accurate samples of data items. An example of such a system that can be used, for example, to generate images is described in Aaron van den Oord et al., “Conditional Image Generation with PixelCNN Decoders”, arXiv:1606.05328 and in S. Reed et al., “Few-Shot Autoregressive Density Estimation: Towards Learning To Learn Distributions”, arXiv 1710.10304. An example of such a system that can be used, for example, to generate sounds (waveforms) is described in A. van den Oord et al. "WaveNet: A generative model for raw audio" arXiv 1609:03499. In this context, an autoregressive neural network may be configured to generate an output data item value x t A neural network with a sequence of values, each of which has a value x t With the previously generated output data item value x <t and condition variable s as condition: The data item value may be, for example, a pixel value or a sound signal value. For example, when generating an image, an autoregressive neural network may generate pixels sequentially, for example row by row and pixel by pixel in a raster scan.

[0011] However, when such a powerful decoder is incorporated into a VAE, the latent variables can be ignored. In an embodiment, the above-described variational autoencoder neural network system facilitates the use of such an autoregressive neural network system as a decoder while still using information from the latent variables. More specifically, the autoregressive structure of the decoder can provide local structure to the output data items, while the decoder output can also be conditioned on the information provided by sampling from the latent variable distribution, which can provide longer-range structure and / or global structure to the output data items.

[0012] Therefore, in some embodiments, the encoder is configured to determine a sequence of parameter sets, the sequence of parameter sets defining a distribution sequence of a sequence of latent variable sets, one latent variable set for each of a plurality of time steps. Here, the encoder time step may be different from the decoder sample generation step. For example, the interval between consecutive latent variable sets may be longer than the decoder time step, i.e., more than one output data item value may be generated for each latent variable set.

[0013] Thus, in some embodiments, a sequence of sets of latent variables is generated. The prior distribution of the latent variables may then include an autoregressive distribution, such that at each time step the prior distribution depends on the prior distribution at the previous time step. However, the posterior distribution may be determined independently at each time step. This approach can help capture correlations in the latent space from one time step to another (where a time step may, for example, correspond to a spatial position in an image). The autoregressive process that defines the evolution of the prior distribution of the latent variables over time may be a linear autoregressive process. For example, the value z of the set of latent variables at time step t t It can be calculated by multiplying α by the value z of the latent variable set at the previous time step t-1 and a noise component, such as a Gaussian noise component, where |α| < 1. The parameter α defines the degree of temporal correlation in the latent variable, where the correlation is lower as α approaches zero.

[0014] In the case where the decoder neural network is an autoregressive neural network, the VAE may also include a system for restricting the values ​​of the set of latent variables passed to the decoder at each time step to those values ​​that encode information about the sequence of output data values ​​that are still to be generated, i.e., about the future values ​​of the sequence of output data values. Thus, it is possible to obtain the value of the latent variable for x from >tThe values ​​of the input data items derive the values ​​of the set of latent variables passed to the decoder. This is because the autoregressive neural network has been able to effectively access information about the past, i.e., previously generated values ​​of the output data items. In an embodiment, the encoder can be configured to limit the values ​​of the set of latent variables passed to the decoder in this way. The encoder can therefore be configured to have an "anti-causal" structure. When a VAE system is being trained, the system will typically be able to access the complete data items, and will therefore be able to access values ​​of data items that are later than those being generated by the decoder at any particular decoder time step, so causality is not violated. This can promote computational efficiency and can also allow for increased learning rates.

[0015] The decoder may include a convolutional autoregressive neural network configured to implement causal convolution, i.e., in which generated data item values ​​depend on previously generated data item values, but not on future data item values. Implementing causal convolution may be implemented using a mask to mask the inputs of data item values ​​from those data item values ​​that follow in the sequence at the current time step, or by shifting the convolution position by (filter length - 1) time steps.

[0016] Although the system can be trained to match the posterior distribution and the prior distribution, there is a built-in mismatch in encoding information. The VAE system can therefore include a system configured to learn a distribution sequence of sequences of latent variables - i.e., configured to learn an approximate (aggregate) posterior distribution - and in particular an auxiliary prior neural network. The auxiliary prior neural network can be an autoregressive neural network and can be trained simultaneously with the encoder neural network and the decoder neural network.

[0017] Also provided is a method for training a variational autoencoder neural network system as described above, which may be unsupervised. The method may include: receiving training data, the training data including training data items; providing each training data item to an input of the variational autoencoder neural network system to generate a corresponding output data item; and determining a gradient of an objective function based on a difference between the training data item and the corresponding output data item and based on a difference between a posterior distribution and a prior distribution of a set of latent variables. The training data items may be processed in batches. The method may also include back-propagating gradients through the variational autoencoder neural network system to adjust parameters of the encoder neural network and the decoder neural network to optimize the objective function.

[0018] In a broad sense, training can employ stochastic gradient descent (SGD), whose objective includes the reconstruction cost and the closed form KL divergence term. The "reparameterization trick" can be used to back-propagate the gradients through the decoder into the encoder (see, for example, Kingma and Welling 2013, arXiv1312.6114 (Kingma and Welling 2013, arXiv1312.6114)), where the sampling nodes are replaced with deterministic operations with noisy inputs to allow gradients to flow through the nodes. The objective function defining the built-in difference between the posterior distribution and the prior distribution ensures the (minimum) rate of information flow from the encoder to the decoder via the latent variable.

[0019] A VAE system during training includes both an encoder and a decoder. However, once trained, each of these can have independent utility.

[0020] For example, because the latent variable distribution effectively defines a compressed version of the input data items, the encoder can be used to compress data items of the same type as used to train the VAE system. In another example, the encoder can be used as a front end for another machine learning system, such as a classifier. Because the encoder has learned the distribution of the training data items, the classifier trained / operated on the latent variables can perform better than the classifier trained on the original data items. In another example, the encoder can be used as a front end for a reinforcement learning (RL) system, where the learned latent variable distribution is used to represent an image of the environment in which the RL system operates and / or to encode other sensor data such as data representing the state of a mechanical agent, such as the configuration of a robotic arm. Although the examples have been described using sound and image data, the VAE system can also be trained on video data, so the trained encoder can encode or compress the video data.

[0021] The decoder of the trained system can also have independent utility. For example, samples can be taken from the prior and provided to the decoder to generate sample output data items. In a system with an auxiliary neural network, samples can be provided to the auxiliary neural network to generate a sequence of latent variables, which can then be provided to the decoder to generate sample output data items. The generated data items can be one-dimensional data items such as sound, in which case the latent variables can have a 1D feature space and the data item values ​​can include values ​​that define the sound waveform. The generated data items can be two-dimensional data items such as images, in which case the latent variables can have a 2D feature space and the data item values ​​can include pixel values ​​of the image, such as brightness values ​​and / or color values. The generated data items can be three-dimensional data items such as image sequences (videos), in which case the latent variables can have a 3D feature space and the data item values ​​can include pixel values ​​of the image sequence (video). In the case of video, the decoder can be an image decoder. For example, the VAE system can be trained to generate images conditioned on additional data inputs that define the movement / change of the viewpoint, and then a sequence of sets of latent variables can be generated by applying a series of such additional data inputs, each image being generated independently of the corresponding set of latent variables. In another example, the data items may represent the structure of a molecule or other biological entity, such as a protein molecule, and the decoder may be used to generate an output data item, such as a candidate drug molecule, having similar properties to the training data items.

[0022] More generally, encoders and decoders can be configured to be conditioned on additional data inputs, such as labels and / or text for generating images, or labels and / or text and / or speaker identifiers for generating audio, for example in a text-to-speech system. The trained decoder can be used to generate output data items of a desired type, for example, for generating a waveform representing speech from natural language text.

[0023] Input and output data items can generally be any kind of digital data items.Therefore in another example, input data items and output data items can each include a tag (token) defining a sentence in a natural language.The encoder and / or decoder can then be used, for example, in a system for machine translation or for generating sentences representing concepts expressed with latent variables and / or additional data.Latent variables can be used additionally or alternatively to control the style or emotion of the generated text.In still other examples, input and output data items can generally include voice, video or time series data.

[0024] In still other applications, the trained VAE system and / or encoder / decoder can be used for image or other data item processing tasks, such as completing a task by generating or filling in missing parts of a data item with an image or other data item.

[0025] In another example, a VAE system can be used to make personalized recommendations for a user. For example, a latent variable can be used to characterize a user's preferences for data items. For example, in the case of training a system using data items including identifiers of items / content that a user has selected, such as a playlist, the distribution of the latent variable can represent features that characterize the user's preferences. Sampling from the distribution of the latent variable can then be used to generate a recommendation output data item.

[0026] In yet another example, the VAE system, in particular the trained decoder, can be used to generate additional examples of data items for training another machine learning system. For example, the VAE system can be trained on a set of data items, and then a set of latent variables can be determined and used to generate new data items similar to the data items in the training data set. The set of latent variables can be determined by sampling from the (prior) distribution of the latent variables and / or using an auxiliary neural network. In the case where the VAE system has been trained with additional data, such as labels, it is possible to generate new data items with additional data, such as labels provided to the decoder. In this way, additional labeled data items can be generated, for example, to supplement the lack of unlabeled training data items.

[0027] Therefore, a method for obtaining an encoder / decoder includes: training the variational autoencoder neural network system as described above; and then using the trained encoder / decoder neural network as an encoder / decoder.

[0028] As mentioned above, a trained decoder and a trained encoder are also provided.

[0029] As described above, a method for encoding and / or decoding data using a trained encoder and / or decoder is also provided.

[0030] For example, in one embodiment the autoregressive decoder includes a causal convolutional neural network configured to generate a data item by generating, at each iteration of a plurality of iterations, a value of the data item conditioned on a value of a previously generated data item at a previous iteration, wherein generation queries a memory including values ​​derived from a set of latent variables at each time step using a soft attention query vector that depends on the previously generated values ​​of the data item.

[0031] For example, in one embodiment the anti-causal autoregressive encoder includes a causal convolutional neural network that is configured to input a data item, reverse the order of values ​​of the data item, and generate a representation of the data item from the reversed order values.

[0032] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following additional advantages.

[0033] Some embodiments of the system are capable of using a powerful decoder such as an autoregressive decoder to generate examples of output data items with improved fidelity, i.e., improved accuracy and more details. In particular, embodiments of the VAE system can be trained even when the decoder is implemented using autoregressive techniques. The system is also potentially capable of learning latent variable distributions that provide improved representations of training data items. This in turn promotes advantages such as reduced memory usage and better control over the generated data items. In the case where, for example, the system is used for data compression, this can promote a greater degree of data compression. Some embodiments of the system reduce the computational power required for training because they can be trained faster and better than previous systems. Some embodiments of the system are capable of effectively modeling time series data such as speech and video. As described above, some embodiments of the system allow efficient calculation of convergent closed objective functions. Some embodiments of the system allow tuning of the minimum information rate between the encoder and the decoder with a parameter; this parameter can be adjusted according to the type of data items being processed, such as the degree of correlation in the data items, to tune the effectiveness of the system in terms of computing resources and memory usage.

[0034] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An example variational autoencoder (VAE) neural network system is shown.

[0036] Figure 2a-2b Illustration of the operation of the anti-causal encoder neural network and the autoregressive decoder neural network.

[0037] Figure 3 Shown for Figure 1 Flowchart of an example training process for a VAE neural network system.

[0038] Figure 4a to Figure 4b A flow chart illustrating encoding a data item using a trained encoder neural network and generating a data item using a trained decoder neural network is shown.

[0039] Figure 5 Details of an example implementation of a VAE neural network system are shown.

[0040] Like reference numerals and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0041] This specification generally describes a variational autoencoder (VAE) neural network system implemented as a computer program on one or more computers in one or more locations.

[0042] When trained, the VAE neural network system includes a trained encoder neural network and a trained decoder neural network. The encoder neural network learns to compress data from the training data distribution into a simpler distribution represented by a set of latent variables. The decoder neural network learns to decode the set of latent variables into examples obtained from a distribution that approximates the training distribution. Therefore, the trained VAE neural network system operates as a data compression / decompression system, where the encoder neural network acts as a data compressor and the decoder neural network acts as a complementary data decompressor.

[0043] During training, the VAE learns the structure in the training data and is therefore able to perform efficient data compression / decompression. The data may be any type of data, including, for example, image data (including video), sound data, and text data.

[0044] The latent variable representations from the encoder neural network can also be used, for example, for classification and reinforcement learning tasks. The data from the decoder neural network can be used to generate examples from a training data distribution, such as examples of image data, sound data, or text data, conditioned on the set of (latent) variables provided as input, and optionally also on labels defining the type of the example, such as image content for image data or natural language words for sound data.

[0045] The problem with VAE neural networks is that they may not be able to be trained efficiently due to posterior collapse, which is explained below. Another problem is that while they may be good at capturing global structure in the training data, they may not be able to capture more complex local structure, for example the generated images may look blurry. The solution to the latter problem is to use a more powerful generative model for the decoder, i.e., a model that is better able to reproduce details in the training data distribution such as image details, but this exacerbates the posterior collapse problem.

[0046] A VAE objective function aims to maximize the probability p(x) of obtaining a training data example x from the decoder neural network (generative model) by maximizing a lower bound on this probability Here D KL is the Kullback-Leibler divergence; p(z) defines the prior distribution of z, which should be continuous but can be arbitrary in other ways and can be, for example, a standard Gaussian And the encoder neural network is represented by the function q(z|x), i.e. it defines the parameters of the approximate posterior distribution for each component of z. The encoder neural network therefore defines the parameters of the multivariate distribution for z from which samples z~q(z|x) are taken and provided to the decoder neural network represented by p(x|z). During training, the second term can become zero, that is, the probability that the decoder produces examples that match the training data distribution examples can be maximized without using latent variables, so-called posterior failure. This is particularly problematic in the case of powerful decoders, such as autoregressive decoders from the PixelCNN or WaveNet series (described above) that are able to generate output examples with fine details that model the complete data distribution without any conditioning input.

[0047] Figure 1 An example variational autoencoder (VAE) neural network system 100 that solves this problem is shown. The VAE neural network system 100 can be implemented as a computer program on one or more computers in one or more locations.

[0048] The VAE neural network system 100 is provided with training data items, for example from a data store 102. These may include audio data items, for example digitized audio in the time and / or frequency domain, or image data items including, for example, video and / or LIDAR data items, text data items, data items defining the structure of entities such as molecules, or any other type of data items.

[0049] The data items are provided to an encoder neural network 104, which outputs a set of parameters 106 defining a posterior distribution of a set of latent variables, such as the mean and variance of a multivariate Gaussian distribution. The system is configured to sample values ​​of a set of latent variables 108 from the posterior distribution. The set of latent variables may define values ​​of a latent variable data structure such as a latent variable vector z.

[0050] The latent variables are processed using a decoder neural network 110 that generates data item outputs 112. In some embodiments the decoder neural network 110 generates data items directly; in other embodiments it generates parameters of a distribution of output data items that are sampled to obtain example output data items. For example, the decoder output may specify the parameters of a distribution of the intensity of each pixel (or color sub-pixel) of an image or the value of each individual audio sample of a sound.

[0051] The training engine 120 is configured to train the VAE neural network system 100 by back-propagating the gradient of the objective function to update the neural network parameters 122 of the encoder neural network 104 and the decoder neural network 110. The training engine uses the prior distribution parameters 124 of the prior distribution of the latent variable set together with the posterior distribution parameters to determine the divergence loss term of the objective function. The training engine 120 also determines the reconstruction loss term of the objective function, such as log p(x|z), based on the input data items and the corresponding output data items, which aims to match the distribution of the output data items with the distribution of the training data items.

[0052] In some embodiments, an auxiliary prior neural network 126 is provided to determine the parameters of an auxiliary prior distribution, as described later. The auxiliary prior neural network 126 (if present) can also be trained by the training engine 120.

[0053] The log p(x|z) term of the objective function described previously is the reconstruction term and D KL The term encourages the system to keep the posterior distribution close to the prior distribution. KL The term measures the information flow from the encoder to the decoder, more specifically the number of nats required on average to be sent from the encoder to the decoder via the latent variable. In an embodiment, the training engine 120 is configured to use an objective function for which there is a guaranteed, non-zero minimum rate, δ, of information flow from the encoder to the decoder, referred to herein as the "committed rate". The rate of information flow δ is constrained to be equal to or greater than the committed or minimum rate to suppress a posteriori failures. For example, the a posteriori-prior divergence term of the objective function, such as D KL (q(z|x)||p(z)), ≥δ.

[0054] In an embodiment this may be achieved by defining a structure of a (continuous) prior distribution p(z) that is different from the structure of the posterior distribution q(z|x) such that the posterior distribution cannot match the prior distribution. The structure of the distribution may be defined by the intrinsic shape of the distribution - e.g. whether it is a Gaussian distribution - and / or by constraining one or more parameters that define the shape of the distribution - e.g. by defining the variance of the Gaussian distribution. In an embodiment, the committed rate δ of the information flow is not fixed, but may be different for each data item, allowing the system to allocate more bits to more complex input data items. Metrics other than the Kullback-Leibler divergence may be used for the difference between p(z) and q(z|x), such as the Jensen-Shannon divergence. Some example distribution structures are described later.

[0055] The application of these techniques does not depend on any particular form or architecture of the encoder or decoder neural network. In particular, when using a neural network such as an autoregressive decoder - i.e., a decoder configured to generate an output data item value x t The decoder of the sequence, each output data item value x t With the previously generated output data item value x <t These techniques avoid a posteriori failure when constructing a robust decoder for conditional learning.

[0056] Figure 1 The latent variables in the system can capture and represent global characteristics of data items. However, data items may include spatial, temporal or other sequential changes. For example, images usually have some spatial continuity, speech usually has some temporal continuity, and natural language text usually has some sequential continuity. It may also be advantageous to capture such finer, often shifted attribute changes in latent variables as image texture or posture, voice pitch, volume or stress, and language style or emotion.

[0057] The VAE neural network system 100 can therefore be configured to model data items using a sequence of latent variables taken from a sequence of prior distributions. Each data item can therefore be subdivided into a sequential set of data item parts. As an example, for image data items, these parts may include rows, columns, or regions of an image; for video data items these parts may include individual images; for speech data items these parts may include words; for natural language text data items these parts may include tokens (e.g., words, phrases, symbols, etc.). The steps in the sequence may be referred to as time steps, for example when the data item parts are processed sequentially.

[0058] The same posterior distribution model, such as a (diagonal) multivariate Gaussian distribution, can be used for each step of the sequence. That is, the same encoder neural network can process each of the data item portions. However, the prior distribution can change with each step in a related manner. More specifically, the sequence of prior distributions can be an autoregressive sequence, such as a first-order and / or linear autoregressive sequence. Therefore, at each time step, the prior distribution can depend on the prior distribution at the previous time step. This produces a mismatch in the correlation structure between the prior distribution and the posterior distribution, which produces a positive lower bound on the divergence between these distributions.

[0059] For example, at time step t, the latent variable vector z t Can be z t =αz t-1 +∈ t Definition, where ∈ t is a noise component, e.g. with zero mean and constant variance σ ∈ 2Gaussian noise, and |α|<1 (so that the prior distribution has constant enough statistics throughout its time evolution). Then z t has zero mean and variance and be able to make choices Make This facilitates the determination of analytical forms for the committed minimum rate of information flow, such as D KL The lower limit on (q(z|x)||p(z)). Therefore, during training, the optimization process is allowed to stabilize at a rate higher than the minimum, but the optimization process is restricted to be lower than this rate. The latent variable vector z t Different dimensions of can, but need not, have different values ​​of α.

[0060] In an embodiment, the committed information rate is given by:

[0061]

[0062] where n is the length of the sequence, d is the dimension of the (multidimensional) latent variable vector, and α k is the value of α to use for each dimension. This allows either α or α k A choice is made to achieve a target value for δ. In a broad sense the value of α determines the degree of correlation between the prior distributions from one step to the next - there is no correlation at α = 0, and as α approaches unity the correlation increases. This can be viewed as a variable rate signature analysis, where different values ​​of alpha correspond to different rates of change.

[0063] In embodiments where the decoder comprises an autoregressive neural network, the decoder neural network is in principle able to take into account previous portions x of the output data items it has generated. ＜t Accurately estimate the output data term x t The latent variable set therefore does not need to transmit this information to the decoder neural network. Therefore, in some embodiments, the encoder neural network can have an anti-causal structure, that is, an anti-causal structure in which the latent variable set does not encode information about the part of the output data item that has already been generated. For example, the latent variable set can encode information about the current part of the data item in the sequence and about those parts that still need to be generated.

[0064] This is schematically illustrated in Figure 2a middle, Figure 2a The operation of the anti-causal inference model, i.e., the anti-causal encoder neural network, is shown as follows: at the first time step, using the 1 、x 2 and x 3 The information is used to generate z 1 ; In the second time step, use the 2 and x3 The information is used to generate z 2 ; and in the third time step, using 3 The information is used to generate z 3 In practice x 1 、x 2 and x 3 Anti-causal dependencies of the encoder neural network can be defined by appropriate selection of parts of the data items, for example via masks and / or by reversing the sequential order of the parts of the data items. Figure 2b Schematically illustrating the autoregressive nature of the corresponding generative model, namely the autoregressive decoder neural network.

[0065] The use of anti-causal encoder neural networks can also help avoid posterior failure and can also improve the data compression of the system by removing potentially redundant information from the set of latent variables.

[0066] Once trained, the encoder neural network 104 can be used as a data compression engine because the set of latent variables encodes meaningful information about both global and finer features of the input data items. However, the fine details can be produced by the autoregressive decoder neural network 110. This approach is able to achieve high data compression ratios.

[0067] The trained decoder neural network 110 may be used to generate example data items by sampling a set of latent variables from the prior distribution and providing the set to the decoder neural network 110 .

[0068] In the whole training data items Between the prior distribution p(z) and the aggregated posterior distribution If there is a significant mismatch between the prior distribution and the aggregated posterior distribution, there may be regions of the prior distribution that the decoder neural network cannot see during training. Therefore, the VAE neural network system 100 may include an auxiliary prior neural network 126 that is trained to output an auxiliary distribution p that matches the aggregated posterior distribution. aux The auxiliary prior neural network 126 can be trained as an encoder neural network and a decoder neural network at the same time, but does not participate in the training of the encoder neural network and the decoder neural network. However, after training, the latent variable set can be sampled from the auxiliary prior distribution instead of the prior distribution.

[0069] The autoregressive model can be used to estimate p aux For example, a single-layer LSTM (Long Short-Term Memory) neural network can be used to estimate the posterior distribution q(z) of the i-th latent variable conditioned on the previous latent variable samples. i |x) parameters, where p aux (z i|z< i ) can be a Gaussian distribution with mean and variance parameters output by the LSTM after processing the previous latent variable sample for step i. This can be done by letting D KL (q(z|x)||p aux (z)) is minimized to train the LSTM neural network.

[0070] In some embodiments, it can be modified Figure 1 The VAE neural network system 100 is configured such that the decoder neural network 110 generates output data items from labeled classes, such as example images of specific digits. This can be achieved by adding conditional inputs to the encoder neural network and the decoder neural network during training. The conditional input can be, for example, a one-hot conditional vector c that identifies the class to which the training data item belongs. The corresponding conditional input can then be provided to the decoder neural network 110 to generate example output data items of the specified class.

[0071] Figure 3 is a flow chart of an example training process for the VAE neural network system 100. Encoder neural network and decoder neural network parameters are initialized, for example randomly (step 300), and training data items are obtained, for example, from the data store 102 (step 302). The training data items are processed using the encoder neural network 104 to obtain parameters defining a posterior distribution (step 304), and a set of latent variables are sampled from this distribution (step 306). The set of latent variables is then processed by the decoder neural network 110 to obtain output data items directly or, for example, by sampling from a multivariate distribution parameterized by the output of the decoder neural network (step 308). The process then backpropagates the gradients of the objective function of the type previously described to update the parameters of the encoder neural network and the decoder neural network. Any suitable backpropagation method can be used, such as Adam. In order to backpropagate through the latent variable sampling, a "reparameterization trick" can be used, whereby the sampling operation for each latent variable is rewritten as z=μ+σ∈, where ∈ is standard Gaussian noise. The encoder neural network generates the mean (μ) and variance (σ) of the distribution defining the latent variable. 2 ), but with random (sampled) elements provided by noise, enabling the gradients to be back-propagated through the “sampling” step. The gradients can be averaged over mini-batches; they can be repeated Figure 3 The process continues until the neural network parameters converge.

[0072] The gradient of the objective function of the auxiliary prior neural network is optionally back-propagated through the auxiliary prior neural network to train this neural network in parallel with the main VAE system.

[0073] The objective function can have the general form log p(x|z)-D KL(q(z|x)||p(z)). The reconstruction loss term log p(x|z) can be evaluated from the training data item and the output data item, for example, by determining a cross entropy loss or an MSE (mean square error) loss between the training data item and the output data item.

[0074] The second divergence term D of the objective function can be calculated based on the parameters of the prior distribution and the posterior distribution KL (q(z|x)||p(z)). As described previously, the structures of the prior and posterior distributions are different, making them unmatched.

[0075] For example, in one embodiment, the encoder neural network 104 may output a set of parameters 106 defining a multivariate Gaussian posterior distribution (with diagonal covariance) for a set of latent variables, and the prior distribution may be defined by a standard Gaussian distribution (i.e., zero mean, unit variance). The KL divergence of is given by:

[0076]

[0077] To calculate the KL divergence of the d-dimensional latent variable, the KL divergence of each of the d dimensions can be summed. In a simple implementation, the prior Gaussian distribution and the posterior Gaussian distribution can be defined to have different but fixed variances, for example, differing by a factor α, where α≠1, to constrain δ to be non-zero.

[0078] In some embodiments the prior distribution is defined similarly to a standard Gaussian distribution and the posterior distribution includes a distribution with mean μ q and variance and a multivariate Gaussian distribution with diagonal covariances, i.e., assuming that the latent variables in the set of latent variables are independent of each other. The posterior distribution is subject to a promised rate δ of information flow, while allowing rates higher than δ. This can be accomplished by constraining the mean μ of each latent variable (i.e., each component of the posterior distribution) according to q and variance To achieve

[0079]

[0080] This can be done for μ q and δ are solved numerically to obtain the feasible interval in Define the corresponding lower and upper values ​​of the variance. This in turn defines the mean μ of the posterior distribution in terms of the corresponding outputs μ(x) and σ(x) from the encoder neural network q and variance

[0081]

[0082]

[0083] In the case where the latent variable is taken from an autoregressive sequence with a prior distribution, the divergence term is given by:

[0084]

[0085] Among them, as previously described, is, for example, a standard Gaussian, and for t>1, z t is defined such that And the posterior distribution q(z t |x) includes t and variance and the multivariate Gaussian distribution of diagonal covariance

[0086]

[0087] where f(a)=a-ln(a)-1

[0088] Figure 4a A flow chart showing a process for encoding a data item using a trained encoder neural network, e.g., to generate a compressed representation of the data item. At step 400 the process obtains a data item, e.g., directly or indirectly from a sensor, from storage, or via a computer network or other communication link. The encoder neural network 104 then processes the data item and outputs a set of parameters defining a posterior distribution (step 402). A set of latent variables is then sampled from the posterior distribution (step 404), which represents an encoded, compressed representation of the data item.

[0089] Figure 4b A flow chart showing a process for generating data items using a trained decoder neural network. At step 450 the process samples a set of latent variables from a prior distribution, and then processes the set of latent variables using the decoder neural network 110 to generate data items (step 452). The training process allows valid example data items to be generated by randomly sampling from the prior, which may be a multivariate Gaussian distribution or may be an auxiliary prior.

[0090] Figure 5 An example implementation of a VAE neural network system 100 is shown that uses an autoregressive approach to implement both the encoder neural network and the decoder neural network. The techniques described in this specification are not limited to a particular form of encoder and decoder, but rather facilitate the use of a powerful, autoregressive decoder; when the decoder is autoregressive, the encoder need not be autoregressive.

[0091] exist Figure 5In the example of the decoder neural network 110 comprises a PixelCNN neural network (arXiv:1606.05328) or a variant thereof, such as PixelCNN++ (arXiv:1701.05517) or PixelSNAIL (arXiv:1712.09763). Thus, for example, the output of the neural network may define parameters that are sampled to obtain a mixture of logistic distributions (schematically shown) of pixel values, and the neural network may incorporate an attention layer, such as attention to the output of the encoder neural network. The encoder neural network may have the same architecture as the decoder neural network. The VAE neural network system 100 may be trained using dropout and layer normalization (arXiv:1502.03167).

[0092] As mentioned previously, anti-causal frameworks can be used to reduce posterior invalidation and encourage latent variables to be meaningful. Figure 5 An example of how anti-causal context can be implemented when processing images is shown. In this particular example, the pixel order of the input image is reversed and each spatial dimension is padded with one before the image is provided to the encoder. The output of the encoder is cropped and reversed again, giving each pixel anti-causal context, i.e., pooled information in terms of its own and future values ​​(as shown in curly brackets). Each encoded pixel value can be used as a latent variable, but for improved computational efficiency, pooling, such as average pooling, can then be applied row by row to encode each row as a multi-dimensional latent variable.

[0093] In an embodiment, a VAE neural network system can be trained to produce example data items of a particular type by conditioning each of the encoder neural network and the decoder neural network on additional conditional inputs during training. The conditional input may identify the type of the data item and may include, for example, a one-hot vector labeling the type of the data item or an embedding of the type of the data item. Thus the conditional input may identify, for example, a word or word fragment in a text-to-speech system, or an image label, description, viewpoint, and / or pose when generating images.

[0094] For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0095] The embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not a propagation signal, however.

[0096] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. The apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0097] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines or codes). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0098] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is different from the input. An engine can be a functional coding block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on a computing device of any suitable type including one or more processors and a computer-readable medium, such as a server, a smart phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. Additionally, two or more of these engines can be implemented on the same computing device or on different computing devices.

[0099] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by special purpose logic circuits, and the apparatus may also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). For example, the processes and logic flows may be performed by a graphics processing unit (GPU) and the apparatus may also be implemented as a graphics processing unit (GPU).

[0100] Computers suitable for executing computer programs include, as examples, a central processing unit that can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operationally coupled to receive data from one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or transfer data to the one or more large-capacity storage devices, or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0101] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0102] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0103] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0104] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0105] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0106] Similarly, although operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired effect. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0107] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0108] What is sought to be protected is:

Claims

1. A variational autoencoder neural network system, comprising: for receiving an input data item, wherein the input data item comprises one or more of: a still or moving image, an audio signal, and a representation of a text string; an encoder neural network configured to encode the input data item to determine a set of parameters defining a first, posterior distribution of a set of latent variables; a subsystem that samples from the posterior distribution to determine values ​​for the set of latent variables; a decoder neural network configured to receive the values ​​of the set of latent variables and to generate output data items representative of the values ​​of the set of latent variables; Wherein, the variational autoencoder neural network system is configured for training using an objective function having a first term depending on the difference between the input data item and the output data item and a second term depending on the difference between the posterior distribution and a second, prior distribution of the set of latent variables, wherein the structure of the prior distribution is different from the structure of the posterior distribution such that the posterior distribution cannot match the prior distribution.

2. The variational autoencoder neural network system according to claim 1, wherein: Each of the posterior distribution and the prior distribution comprises a multivariate Gaussian distribution, and wherein a variance of the posterior distribution differs from a variance of the prior distribution by a factor β, where β≠1.

3. The variational autoencoder neural network system according to claim 1, wherein: The encoder neural network is configured to determine a sequence of sets of parameters that define a sequence of distributions of a sequence of sets of latent variables, one set of latent variables for each time step of a plurality of time steps.

4. The variational autoencoder neural network system according to claim 3, wherein: The prior distribution comprises an autoregressive distribution such that the prior distribution at each time step depends on the prior distribution at a previous time step.

5. The variational autoencoder neural network system according to claim 4, wherein: The value of the latent variable set at time step t is defined by α times the sum of the value of the latent variable set at the previous time step and a noise component, where α defines the degree of temporal correlation in the latent variable set, and where |α|<1.

6. The variational autoencoder neural network system according to claim 3, wherein: The decoder neural network is an autoregressive neural network configured to generate a sequence of output data item values, each of which is conditioned on a previously generated output data item value; and further includes a second subsystem for restricting the values ​​of the set of latent variables passed to the decoder neural network at each time step to values ​​that encode information about the values ​​to be generated in the sequence of output data item values.

7. The variational autoencoder neural network system according to claim 3, further comprising an auxiliary neural network configured to learn the distribution sequence of the sequence of the latent variable set.

8. A method for training a variational autoencoder neural network system according to any one of claims 1 to 7, comprising: receiving training data, the training data comprising training data items; providing each training data item to the input of the variational autoencoder neural network system to generate a corresponding output data item; determining a gradient of the objective function based on a difference between the training data items and the corresponding output data items and based on a difference between a posterior distribution and a prior distribution of the set of latent variables; and The gradient is back-propagated through the variational autoencoder neural network system to adjust parameters of the encoder neural network and the decoder neural network to optimize the objective function.

9. A method according to claim 8 when dependent on claim 7, wherein: Providing each training data item to the input of the variational autoencoder neural network system to generate a corresponding output data item includes sampling from the posterior distribution to determine sampled values ​​of the set of latent variables; the method further includes using the sampled values ​​of the set of latent variables to simultaneously train the auxiliary neural network with the encoder neural network and the decoder neural network.

10. A method for obtaining a decoder, the method comprising training a variational autoencoder neural network system according to the method of claim 8 or 9, and then using the decoder neural network of the trained variational autoencoder neural network system to obtain the decoder.

11. A method for obtaining an encoder, the method comprising training a variational autoencoder neural network system according to the method of claim 8 or 9, and then using the encoder neural network of the trained variational autoencoder neural network system to obtain the encoder.

12. A system comprising one or more computers and one or more storage devices, the storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 8-11.

13. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the system of any one of claims 1-7 or perform the operations of any one of claims 8-11.

Images