Neural network system implementing conditional neural processes for efficient learning

Abstract

According to a first aspect, a network system for generating output data values from input data values according to one or more learned data distributions includes an input receiving a set of observations, each observation including a respective first data value of a first variable and a respective second data value of a second variable dependent on the first variable. The system may include an encoder neural network system configured to encode each observation in the set of observations to provide an encoded output for each observation. The system may further include an aggregator configured to aggregate the encoded output for the set of observations and provide an aggregated output. The system may also include a decoder neural network system configured to receive a combination of the aggregated output and the target input value, and provide a decoder output. The target input value may include a value of the first variable, and the decoder output may predict a corresponding value of the second variable.

Description

[0001] This application is a divisional application of the invention patent application with the application date of February 11, 2019, application number 201980009347.3, and invention name “Neural network system for realizing conditional neural processes for efficient learning”. Technical Field

[0002] The present specification relates to neural network systems that generate output data values. Background Art

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values ​​of a corresponding set of parameters.

[0004] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence based on the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from the previous time step to calculate the output at the current time step. An example of a recurrent neural network is a long short-term memory (LSTM) neural network that includes one or more LSTM blocks. Each LSTM memory block can include one or more cells, each cell including an input gate, a forget gate, and an output gate that allow the cell to store the previous state of the cell, for example, for generating the current activation, or to be provided to other components of the LSTM neural network. Summary of the invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations that generates output data values, such as predicted observations.

[0006] According to a first aspect, a network system for generating output data values ​​from input data values ​​according to one or more learned data distributions includes: an input that receives a set of observations, each observation including a corresponding first data value of a first variable and a corresponding second data value of a second variable that depends on the first variable. The system may include an encoder neural network system configured to encode each observation in the set of observations to provide an encoded output for each observation. The system may further include an aggregator configured to aggregate the encoded outputs for the set of observations and provide an aggregated output. The system may also include a decoder neural network system configured to receive a combination of the aggregated output and a target input value and provide a decoder output. The target input value may include a value of the first variable, and the decoder output may predict a corresponding value of the second variable.

[0007] Certain implementations of neural network systems can learn general information from a collection of observations, aggregate this knowledge, and then use the aggregated knowledge to facilitate further learning. The system can learn one or more distributions conditioned on a collection of input observations, for example, by embedding the observations and aggregating the embeddings, and then conditioning the data generator (i.e., decoder neural network system) on the aggregated embeddings.

[0008] The first and second data values ​​and the target input value may comprise a single data value or vector, for example, which defines an image or an audio segment. For these data / target values, there may be a single physical input to the system. For example, by using a mean function on the aggregator, the output of the aggregation may be invariant under permutations of observations in the set of observations. Permutation may be considered as a change in the order of values. Thus, the aggregator may be a symmetric aggregator. The aggregated parameters may have a predefined size. The size may be related to the number of bits and / or dimensions of the parameter.

[0009] The system may be configured to perform a set of operations for each of a plurality of training steps. The operations may include selecting a subset of observations from the observations to input to the encoder neural network system, and selecting a set of targets for training the system. The operations may further include encoding each observation in the subset of observations using the encoder neural network system to provide an encoded output for each observation. The operations may further include aggregating the encoded outputs. The operations may further include combining (e.g., concatenating) the aggregation of the encoded outputs with each first data value from the set of targets to provide a combined set for decoding. The operations may also include decoding each combination in the combined set using a decoder neural network system to provide a set of decoder outputs. The operations may further include training the encoder neural network system and the decoder neural network system by back propagation using a loss function that depends on the set of decoder outputs and the second data value of the set of targets.

[0010] The encoder and decoder may each comprise a feed-forward neural network, such as a multi-layer perceptron (MLP), and / or the encoder may comprise a convolutional neural network, such as when the input data comprises an image, and / or the decoder may comprise a deconvolutional neural network.

[0011] The decoder output may define parameters of the decoder output distribution. The system may then be configured to determine a corresponding value of the second variable based on the decoder output distribution. For example, the system may sample from the distribution or simply output the mean. For example, the decoder output distribution may define a mean and variance of the corresponding values ​​of the second variable. The mean and variance may be Gaussian distributed, and the Gaussian distribution may be used to determine the corresponding value of the second variable, and optionally determine the uncertainty of the second variable. Thus, the system may be configured to perform a regression task.

[0012] The regression task may be an image or sound completion task. For example, a first data value may include a spatial and / or temporal index (e.g., specifying the location of a sample (e.g., a pixel or audio value) in space and / or time). Then, a second data value may include an image or sound data value corresponding to the spatial and / or temporal index. Then, the target input value may define the spatial and / or temporal index of the image or sound data value to be generated.

[0013] In some other implementations, the system can be configured to perform a classification task. For example, the first data value can include image or sound data (e.g., defining a vector of data items including digitized image or sound data), and the second data value can include a classification of the image or sound data. The decoder output distribution can define a probability distribution over the classification (e.g., via training).

[0014] For classification tasks, for each class, the aggregation of encoded observations may include aggregation of encoded first data values ​​of observations in that class (e.g., via permutation invariant aggregation, such as mean calculation). The aggregated values ​​for each class may then be combined (e.g., via concatenation) to form an aggregated output (combined aggregated output) that is input into the decoder neural network system.

[0015] In some other implementations, the system may be configured to perform a text classification task.The first data value may include text data, and the second data value includes a classification of text or sound data.

[0016] In some other implementations, the system can be configured to control an agent that interacts with an environment. This can be achieved via reinforcement learning. In a broad sense, a reinforcement learning system is a system that selects actions to be performed by a reinforcement learning agent that interacts with an environment. In order for the agent to interact with the environment, the system receives data representing the current state of the environment and selects actions to be performed by the agent in response to the received data. In this specification, data representing the state of the environment is referred to as observation. Optionally, an observation at a time step may include data from a previous time step, for example, an action performed at a previous time step, a reward received at a previous time step, and so on.

[0017] In some implementations, the environment is a real-world environment, and the agent is an electromechanical agent that interacts with the real-world environment. For example, the agent can be a robot or other static or mobile machine that interacts with the environment to complete a specific task (e.g., locating an object of interest in the environment or moving an object of interest to a specified location in the environment or navigating to a specified destination in the environment); or the agent can be an autonomous or semi-autonomous land or air or sea vehicle that navigates through the environment.

[0018] In these implementations, observations may include, for example, one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment, such as sensor data from images, distance or position sensors, or sensor data from actuators.

[0019] In these implementations, an action may be a control input to control a robot, such as a torque or higher-level control command to a joint of the robot; or a control input to control an autonomous or semi-autonomous land or air or sea vehicle, such as a torque or higher-level control command to a control surface or other control element of the vehicle; or, for example, motor control data. In other words, an action may include, for example, position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent.

[0020] In some implementations, the environment is a simulated environment and the agents are implemented as one or more computers that interact with the simulated environment.

[0021] For example, the simulation environment can be a simulation of a robot or vehicle, and the reinforcement learning system can be trained on the simulation. For example, the simulation environment can be a motion simulation environment, such as a driving simulation or a flight simulation, and the agent is a simulated vehicle that navigates through the motion simulation. In these implementations, the action can be a control input that controls a simulated user or a simulated vehicle. The simulation environment is very useful for training the reinforcement learning system before using the system in a real environment. In another example, the simulation environment can be a video game, and the agent can be a simulated user playing the video game. Typically, in the case of a simulated environment, observations can include simulated versions of one or more previously described observations or observation types, and actions can include simulated versions of one or more previously described actions or action types.

[0022] Thus, in some implementations, the first data value may include control signals and / or sensor data for a mechanical device, such as a robot or a semi-autonomous or autonomous vehicle. The second data value may include a data value characterizing a predicted behavior of the mechanical device in response to the control signals and / or sensor data. The mechanical device may be controlled in response to the predicted behavior.

[0023] In some implementations, combining the aggregated output and the target input value includes concatenation of the aggregated output and the target input value. In some implementations, aggregating the encoded outputs includes calculating a mean of the encoded outputs.

[0024] In some implementations, the combination received by the decoder neural network system also includes a set of latent variables that collectively characterize the target input value, for example, defining the class to which the target input value belongs. In this way, the system can be used to define the target type to which the predicted second variable should belong, such as the type or example of the target image. In this way, an example of the target type can be generated by fixing the set of latent variables and generating the value of the second variable, for example, to generate a set of pixels of an image of the target type or a set of audio samples of sounds of a sound type, or more generally, samples from a specified learned distribution. The system can therefore also be used to convert from one data mode to another. For example, the sound or image type can be defined by text, so the system can be used to generate an image or sound corresponding to the text, or in the case of automatic conversion, the latent variable can be generated from another sound sample.

[0025] The system can be configured to determine parameters of a distribution of the set of latent variables. The decoder neural network system can then be configured to sample from the distribution of the set of latent variables to provide a decoder output. The sampled latent variables can be combined with (e.g., concatenated to) the aggregated output and input into the decoder neural network.

[0026] For example, the system may include a prior neural network and a posterior neural network, the prior neural network determining parameters of a prior distribution of a set of latent variables based on a set of observations, and the posterior neural network determining parameters of a posterior distribution of a set of latent variables based on a set of targets. During training, the prior distribution and the posterior distribution may become increasingly similar, such that values ​​predicted based on the targets and latent variables are closer to the values ​​of the observations used during training.

[0027] Typically, the predicted data values ​​can correspond to predicted observations and can be used in control tasks in real or simulated environments to predict the results of planned actions in the environment. Such predictions can be used in reinforcement learning systems, for example, to predict the results of actions.

[0028] In some implementations, the system can be used to query a database. For example, a query can be provided to the system in conjunction with an aggregated output or parameter encoded to data in the database. The system can then provide a prediction or result of querying the database based on a target input value and based on previously encoded observations of values ​​stored in the database.

[0029] Predictions may be learned from observations obtained from one or more sensors; predicted data items may include data as if generated from the sensor(s) at a future time. Predicted data items may be used to fill in missing data, or for semi-supervised learning where labels are available for only a small number of data items, or for providing multimodal data output, or for mapping from one domain to another. Input and / or predicted data values ​​may include values ​​of audio data, such as digitized signal values, or values ​​of still or moving images, such as pixel color or brightness data; values ​​defining the state or motion of an object; medical data, such as biomarkers or patient health data; music data; recommendation data characterizing user preferences or interests; spatiotemporal data related to physical and / or chemical processes; and the like.

[0030] In one embodiment, aggregation is performed via a swap operation such as a mean operation. This provides permutation invariance. The system can be configured to iteratively update the aggregate output upon receiving (one or more) new observations. This can be achieved by encoding the new observation to provide an encoded output (an encoded representation) of the new observation, and then applying an aggregation function to aggregate the previous aggregate output with the encoded output of the new observation to produce an updated aggregate output. The updated aggregate output can be input to a decoder neural network and used to generate predictions for other (one or more) target input values.

[0031] According to another aspect, a computer-implemented method is provided for generating decoder output data values ​​from input data values ​​according to one or more learned data distributions. The method includes: receiving a set of observations, each observation including a corresponding first data value for a first variable and a corresponding second data value for a second variable that depends on the first variable; inputting each observation in the set of observations into an encoder neural network system to encode each observation, thereby providing an encoded output for each observation; inputting each encoded output into an aggregator to aggregate the encoded outputs and provide an aggregated output; and inputting a combination of the aggregated output and a target input value into a decoder neural network system to provide a decoder output, wherein the target input value includes a value for the first variable and the decoder output predicts a corresponding value for the second variable.

[0032] According to another aspect, a method for generating output data values ​​from input data values ​​according to one or more learned data distributions is provided. The method may include receiving a target input value related to a domain, the target input value for a first variable. The method may further include combining the target input value with an aggregated parameter, the aggregated parameter being an aggregate of multiple parameters encoded via an encoder neural network from a set of training observations related to the domain, each training observation including a corresponding first data value for the first variable and a corresponding second data value for a second variable that depends on the first variable. The method may further include inputting the combination of the target input value and the aggregated parameter into a decoder neural network system to provide a decoder output, the decoder output predicting a value of the second variable corresponding to the target input value.

[0033] By utilizing aggregated parameters, which can be considered to encode information about training observations (that relate to the same domain as the target input values), the neural network can more efficiently determine predictions for the second variable. The training observations and the target input values ​​relate to the same domain (e.g., relate to the same data type), so they share similar properties.

[0034] In one implementation, the method further includes: for each of a plurality of training steps: receiving a training set of observations, each observation comprising a corresponding first data value for a first variable and a corresponding first data value for a second variable; combining aggregated parameters with each first data value from the training set of observations to provide a set of combinations for decoding; decoding each combination in the set of combinations using a decoder neural network system to provide a combination of decoder outputs; and training the decoder neural network system by back-propagation using a loss function that depends on the set of decoder outputs and the second data value of the set of targets.

[0035] By leveraging aggregated parameters, the encoder neural network can be trained faster and more efficiently.

[0036] The implementations described herein may be implemented in a system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0037] Furthermore, implementations may be embodied in one or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the systems or methods described herein.

[0038] Any reference to a set of values ​​herein should be interpreted as meaning a set of one or more values. Thus, a set of observations can be one or more observations. Likewise, a set of targets can be one or more targets.

[0039] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0040] Certain implementations of neural network systems can learn faster and use less data than conventional systems because they can rely on distributions learned from training data and can use this knowledge to learn similar distributions faster. Thus, for example, a system can be trained on observations of several different robotic arms and learn the behavior of the robotic arms in general. The system may then be able to learn the behavior of a specific new robotic arm much faster. Thus, the system may be able to learn to make predictions faster and more accurately, consume less memory, and use less computing resources.

[0041] Since learning can be shared between tasks, the methods and systems described herein provide flexibility for application on a variety of related tasks and datasets and reduce training. In addition, the mechanisms described herein avoid the need for pre-specified priors as is common in stochastic processes by learning prior knowledge during training and encoding it in the form of aggregated parameters used to condition the decoder neural network system.

[0042] Certain implementations of neural network systems are scalable to large data sets and complex distributions. By leveraging the aggregation of encoded outputs, the neural network systems described herein are more efficient than alternative approaches such as Gaussian processes and scale more effectively. For example, while Gaussian processes are adaptable to different data sets, they scale poorly and quickly become computationally intractable as the data set or dimensionality increases. In contrast, the methods and systems described herein can scale more efficiently. For a set of n observations and a set of m targets, the runtime complexity of an embodiment is of magnitude, since n observations are aggregated into a single representation. This is faster than the running time complexity of Alternative methods are orders of magnitude more efficient. Therefore, distributions can be learned from large datasets with relatively fewer resources than conventional techniques.

[0043] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the following description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A A schematic diagram showing a general framework for supervised learning problems;

[0045] Figure 1B A schematic diagram showing an example of a general supervised learning training scheme;

[0046] Figure 1C A schematic diagram of a conditional neural process (CNP) for predicting output according to an embodiment is shown;

[0047] Figure 2 shows an exemplary process for predicting decoder output using a conditional neural process (CNP) according to an embodiment; and

[0048] Figure 3 An exemplary process for training a Conditioned Neural Process (CNP) system is shown, according to an embodiment. DETAILED DESCRIPTION

[0049] Deep neural networks excel at function approximation, but they are usually trained from scratch for each new function. On the other hand, Bayesian methods, such as Gaussian Processes (GP), exploit prior knowledge to quickly infer the shape of new functions at test time. However, GPs are computationally expensive and it is difficult to design appropriate priors.

[0050] This application proposes a new family of neural models, the Conditional Neural Process (CNP), which combines the advantages of neural networks and Bayesian methods. CNP is inspired by the flexibility of random processes such as GP, but is constructed as a neural network and trained via gradient descent. CNP can make accurate predictions after observing only a small number of training data points, but can be extended to complex functions and large data sets.

[0051] In view of the above, the embodiments described herein provide an improved predictive neural network that can be accurately trained on a smaller dataset. This ensures accuracy while improving the computational efficiency of the system by avoiding the need to process large labeled datasets. CNP is scalable, and for m predictions with n observations, the runtime complexity achieved is This is faster than the running time complexity of The equivalent system significantly improves efficiency.

[0052] The performance and versatility of CNPs ensure that they can be applied to a range of typical machine learning tasks, including regression, classification, and image completion.

[0053] Supervised learning problems can generally be framed as function approximations given a finite set of observations. Classification, regression, dynamics modeling, and image generation can all be performed in this framework.

[0054] Figure 1A A schematic diagram of a general framework for supervised learning problems is shown. The set of observations includes inputs 10 and outputs 14. The inputs are mapped to outputs via corresponding hidden functions 12 (the hidden functions and parameters are indicated by dashed lines). The set of targets 16 includes the inputs, but has unknown outputs 18.

[0055] Observe that there are n inputs x i ∈X and output y i ∈Y dataset Assume that these represent the evaluation y of some unknown function f:X→Y i =f(x i ), the unknown function can be fixed or can be the implementation of some random function.

[0056] Figure 1B A schematic diagram of an example of a general supervised learning training scheme is shown. A set 20 of known observations including inputs and outputs is used to determine an approximation 22g of a hidden function f. This approximation function 22 can then be used to predict outputs 26 for a set of inputs 24 by providing a mapping from inputs to outputs.

[0057] More formally, a supervised learning algorithm returns an approximate function g: X→Y or a distribution over such functions. The goal is to minimize the loss between f and g over the entire space X, but in practice the routine is evaluated on a finite set of observations that are held-out (so that they are effectively unlabeled). These unlabeled data points are called targets. By minimizing the loss between f and g, the approximate function g can be fit to the observed data in order to empirically determine a function that accurately predicts the output for a given input.

[0058] One approach to the supervised problem is to re-randomly initialize the parameter function g for each new task, and spend a lot of computation on an expensive fitting phase. Any prior information the practitioner may have about f is specified via the architecture of g, the loss function, or the training details. This approach covers most of deep supervised learning. Because the degree of prior knowledge that can be expressed in this way is relatively limited, and learning cannot be shared between different tasks, the amount of training required is large, and deep learning methods tend to fail when training data is not abundant.

[0059] Another approach is to take a probabilistic stance and specify distributions over functions, which are called random processes; Gaussian processes (GPs) are an example. From this viewpoint, the practitioner's prior knowledge about f is captured in a distributional assumption about the prior process, and learning corresponds to Bayesian inference on the space of functions conditioned on the observed values. In the GP example, the assumption about the smoothness of f is captured a priori via a parametric kernel function, while g is treated as a random function distributed according to the predicted posterior distribution. Unfortunately, such Bayesian approaches quickly become computationally intractable as the dataset or dimensionality increases.

[0060] This application proposes a new family of models that represent solutions to supervision problems; and an end-to-end training method for learning them, which combines neural networks with features commensurate with Gaussian processes. This family of models is called conditional neural processes (CNPs) because they define the conditional distribution of a function given a set of observations. The dependence of the CNP on the observations is parameterized by a neural network that is invariant under permutations of its inputs. The embodiments described herein include scaling up to where n and m are the number of observations and targets, respectively.

[0061] In its most basic form, CNP embeds each observation, aggregates these embeddings into further embeddings of fixed dimensionality using a symmetric aggregator, and conditions the function g on the aggregated embeddings.

[0062] Figure 1C A schematic diagram of a conditional neural process (CNP) for predicting an output according to an embodiment is shown. A collection of n observations 30 is input to the neural network encoder h θ 32, the encoder h θ Map each observation to the corresponding encoded observation r according to the parameter θ i 34.

[0063] The set of encoded observations 34 is then input into an aggregator 36 which aggregates all encoded observations based on an aggregation operation a to produce an aggregate r 38 .

[0064] Aggregation 38 is then used to condition the neural network decoder g when predicting the output 44 for the target input 40. θ 42. That is, the neural network decoder 42 aggregates r and the target input x i as input and maps it to output y based on the parameters θ i .

[0065] CNP can be trained by sampling random datasets and following gradient steps to maximize the conditional likelihood of a random subset of the target given a set of random observations. This enables CNP to perform well in a variety of settings (i.e., n<<m or n>>m).

[0066] To frame the problem, consider the input x i ∈X and output y i The set O of pairs of ∈Y =

[0067] and another set of unlabeled points These are called the observation set and the target set, respectively. Suppose the output is the realization of the following process; let P be the probability distribution over the function f:X→Y, formally called a random process, then for f~P, set y i =f(x i ). P limits the random variable The joint distribution over O, thus defining the conditional distribution P(f(T)|O,T); the task of the system is to predict the output value f(x) for each x∈T given O. In other words, given a set of labeled observations, the system predicts the output value for each target input value.

[0068] As an inspiring example, consider ) is a random one-dimensional function f~P defined on the real line. O will be constructed at different positions x i The value of f at y i n observations of f. Given these observations, the goal is to predict the value of f at a new location on the real line. The classic assumption made on P is that all finite sets of function evaluations of f are jointly Gaussian distributed. This class of random functions is called a Gaussian process (GP). In this case, the predictive distribution P(f(T)|O,T) has a simple analytic form bounded by prior assumptions on the pairwise correlation structure (specified via a kernel function). The framework of random processes is attractive because Bayes’ rule allows one to make consistent inferences about the predictive distribution over f from observations O under a set of probabilistic assumptions. This allows the model to be data efficient, a property not common in most deep learning models. However, in practice, it is difficult to design appropriate priors, and the most interesting examples of random processes are computationally expensive and scale up greatly for n and m. This includes scaling up to GP.

[0069] To overcome this problem, the embodiments described herein utilize a novel mechanism to model the output, called conditional neural processes (CNPs). These conditional neural processes are much more efficient than alternative approaches (e.g., GPs) and can be scaled up to

[0070] CNP directly parameterizes conditional random processes without imposing consistency on some a priori process. Given a distributed representation of O of fixed dimensionality, CNP parameterizes the distribution over f(T). By doing so, the model sacrifices the mathematical guarantees associated with random processes in exchange for functional flexibility and scalability. Specifically, given a set of observations O, CNP is a conditional random process Q θ , which defines the distribution over f(x) for input x∈T. θ is a real vector defining all the parameters of Q. Inheriting the properties of random processes, assume that Q θ It is invariant to permutations of O and T. If O' and T' are permutations of O and T respectively, then Q θ (f(T)|O,T)=Q θ (f(T′)|O,T′)=Q θ (f(T)|O′,T). In this embodiment, permutation invariance with respect to T is usually enforced by adopting a factored structure. Specifically, Q θ , factor Q θ (f(T)|O,T)=∏ x∈T Q θ(f(x)|O,x). This is the simplest way to ensure a valid random process without assumptions about the output space Y. Nevertheless, the framework can be extended to non-factored distributions. An embodiment using such a model is described below.

[0071] The defining property of CNP is that it is conditioned on O via an embedding of fixed dimensionality. In more detail, using the following architecture,

[0072]

[0073]

[0074]

[0075] in, is the neural network encoder; is the neural network decoder; is a commutative operation, which takes and maps them to A single element of i It's Q θ (f(x i )|O,x i )=Q(f(x i )|φ i ) parameters.

[0076] Therefore, the CNP outputs parameters for a probability distribution. Depending on the task, the model learns to parameterize different output distributions. For regression tasks, φ i Can be used for each x i ∈T parameterized Gaussian distribution The mean and variance of For classification tasks, φ i The class probabilities p over c classes that can be used to parameterize the classical distribution c In certain embodiments, the logarithm of the class probabilities may be parameterized (e.g., via a logarithmic layer) such that the probabilities are extended to real numbers between -∞ and +∞. In certain embodiments, the aggregator function As an average operation (a1+…+a n ) / n, but alternative aggregator functions can be used.

[0077] Figure 2 An exemplary process for predicting decoder output using a conditional neural process (CNP) according to an embodiment is shown. Figure 1C CNP arrangement shown.

[0078] First, the system receives a set of observations O, each of which includes a corresponding data value x for a first variable i and the corresponding second data value y for a second variable that depends on the first variable i 201. The observations may be computer-readable data representing various forms of content, such as video data, numerical data, image data, sensor data, classification data, etc. The observations are labeled observations used to condition the decoder.

[0079] Each observation is then input into an encoder neural network to encode each observation, thereby providing an encoded output 202 for each observation. The encoder neural network can be a multi-layer perceptron (MLP) configured according to the dimensions required for the input and output. In one embodiment, the encoder is a three-layer MLP, but other types of encoders and other numbers of layers can be used. The encoder maps the observations to a latent space according to the parameters θ of the encoder neural network (as described in reference to equation (1)). In this way, the encoder maps the observations to a latent space according to the learned function h θ (x i ) produces a set of encoded outputs (encoded observations)

[0080] Then, the output of the aggregation code To provide an aggregated output r 203. The encoded outputs are aggregated according to an aggregation function (as described with reference to equation (2)). The aggregation function is a permutation-invariant commutative operation. The aggregation function may include taking the mean of the encoded outputs.

[0081] Then, the aggregated output r is compared with the target input value x t The decoder output y is combined and passed to a decoder neural network system which takes the combination as input and provides a decoder output 204. t is the target input value x conditioned on the aggregate r of encoded observations t The output of the aggregation r and the target input value x t Can be combined by concatenation.

[0082] The decoder neural network can be a multilayer perceptron (MLP) configured according to the required dimensions of the input and output. In one embodiment, the encoder is a five-layer MLP, but other types of decoders and numbers of layers can be used. The decoder converts the aggregate output r and the target input value x into a matrix according to the parameters θ of the decoder neural network. i The combination of is mapped to the output space (with equal y iIn this way, the decoder can be based on the learned function g θ (x t ) generates for each input target value x t prediction.

[0083] The architecture described above ensures permutation invariance of conditional predictions and Because the decoder is conditioned on an aggregate over the set of encoded observations, the system can be trained to high levels of accuracy and precision with fewer observations than is typically required when training a standard neural network.

[0084] Please note that since you can by calculate Therefore, the architecture supports streaming observations with minimal overhead. That is, each time a new observation r is received i , the aggregation can be updated by encoding this new observation and aggregating the new encoded observation with the previous aggregation. In this case, the updated aggregation r′ will be:

[0085]

[0086] This allows the aggregate to be efficiently updated each time a new observation is received, using only the new observation and the previous aggregate as input. Hence, there is no need to store the original observations or their corresponding encoded representations in memory for future computations, as the aggregate can be updated without them.

[0087] Encoder neural network h θ (x i ,y i ) and the decoder neural network g θ (x i ) can be trained by back propagation to minimize the loss function (e.g., via stochastic gradient descent). In one embodiment, by letting the neural network system Q θ Train a neural network system Q by predicting O conditioned on a randomly selected subset of O θ This pair of models signals the uncertainty inherent in the distribution P over the space X given a set of observations. More precisely, let f~P, is the set of observations, then N ~ uniform [0, ..., n-1]. The system is a subset (the first N elements of O) as the condition. Then, the system is trained by minimizing the negative conditional log probability

[0088]

[0089] That is, the system minimizes the correlation with the ground truth label The output of the matching decoder This is equivalent to maximizing the probability of The output of the matching decoder ; however, given that many optimizers implement loss functions, minimization of the negative value of the probability can be exploited to maintain consistency.

[0090] Therefore, the model Q θ The target of the score includes both observed and unobserved values.In a particular embodiment, a Monte Carlo estimate of the gradient of the loss function (via probabilistic or Monte Carlo simulation) is obtained by sampling f and N.

[0091] This approach shifts the burden of imposing prior knowledge from analytical priors to empirical data. This has the advantage of freeing practitioners from having to specify analytical forms of priors that are ultimately intended to generalize their empirical experience. Still, it is important to emphasize that Q θ There is not necessarily a consistent set of conditionals for all sets of observations, and the training routine cannot guarantee this.

[0092] Figure 3 An exemplary process for training a Conditioned Neural Process (CNP) system is shown, according to an embodiment.

[0093] The process receives a set of 301n observations from Start. Observations are labeled observations consisting of n inputs and n outputs Then, select a subset O of N observations from O N 302. For the subset O N Each observation in is encoded 303 to determine the set of corresponding encoded observations The encoded observations are then aggregated 204 to provide an aggregate r. The aggregate r is then combined with the original set of observations O (including O N and O N The predicted output for each input is determined 305 by combining (e.g., concatenating) each input in the decoder to determine the corresponding predicted output. The predicted output can be the target output The form of the Gaussian mean and variance.

[0094] The predicted output is then compared to the ground truth value (from O ) and update the parameters θ of the encoder and decoder to reduce the loss function 306. The method can then be repeated with the updated parameters θ until the parameters converge to a set of optimal parameters that minimize the loss function, or until a maximum number of iterations is reached. The parameters can be trained to maximize the log-likelihood of the target point. This can be achieved, for example, using stochastic gradient descent, but various optimization methods can also be used. In one embodiment, the parameters are optimized using an adaptive moment estimation (Adam) optimization method.

[0095] It should be noted that although Figure 3 The method determines the same observation O that was used to generate the aggregation N predictions, but in alternative embodiments, different subsets of O may be used for generation of aggregates and computation of predicted outputs.

[0096] In summary, in the embodiments described herein:

[0097] 1. CNP is a conditional distribution over functions trained to model the empirical conditional distribution of functions f~P.

[0098] 2.CNP is permutation invariant in O and T.

[0099] 3. CNP is scalable. The running time complexity of implementing m predictions with n observations is

[0100] The methods described in this article can be implemented for various use cases such as classification, image completion and prediction, and function regression.

[0101] To test function regression, the system is trained using curves generated by a Gaussian process (GP) with an exponential kernel. The curve is sampled, a subset of n points on the curve is selected as observations, and a subset of other points is selected as target points. The model is shown in Figure 1 and has a three-layer MLP encoder h with a 128-dimensional output representation r i ; and a five-layer MLP decoder configured to output each target output The variance allows the system to quantify the error in the prediction.

[0102] When the trained system is compared to predictions generated by a GP with the correct hyperparameters (which constitutes an upper bound on performance), the system is able to regress with fewer context points (observations used to generate the aggregation) than the GP requires. Despite the predictions generated by , the accuracy of the model improves and the uncertainty of the model decreases as the number of context points increases. Crucially, the model learns to estimate its own uncertainty given an observation very accurately. Nonetheless, it provides a good approximation, and its accuracy increases as the number of context points increases.

[0103] Furthermore, when the model is provided with a set of context points generated by a GP in which the parameters of the kernel are varied, the model achieves similarly good performance. This type of regression task is not trivial for GP, but rather, in embodiments herein, can be efficiently performed by only changing the dataset used for training.

[0104] The embodiments described herein can be used as f:[0,1] for grayscale images 2 →[0,1] or f∶[0,1] for RGB images 2 →[0,1] 3 The input x is normalized to [0,1] 2 The output y is a vector of grayscale or RGB intensities of the corresponding pixels. For this completion task, the same model architecture as 1D function regression can be used (except that the last layer is set to 3D for RGB).

[0105] The training process involves selecting images from the dataset at each step and picking a subset of pixels as observations. Conditioned on these, the model is trained to predict the values ​​of all pixels in the image (including pixels that have been conditioned on it). Like in 1D regression, the model outputs a Gaussian mean and variance for each pixel and is optimized for the log-likelihood of the ground truth image.

[0106] As with the 1D regression task, the system makes accurate predictions even with a small number of context points. Crucially, when conditioned on just one uninformative context point (e.g., a black pixel on the edge of the image), the model’s predictions correspond to the average over all training images. As the number of context points increases, the predictions become more similar to the underlying ground truth. This demonstrates the model’s ability to extract dataset-specific prior knowledge. It is worth mentioning that even with the full set of observations, the model does not achieve pixel-perfect reconstruction due to bottlenecks at the representation level.

[0107] Since this embodiment returns factored outputs, the best prediction it can produce given limited context information is the average over all possible predictions consistent with that context. Such an alternative is to add latent variables to the model so that they can be sampled conditioned on the context to produce predictions with high probability in the data distribution. This embodiment is discussed below.

[0108] An important aspect of this model is its ability to estimate the uncertainty of its predictions. As more observations are added, the variance shifts from being almost uniformly spread across the image to being localized around regions that are specific to the underlying image, such as the edges of distinguishing features within the image. Being able to model uncertainty given some context can be helpful for many tasks. One example is active exploration, where the model can choose where to observe. This can be tested by comparing the predictions of the CNP when choosing observations based on the uncertainty (i.e. the pixel with the highest variance at each step) to a random pixel. This approach is a very simple way to do active exploration, but it already produces better predictions than randomly choosing a conditional point.

[0109] An important aspect of the embodiments described herein is their flexibility not only in the number of observations and targets received, but also in terms of input values. The first type of flexibility is that during testing, the system can be conditioned on subsets that were not encountered during training. For example, the model can be conditioned on half of an image. This forces the model to predict pixel values ​​not only based on certain stationary smoothness properties of the image, but also based on global spatial properties (e.g., symmetry and relative positions of different parts of the face). The embodiments described herein are able to capture these properties. In contrast, a GP with a fixed kernel cannot capture this property and will revert to its mean in the absence of observations (the mean itself can be non-stationary, but this is usually not enough to capture the property of interest).

[0110] Additionally, the embodiments described herein are flexible with respect to target input values. This means, for example, that the model can be queried at resolutions that were not observed during training. At test time, a model trained using only pixel coordinates at a specific resolution can be used to predict sub-pixel values ​​of targets between the original coordinates. Only a single forward pass is required to query the system at different resolutions. While GPs also exhibit this type of flexibility, this is not the case for trained generative models, which can only predict values ​​for the pixel coordinates on which they were trained. In this sense, the methods described herein have the upper hand on both counts – they are flexible with respect to conditioning and prediction tasks, and have the ability to extract domain knowledge from the training set.

[0111] In the above embodiments, there are still some aspects that can be modified to suit specific needs. For example, the exact implementation of h can be adapted to the data type. For low-dimensional data, the encoder can be implemented as a multilayer perceptron (MLP), and for inputs with larger dimensions and spatial correlation, it can also include convolution. Finally, in the above embodiments, the system cannot produce any coherent samples because it learns to model only factored predictions of the mean and variance, regardless of the covariance between the target points. This is a result of this particular implementation of the model. One way to obtain coherent samples is by introducing hidden variables from which sampling can be performed.

[0112] The main embodiment described above utilizes a factored model that predicts the mean and variance of the target output. Although the mean itself is a useful prediction and the variance is a good way to capture uncertainty, the factored model cannot obtain coherent samples on multiple targets.

[0113] Consider the above embodiment referring to image completion conditioned on a small number of observations. This embodiment can be implemented to complete digital images. Rather than just predicting the mean of all numbers, sometimes a system is needed that can be used to sample different coherent images of all possible numbers conditioned on an observation.

[0114] GPs can do this because they contain a parameter kernel that predicts the covariance between all points in the observation and target. This forms a multivariate Gaussian distribution that can be used to coherently extract samples. In order to maintain this property in the trained model, one approach is to train the model to predict the GP kernel. However, the difficulty is that backpropagation through sampling is required, which involves large matrix inversions (or some approximation thereof).

[0115] In contrast, according to one embodiment, a system as described above is provided, but with the addition of a latent variable z to the decoder g. This allows the model to capture global uncertainty. To generate coherent samples, a representation r is computed from the observations, which parameterizes a Gaussian distribution over the latent variable z. The latent variable z can then be sampled, where the same value is used to generate samples for all targets (all pixels in the case of an image). To obtain different coherent samples, new samples can be drawn from the latent variable z and input into the decoder again for all targets. Similar to a standard VAE, the model can be trained by optimizing a variational lower bound on the log-likelihood using a conditional Gaussian prior p(z|O) conditioned on the observations and a Gaussian posterior p(z|O,T) also conditioned on the target points.

[0116] We use the same model that we could have used before, but instead concatenate the representation r to a vector of latent variables z. For image completion (e.g., with r of dimension 128), z can be 64. For both the prior and posterior models, a three-layer MLP can be used and the mean of their outputs can be taken (for the observation encoder). It is important to emphasize that the difference between the prior and the posterior is that the prior only sees the observed pixels, while the posterior sees both the observed pixels and the target pixels. When sampling from this model with a small number of observed pixels, coherent samples are obtained and the variability of the dataset is captured. As the model is conditioned on more and more observations, the variability of the samples decreases, and they eventually converge to a single possibility.

[0117] In addition, the methods described in this article can be applied to classification (e.g., image classification). In this case, by using the information provided in the input labels, aggregation is only generated on observations from the same class. In other words, observations are divided into their corresponding classes based on their labels (their ground truth classifications). The observations of each class are encoded and then aggregated to form class-specific aggregates. The class-specific aggregates (aggregated class-specific representations) are then combined (e.g., concatenated) to form the final representation. This is then used as input to condition the decoder to classify the target input. The decoder outputs the probability that the target input belongs to that class for each potential class.

[0118] Assuming that the size of the class-specific representation and the number of classes are constant, the size of the final representation is still constant, so The running time still holds.

[0119] Table 1 shows the performance of the CNP classification embodiment on the Omniglot dataset relative to alternative systems such as MANN (Santoro et al., 2016) and matching networks (MN) (Vinyals et al., 2016).

[0120]

[0121] Table 1: Classification results of CNP embodiment on Omniglot relative to MANN and MN

[0122] As can be seen, CNP achieves higher accuracy than much more complex models such as MANN. Although CNP does not beat these alternatives for shot classification, the accuracy values ​​are comparable. Crucially, a much simpler architecture is used at test time and Rather than These values ​​are achieved with a relatively low running time of . Therefore, the embodiments described herein have comparable accuracy and greatly improved computational efficiency.

[0123] The embodiments described herein utilize a new set of models, namely conditional neural processes. These models are flexible at test time and have the ability to extract prior knowledge from training data. The embodiments described herein can perform a variety of tasks, including regression, classification, and image completion.

[0124] It is important to note that the specific CNP implementation described in this article is just a simple proof of concept and can be substantially extended, for example, by including more sophisticated architectures compatible with modern deep learning advances. In addition, the model itself can be updated to be "more Bayesian" by sharing priors and posteriors or adding hierarchies of latent variables, among other options.

[0125] In summary, this work can be seen as a step towards learning high-level abstractions, which is one of the grand challenges of contemporary machine learning. Most conventional deep learning models learn functions that are associated with a specific, constrained statistical context at any stage of training. In this sense, the trained CNP is more general because it encapsulates the high-level statistics of a family of functions. Therefore, it constitutes a high-level abstraction that can be reused for multiple tasks. These models can help solve many key machine learning problems that seem to be related to abstraction, such as transfer learning, meta-learning, and data efficiency.

[0126] In view of the above discussion, by conditioning the decoder using an aggregation of observed encoded representations, the methods described herein enable accurate predictions based on target inputs in a much more computationally efficient manner. Embodiments implement a simpler architecture that provides a reduced memory burden on the implementing computing system relative to alternative more complex architectures. In addition, the embodiments described herein enable accurate predictions to be obtained with only a small number of training observations and only a small number of training steps. This further improves efficiency and provides the ability to learn quickly based on new observations.

[0127] This specification uses the term "configured" in relation to systems and computer program components. A system of one or more computers is configured to perform a particular operation or action in the sense that the system has software, firmware, hardware, or a combination thereof installed thereon that, when in operation, causes the system to perform the operation or action. One or more computer programs are configured to perform a particular operation or action in the sense that the one or more programs include instructions that, when executed by a data processing device, causes the device to perform the operation or action.

[0128] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transient program carrier for data processing device execution or control of the operation of the data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, for example, machine-generated electrical, optical or electromagnetic signals, which are generated as coded information to be transmitted to a suitable receiver device for data processing equipment execution. Computer storage media may be machine-readable storage devices, machine-readable storage substrates, random or sequential access memory, or a combination of one or more of them. However, computer storage media are not propagation signals.

[0129] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0130] A computer program (which is also referred to or described as a program, software, software application, module, software module, script or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including in the form of a stand-alone program or in a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may but need not correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program being targeted, or in multiple collaborative files, for example, in a file storing one or more modules, subroutines or code portions. A computer program may be deployed to execute on one or more computers, the multiple computers being located at one site or distributed between multiple sites and connected via a communication network.

[0131] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is different from the input. An engine can be a coded functional block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device including one or more processors and a computer-readable medium, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. In addition, two or more of the engines can be implemented on the same computing device or on different computing devices.

[0132] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by dedicated logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)), and the apparatus may be implemented as such dedicated logic circuits. For example, the processes and logic flows may be performed by a graphics processing unit (GPU), and the apparatus may also be implemented as a graphics processing unit.

[0133] Computers suitable for executing computer programs include, for example, can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for running or executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to one or more of the large-capacity storage devices to receive data from it or transfer data to it or perform both reception and transmission. However, a computer is not necessarily provided with such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.

[0134] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or may be incorporated in, special purpose logic circuitry.

[0135] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to a user, and a keyboard and pointing device through which the user may provide input to the computer, such as a mouse or trackball. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a browser on a user's client device in response to a request received from the web browser.

[0136] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface and a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0137] A computing system may include clients and servers. Generally speaking, clients and servers are remote from each other and typically interact through a communication network. The relationship of client and server is formed by computer programs running on the respective computers and having a client-server relationship to each other.

[0138] Although this specification includes many specific implementation details, these details should not be interpreted as limiting any invention or the scope of claim, but should be interpreted as a description of the features of a particular embodiment that may be specific to a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple independent embodiments or in any appropriate sub-combination. In addition, although features may be described as working in certain combinations as described above and even initially claimed in this way, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may point to a variant of a sub-combination or a sub-combination.

[0139] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, but it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0140] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A neural network system for generating decoder output data values ​​from input data values ​​according to one or more learned data distributions, the system comprising: an input receiving a set of observations, each observation comprising a corresponding first data value of a first variable and a corresponding second data value of a second variable that is dependent on the first variable; an encoder neural network system configured to encode each observation in the set of observations to provide an encoded output for each observation; an aggregator configured to aggregate the encoded outputs for the set of observations and provide an aggregated output; and A decoder neural network system is configured to receive a combination of the aggregated output and a target input value and provide a decoder output, wherein the target input value includes a value of the first variable and the decoder output predicts a corresponding value of the second variable.

2. The system according to claim 1, wherein: The system is configured to, for each of a plurality of training steps: selecting a subset of observations from the observations for input to the encoder neural network system and selecting a set of targets for training the system; encoding each observation in the subset of observations using the encoder neural network system to provide an encoded output for each observation; aggregating the outputs of the encoding; combining an aggregate of the encoded outputs with each first data value in a set from the target to provide a combined set for decoding; decoding each combination in the set of combinations using the decoder neural network system to provide a set of decoder outputs; and The encoder neural network system and the decoder neural network system are trained by back-propagation using a loss function that depends on the set of decoder outputs and second data values ​​in the set of targets.

3. The system according to claim 1 or 2, wherein: The decoder output defines parameters of a decoder output distribution, and wherein the system is configured to determine the corresponding value of the second variable based on the decoder output distribution.

4. The system according to claim 3, wherein: The decoder output distribution defines a mean and a variance of the corresponding values ​​of the second variable.

5. The system according to claim 3, wherein: Each of the first data values ​​comprises a vector defining a data item, and each of the second data values ​​defines a corresponding class of the respective data item, and wherein the decoder output distribution defines a probability distribution over the classes.

6. The system according to claim 5, wherein: The encoder neural network system is configured to encode each observation by encoding a first data value of each observation, and wherein the aggregator is configured to: For each class, determining an aggregate for the class based on the encoded first data values ​​for observations in the class; and The output of the aggregation is determined by combining the aggregations for each class.

7. A system according to any one of claims 1 to 6, wherein: The output of the aggregation is invariant under permutations of observations in the set of observations.

8. The system according to any one of claims 1 to 7, wherein: The combination received by the decoder neural network system also includes a set of latent variables that together characterize the target input value.

9. The system according to claim 8, wherein: The encoder neural network system is further configured to determine parameters of a distribution of the set of latent variables, and wherein the decoder neural network system is configured to sample from the distribution of the set of latent variables to provide the decoder output.

10. The system of claim 9 when dependent on claim 2, further comprising a prior neural network and a posterior neural network, wherein the prior neural network determines the parameters of the prior distribution of the set of latent variables based on the set of observations, and the posterior neural network determines the parameters of the posterior distribution of the set of latent variables based on the set of targets.

Images