Unsupervised learning of object representations from video sequences using spatial and temporal attention

Abstract

A computer-implemented video generation neural network system configured to determine a value for each of a set of object latent variables by sampling from a respective prior object latent distribution for the object latent variable. The system includes a trained image frame decoder neural network configured to process the determined values for the object latent variables for each pixel of each generated image frame and for each generated image frame time step to determine parameters of a pixel distribution for each object latent variable, to combine the pixel distributions for each object latent variable to determine a combined pixel distribution, and to sample from the combined pixel distribution to determine a value for the pixel and for the time step.

Description

Technical Field

[0001] This specification generally describes a system and method implemented as one or more computer programs on one or more computers at one or more physical locations, which train video objects representing a neural network system. Background Technology

[0002] This specification relates to training machine learning models to characterize objects in video frames, as well as global time-varying elements such as viewpoint. The machine learning model receives input and generates output, such as predicted output, based on the received input and the model's parameter values.

[0003] A neural network is a machine learning model that uses one or more non-linear units to predict the output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0004] This specification generally describes a system and method implemented as one or more computer programs on one or more computers at one or more physical locations, which train a neural network system to represent video objects. The encoder portion of the trained system can be used to determine attributes of multiple distinct objects in a video sequence, or to determine the viewpoint position of the video sequence, or both. The decoder portion of the trained system can be used to generate new video sequences. The encoder and decoder can be used together to modify video sequences.

[0005] In one aspect, a method is provided for training a computer implementation of a video object representation neural network system. The video object representation neural network system includes a self-attention neural network comprising one or more self-attention layers (e.g., transformer neural network layers) and having multiple self-attention neural network parameters, each self-attention layer being configured to apply a self-attention mechanism to its layer input.

[0006] The method involves obtaining a video sequence comprising a sequence of T image frames, one image frame at each time point in a series of time intervals, and processing each image frame to generate a set of S spatial codes for each of the T image frames. The method thus obtains a set of S×T spatial codes for the video sequence. Each spatial code includes the encoding of a corresponding region or patch of the image frame. The video can be obtained, for example, from a camera or LiDAR sensor, meaning the image frames may include point clouds.

[0007] The method also includes processing a group of S×T spatial codes using an attention neural network. Specifically, this is achieved by applying a self-attention mechanism to the layer input comprising the group of S×T spatial codes, optionally generating a group of transformed spatial codes after further self-attention processing steps. Each transformed spatial code corresponds to an image frame time and an image frame region. The groups of transformed spatial codes can have the same or different resolutions; for example, a lower resolution may have S×T or fewer transformed spatial codes. In this implementation, applying the self-attention mechanism to the layer input includes applying it to the spatial codes of different image frames, i.e., to image frames from different times within a series of times.

[0008] The method also includes determining one or more values, such as the mean or standard deviation (or variance) of the distribution, for each of a set of object latent variables (typically a vector) to parameterize the object latent variable. This is accomplished by aggregating a group of spatially encoded transforms over a temporal region of the image frame. The method can also determine one or more values ​​of the frame latent distribution for parameterizing the frame latent variable for each of a set of frame latent variables by aggregating a group of spatially encoded transforms over a region of the image frame. The method determines the value of each of the set of object latent variables by sampling from the object latent distribution of the object latent variables. The method can also determine the value of each of the set of frame latent variables by sampling from the frame latent distribution of the frame latent variables.

[0009] This method processes the values ​​of determined object latent variables (and can process determined values ​​of determined frame latent variables) to generate a sequence of decoded image frames. Many types of decoders exist capable of generating images from a set of latent variables, and any of these can be used. An example decoder is described later.

[0010] The method may then include training the system by adjusting the values ​​of at least the parameters of the self-attention neural network to depend on i) a measure of the difference between the sequence of T image frames and the sequence of decoded image frames, and ii) the difference between the latent distribution of each object and the prior object latent distribution. Where determined, the objective function may be optimized based on iii) the difference between the latent distribution of each frame and the prior frame latent distribution. The prior object latent distribution and the prior frame latent distribution may be the same distribution, such as a unit Gaussian.

[0011] The above implementation provides a trained video object representation neural network system. In some applications, once trained, only the encoder portion of the system may be needed—the part of the system that takes the video sequence input as input and is configured to determine the object and (optionally) the potential distribution of frames, optionally including a part configured to sample from these distributions (in other implementations, information may be derived, for example, from the average of these distributions). In some applications, once trained, only the trained decoder portion of the system may be needed, as described later, for example, for video sequences used to generate 2D or 3D images.

[0012] The trained system implementation provides object information from this set of object latent variables. For example, in this implementation, these are substantially distinguishable, i.e., different object latent variables correspond to different changing factors and attributes of objects in the scene, such as size, color, position (e.g., in each dimension), and motion. Similarly, in this implementation, different frame latent variables can distinguish different global attributes of frames in a video sequence, such as, for example, viewpoint position (e.g., in each dimension). Further advantages of this method and system will be described later.

[0013] In this implementation, each spatial code includes a feature map, or visual feature map, of a corresponding region or "patch" of the image frame. Processing the image frame to generate a set of spatial codes can use a feature extraction neural network with multiple feature extraction neural network parameters, such as a convolutional neural network, to generate feature maps for each region of the image frame. The training system may include adjusting the values ​​of the feature extraction neural network parameters.

[0014] In one implementation, an image frame decoder neural network with multiple parameters is used to process the values ​​of determined object latent variables (and optionally frame latent variables) to generate a sequence of decoded image frames. The training system can then include adjusting the values ​​of the image frame decoder neural network parameters.

[0015] In some implementations, the decoded image frame sequence comprises image frames for each of a series of decoded image frame time steps. Generating the decoded image frame sequence uses an image frame decoder neural network to generate parameters for the pixel distribution of each object latent variable for each decoded image frame pixel and for each decoded image frame time step. Specifically, the image frame decoder neural network processes the determined values ​​of the object latent variables (and optionally frame latent variables), along with information specifying the pixel location and the time step, to determine the parameters for the pixel distribution (μ) of each object latent variable. The pixel distributions of each object latent variable are then combined to determine the combined pixel distribution. This is then sampled to determine values ​​for pixels and for the time step. For example, in one implementation, the frame latent variables (used to decode the image frames) are concatenated with each object latent variable to provide input to the image frame decoder neural network, more specifically, a set of inputs.

[0016] Generating a sequence of decoded image frames may include determining mixing weights for each object latent variable for each pixel and for each time step. The pixel distribution of each object latent variable can then be combined according to the corresponding mixing weights, optionally normalized. In this implementation, the per-pixel mixing weights for each object latent variable provide a soft segmentation mask for the object corresponding to the object latent variable, for example, to segment each image frame into regions representing different objects in the image frame.

[0017] Any suitable metric can be used to measure the difference between the image frame sequence and the decoded image frame sequence, such as cross-entropy loss, squared error loss, or Huber loss. Any suitable metric can also be used to measure the difference between distributions, such as KL divergence or Jensen-Shannon divergence, or another metric for the distance between distributions.

[0018] In some implementations, the method may include obtaining conditional inputs that define one or more objects or object attributes of one or more objects in the video sequence, or that define viewpoint position or orientation, such as pose. During training, the conditional inputs may be provided to a self-attention neural network and an image frame decoder neural network. In this way, the system can be trained, for example, to conditionally generate video image sequences based on conditional inputs that define objects or object attributes to be included in the generated video image sequences, or optionally modified viewpoints of the generated video sequences.

[0019] As mentioned earlier, the self-attention layer can be a transformer neural network layer. The application of transformer neural network architecture in computer vision is described in arXiv:2010.11929.

[0020] In an implementation, processing a group of S×T spatially encoded groups using an attention neural network can include adding positional information to each of the S×T spatial encodings that define corresponding regions of an image frame and the time of the image frame. The positional information can include, for example, a positional embedding, i.e., an embedding of the image frame region and time. The embedding can include a vector having the same dimension as the spatial encoding; it can be added to the corresponding spatial encoding. The embedding can be learned or predefined.

[0021] In an implementation, using an attention neural network to process a group of S×T spatial encodings includes using one of one or more self-attention layers to process a layer input that includes the group of S×T spatial encodings to generate a query and key-value pairs for each spatial encoding. The self-attention mechanism can then be used to apply, for example, each query to, for example, each key, more specifically, the key-value pairs, to determine a group of transformed spatial encodings. The self-attention can include masked self-attention, in which case not all queries can be applied. The self-attention mechanism can be a multi-head attention mechanism. In an implementation, the self-attention neural network parameters include the parameters of a learned transformation applied to the layer input to generate the query and key-value pairs.

[0022] Generally, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. The exact self-attention mechanism applied depends on the configuration of the attention neural network. For example, each attention layer can apply a dot product attention mechanism, such as a scaled dot product attention mechanism. In scaled dot product attention, for a given query, the attention layer computes the dot product of the query with all keys, divides each dot product by a scaling factor, such as the square root of the dimension of the query and the keys, and then applies a softmax function to the scaled dot products to obtain the weights of the values. Then, the attention layer computes the weighted sum of the values according to these weights.

[0023] In some implementations, processing a group of S×T spatial encodings using an attention neural network includes providing the group of S×T spatial encodings as an input to a first self-attention layer of the attention neural network at a later stage to generate a first self-attention layer output, and providing the first self-attention layer output to a second self-attention layer of the attention neural network to generate a group of transformed spatial encodings. In an implementation, this includes reducing the number of spatial encodings from S×T spatial encodings to K×T spatial encodings, where K < S is the number of object latent variables. Thus, an implementation can have more feature maps than there are object latents, or the same number. This reduction can be performed after the first self-attention layer or after the second self-attention layer, for example, by pooling in the (tiled patch) horizontal and / or vertical directions, and optionally normalizing.

[0024] On the other hand, a computer-implemented video processing neural network system is provided, comprising a (trained) feature extraction neural network configured to receive a video sequence comprising a sequence of T image frames, one image frame at each time point in a series of time points, and to process each image frame to generate a set of S spatial codes for each of the T image frames, thereby obtaining a set of S×T spatial codes for the video sequence. Each spatial code may include an encoding of a corresponding region of the image frame.

[0025] The computer-implemented video processing neural network system also includes a (trained) self-attention neural network comprising one or more self-attention layers, each configured to apply a self-attention mechanism to its layer input. In this implementation, the self-attention neural network is configured to process the S×T spatial codes by applying a self-attention mechanism to the layer input, which comprises a group of S×T spatial codes, to generate a group of transformed spatial codes, each transformed spatial code corresponding to an image frame time and an image frame region. Applying the self-attention mechanism to the layer input includes applying the self-attention mechanism to the spatial codes of different image frames.

[0026] The video processing neural network system is configured to determine one or more values ​​of the object latent distribution for each of a set of object latent variables by aggregating a group of spatially encoded transformations over time in an image frame, and optionally by sampling from the object latent distribution of the object latent variable to determine the value of each of the set of object latent variables.

[0027] The video processing neural network system can also be configured to determine one or more values ​​of a frame latent distribution for parameterizing each of a set of frame latent variables by aggregating a group of spatially encoded transformations over a region of an image frame, and optionally by sampling from the frame latent distribution of the frame latent variable to determine the value of each of the set of frame latent variables.

[0028] The video processing neural network system is also configured to: i) determine one or more attributes of one or more objects depicted in a video sequence based on values ​​used to parameterize the object latent distribution, i.e., based on object latent variables or based on values ​​used to parameterize the object latent distribution; ii) determine the position or orientation of the viewpoint of the video sequence based on values ​​used to parameterize the frame latent distribution, i.e., based on frame latent variables or based on values ​​used to parameterize the frame latent distribution; or both.

[0029] In another aspect, a computer-implemented video generation neural network system is provided, which is configured to determine the value of each of a set of object latent variables by sampling from the corresponding prior object latent distribution of the object latent variables (and optionally by sampling from the corresponding prior frame latent distribution of the frame latent variables).

[0030] The system includes a (trained) image frame decoder neural network for processing determined object latent variable values ​​(and optionally, determined frame latent variable values) to generate a video sequence comprising a sequence of image frames generated at consecutive time steps. The image frame decoder neural network is configured to, for each pixel of each generated image frame and for each generated image frame time step, process determined object latent variable values ​​(and optionally, determined frame latent variable values), information specifying pixel location, and information specifying time step to determine parameters of the pixel distribution of each object latent variable for each pixel and for each time step; combine the pixel distributions of each object latent variable to determine a combined pixel distribution; and sample from the combined pixel distribution to determine values ​​for each pixel and for each time step.

[0031] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0032] The video object representation neural network system, specifically the encoder portion of a system comprising a feature extraction neural network and a self-attention neural network, can process video sequences of 3D scenes and simultaneously distinguish and recognize, i.e., represent multiple objects. The system is capable of doing this even when the viewpoint (e.g., camera pose) is changing without explicitly given information about the viewpoint or camera pose. The system is implemented in a way that can handle scenes with many objects and complex backgrounds.

[0033] Therefore, this system is able to determine the representation, that is, the latent variables of the set of objects or their distribution, which is allocentric, and is important for many tasks. For example, a system that does not know the viewpoint may confuse the distance of an object from the viewpoint with its size; an embodiment of this system overcomes this problem.

[0034] In embodiments of this system, the set of object latent variables, or their distribution, provides a stable representation of objects in a scene relative to camera pose. In this implementation, this is achieved by spatially and temporally focusing attention across different image frames of a video sequence, combined with a factorized latent space that separates static (i.e., substantially unchanging over time) object properties from global or dynamic attributes of the video sequence, such as camera pose. This also allows the system to determine static object motion characteristics, such as object trajectory summaries.

[0035] The system also determines representations of global, potentially time-varying scene attributes, such as viewpoints or their distribution within the latent variables of the frame set. This ability to determine viewpoints allows the system to be used to determine, for example, the location of a moving mechanical agent equipped with a camera or other video capture device.

[0036] Some implementations of this system also have a simpler architecture than some previous systems, which were less capable.

[0037] The system can also be implemented to generate video sequences, such as video sequences from an existing scene that appear to originate from a specific viewpoint, or video sequences of new or modified scenes containing one or more objects. This can be accomplished, for example, by sampling from existing scenes or from prior distributions, and then feeding them to a video object representation neural network system, specifically the decoder part of that system, to determine the values ​​of latent variables. This can be used, for example, for viewpoint interpolation or extrapolation, or for, for example, for planning when controlling mechanical agents such as robots to perform tasks. Unlike some previous systems that could do this for specific scenes on which they had previously been trained, embodiments of this system can be trained on different scenes and then do so for new, previously unseen scenes.

[0038] For example, the encoder of an untrained or trained video object representation neural network system can be used in reinforcement learning systems that control agents to perform tasks, and / or replace images from the environment in which the agent operates, to allow the reinforcement learning system to learn faster or more effectively. Alternatively, the decoder of an untrained or trained video object representation neural network system can be used as a model-based reinforcement learning system to predict the future state of the environment when learning action selection policies for controlling agents.

[0039] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0040] Figure 1 The video object represents a neural network system;

[0041] Figure 2 The video object represents the encoder neural network of the neural network system;

[0042] Figure 3 An image frame decoder neural network is shown, configured to decode latent variables into pixel values ​​in a video object representation neural network system.

[0043] Figure 4 This is a schematic diagram illustrating the decoding process performed by the decoder of a neural network system that represents video objects;

[0044] Figure 5 This is a flowchart of an example process for training a neural network for video objects using this system.

[0045] Figure 6 It is a flowchart of the process of obtaining object attributes using the trained encoder part of a neural network system to represent video objects;

[0046] Figure 7 This is a flowchart illustrating the process of generating image sequences using a trained decoder portion of a neural network system that represents video objects; and

[0047] Figure 8 A comparison of view synthesis tasks performed by a video object representation neural network system and two other neural network models is shown.

[0048] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0049] Figure 1 A video object representation neural network system 100 is schematically illustrated, which can be implemented by one or more computers located at one or more locations. The video object representation neural network system 100 includes an inference network (encoder 102) and a generative model (decoder 120). The inference network is trained to infer latent variables from an input sequence of images (frames, having a dimension of H×W pixels), and the generative model is trained to decode the latent variables generated by the encoder 102 to generate pixel values ​​(i.e., images). The encoder 102 uses spatiotemporal attention-based inference that allows for stable object tracking. The decoder 120 is capable of combining the object latent variables of a given sequence with the frame latent variables of different sequences to generate consistent renderings of the same scene content (i.e., objects and their attributes, relative rearrangements, and segmentation assignments) from completely different viewpoints.

[0050] Specifically, encoder 102 is trained to process input image sequences and generate a set of K object (or slot) latent variables representing what is depicted in the image sequences. In this implementation, the object's latent variables remain constant across all frames of the input 104. That is, their distribution is constant over time and is expected to summarize information across the entire sequence.

[0051] Encoder 102 also outputs a set of T frame latent variables containing the frame latent variables for each frame t in the input sequence. Frame latent variables represent time-varying global / view content in the input image sequence. Importantly, encoder 102 decomposes and factorizes the scene sequence into object latent variables and frame latent variables without supervision.

[0052] By decomposing the latent spatial factor into K object latent variables and T frame latent variables, the number of latent variables required for a video sequence can be reduced from K·T to K+T. The prior knowledge of the latent variables can assume the independence between the object latent variables, frame latent variables, and feature dimensions; for example, it can be defined as... in The unit is Gauss.

[0053] Encoder 102 receives T image frames x1-x t A sequence X (each with dimensions of H×W pixels) is taken as input 104. This sequence can be a sequence of images of a scene captured from multiple viewpoints. As described above, input 104 is processed to determine the corresponding object (or slot) latent variable O and frame latent variable F. The Gaussian approximation posterior q(O,F|X) is parameterized as the output of encoder 102, ε φ (X) (where φ represents the encoder parameter), the output value defines all the latent variables of a given input sequence 104, such as the mean and variance (e.g., standard deviation) on a logarithmic scale.

[0054] More specifically, frame sequence 104 is processed to obtain output 107, which includes a set of S spatial codes for each of the T image frames. Each of the S spatial codes in output 107 is associated with a specific image frame x and is the code for the corresponding region of the image frame. S can be equal to H × W. That is, each region can be a single pixel.

[0055] For example, processing the frame sequence 104 to generate the set of S spatial codes for each of T image frames may include processing each image frame by means of a feature extraction neural network 106, which has multiple feature extraction neural network parameters and is trained to receive image frames as input and provide the set of S spatial codes for that image frame as output. For example, the feature extraction neural network 106 may include a convolutional neural network with one or more convolutional neural network layers trained to output S spatial feature maps at each time step. Each feature map may contain C channels. s may be greater than the number of object latent variables K.

[0056] The output 107 of S×T spatial codes is fed as input to a self-attention neural network 108 comprising one or more self-attention layers. The attention neural network 108 is trained to process the group of S×T spatial codes by applying a self-attention mechanism to generate the output 109 of a group of spatial codes comprising K transformations. For example, the group of S×T spatial codes can be fed as input to the first layer of one or more self-attention layers as layer input and processed by that layer to generate a query and key-value pair for each spatial code. The self-attention mechanism can then apply the query to the key-value pair to determine the group of spatial codes for the transformations. It will be understood that the parameters of the self-attention neural network are the parameters applied to the layer input to generate the learned transformations for the query and key-value pairs.

[0057] The spatial encoding of each transformation corresponds to an image frame time t and an image frame region. The output 109 of the attention neural network 108 can be represented as...

[0058] Output 109 is provided to latent distribution generator 110. Latent distribution generator 110 is configured to aggregate output 109 along separate axes to obtain T frames and K object posterior parameters. Specifically, for each of the K object latent variables O, one or more values ​​for parameterizing the object latent distribution are determined by aggregating the output 109 of the attention neural network 108 over the time frame of the image frame. Similarly, for each of the T frame latent variables F, one or more values ​​for parameterizing the frame latent distribution are determined by aggregating the output 109 of the attention neural network 108 over the region of the image frame.

[0059] Distribution generators can include one or more neural networks, such as multilayer perceptrons (MLPs). For example, a first MLP, mlp, can be trained. O To process output 109, which is the output 109 that is aggregated temporally from the image as input, and outputs the latent distribution of the objects. Similarly, a second MLP can be trained.f To process output 109, the output 109 is aggregated over regions of the image as input, and the output frame latent distribution is processed.

[0060] The latent distribution generator 110 is configured to determine the value of each of the set of object latent variables by sampling from the object latent distribution of the object latent variables, and to generate an output 112 including the set of K object latent variables. in Represents the potential average value of the object, and This represents the object's latent standard deviation. The latent distribution generator 110 is also configured to determine the value of each of the set of frame object variables by sampling from the frame latent distribution of the frame latent variables, and to generate an output 114 including the set of T object latent variables. in Represents the potential average value of the frame and This indicates the potential standard deviation of the frame.

[0061] The attention neural network 108 may include a first or more self-attention layers T1 and a second or more self-attention layers T2. The first or more self-attention layers T1 and the second or more self-attention layers T2 may include a transformer neural network subsystem. Typically, the transformer network subsystem may be a subsystem characterized by a series of self-attention neural network layers. The self-attention neural network layers have attention layer inputs for each element of the input and are configured to apply an attention mechanism to the attention layer inputs to generate an attention layer output for each element of the input. Many different attention mechanisms are available.

[0062] Generally, the attention layer is configured to: apply a learned query linear transformation to each hidden state at each input position in the current input sequence to generate a corresponding query for each input position; apply a learned key linear transformation to (i) each hidden state at each input position and (ii) each hidden state in each slot within the episode and compressed memory to generate a corresponding key for each input position and each memory slot; and apply a learned value linear transformation to (i) each hidden state at each input position and (ii) each hidden state in each slot within the episode and compressed memory to generate a corresponding value for each input position and each memory slot. The attention layer then uses these queries, keys, and values ​​to apply the aforementioned attention mechanism to determine the output sequence of the layer's input sequence. The output sequence typically includes a vector of the corresponding attention input for each hidden state at each input position. Typically, queries, keys, and values ​​are all vectors. As used in this specification, the term "learned" means, for example, operations or values ​​that have been adjusted during the system's training by backpropagation of the gradient of the objective function.

[0063] In some implementations, multi-head attention is used to allow the attention layers to pay attention to information from different representation subspaces at different locations.

[0064] To achieve multi-head attention, the attention layer applies h different attention mechanisms in parallel. In other words, the attention layer consists of h different attention sub-layers, each of which receives the same original query, original key, and original value.

[0065] Each attention sublayer is configured to use a learned linear transformation to transform the original query, as well as the keys and values, and then apply the attention mechanism to the transformed query, keys, and values. Each attention sublayer typically learns different transformations from each other attention sublayer within the same attention layer.

[0066] Specifically, each attention sublayer is configured to apply a learned linear transformation of the query to each original query to generate a layer-specific query for each original query, a learned linear transformation of the key to generate a layer-specific key for each original key, and a learned linear transformation of the value to generate a layer-specific value for each original value. The attention sublayer then uses these layer-specific queries, keys, and values ​​to apply the aforementioned attention mechanism to generate the initial output of the attention sublayer.

[0067] T1 can receive S×T spatial codes as input and output S×T transformed spatial codes. Each of the S×T spatial codes pays attention to all other spatial codes in the S×T spatial codes. If S×T is greater than K, the S×T transformed spatial codes can be reduced to K×T transformed spatial codes. For example, the S×T transformed spatial codes output from T1 can be input to a pooling layer that performs a pooling operation on the output. The output of T1 (or the reduced output) is fed as input to T2, which is trained to output 109. The output of T2 can include the same number of slots as the input of T2. For example, if the output of T1 includes S slots before pooling and K slots after pooling, then the output of T2 can include K slots.

[0068] Each of T1 and T2 can use absolute position embedding. The position embedding can be three-dimensional to represent the spatiotemporal position of each slot. In an alternative implementation, T1 and T2 can use relative position embedding.

[0069] Decoder 120 includes multiple decoder neural network parameters and is trained to process inputs including object latent variable O and frame latent variable F, and produce a generated image sequence as output.

[0070] The image is decoded / generated via a "query" decoder 120. Decoder 120 receives input 122, which includes object latent variables and frame latent variables (e.g., sampled latent variables 112, 114), and processes the input 122 to generate an output 124, which includes a sequence of decoded image frames generated for one or more time steps. For example, output 124 may include image frames for each of a series of decoded image frame time steps.

[0071] There are many types of decoders capable of generating images from a set of latent variables, and any of these can be used. In the example, decoder 120 is a neural network, and can be a multilayer perceptron or a convolutional neural network. In addition to the object and frame latent variables, input 122 may also include the image region to be generated and the time step being decoded.

[0072] The region to be decoded can be, for example, a single pixel. Decoding a sequence of image frames may include, for each pixel of each decoded image frame and for each time step of each decoded image frame, the decoder processes input including object latent variable values, frame latent variable values, information specifying the pixel location, and information specifying the time step. The decoder determines the pixel distribution parameters of each object latent variable in the input for each pixel and for each time step. The decoder may further combine the pixel distributions of each object latent variable to determine a combined pixel distribution, and may sample from the combined pixel distribution to determine the values ​​of the pixel and the time step.

[0073] For example, each pixel can be modeled as a Gaussian mixture model with K components. If we assume all regions and all frames are independent, then any RGB pixel x... t，i The mixed weights (1≤i≤HW) can capture which object k “interprets” the pixel. A decoder 120 can be used to calculate a specific time step t and a specific pixel position l. i The corresponding blending weight for each slot k and pixel reconstruction average μ k,t,i .

[0074]

[0075] By constraining the decoder 120 to operate on individual pixels, using a subset of pixels instead of the entire image as the training target, memory and computation can be advantageously saved. Once the pixels are decoded, the blending weights can be obtained, for example, by taking the softmax of the masked logit over K slots.

[0076]

[0077] Equation (2) defines the full-pixel likelihood, where σ x It is a scalar hyperparameter.

[0078] At the input of time t, when the generation time is 124, the object latent variable O can be compared with the frame latent variable f at the time step being decoded. t Cascade, such as Figure 4 The schematic diagram shows that Figure 4 The schematic diagram illustrates the process of decoding a single image frame at time t. Figure 4 In the middle, the frame latent variable f t With each object's latent variable o 1…k Cascade. For each position l i The sampled latent variables are used to render each pixel and generate a pixel blending model. Frame latent variables, which are concatenated with object latent variables, can be inferred from the same input sequence from which the object latent variables were inferred. Alternatively, frame latent variables can come from different input sequences. This allows global / view content (such as lighting and camera pose) from one input sequence to be transferred to different input sequences.

[0079] In the example implementation, system 100 can be used to generate novel representations. For example, system 100 can be trained to conditionally generate video image sequences based on conditional inputs, which can define one or more objects or object properties of one or more objects in video sequence 104. For example, the conditional inputs can define objects to be included in the generated video image sequence 124, and / or can define object properties. Alternatively or additionally, the conditional inputs can define static or variable viewpoint positions or orientations. Conditional priors can be learned for objects in the scene. For example, a neural network (e.g., a recurrent neural network) can be used to learn latent conditional priors for objects in the scene and consecutive frames.

[0080] Furthermore, although the above example implementation determines both object latent variables and frame latent variables, it should be understood that it is not necessary to determine and / or output both object latent variables and frame latent variables. For example, instead of encoding / inferring frame latent variables, encoder 102 and decoder 120 can be directly conditioned on the ground truth viewpoint / camera pose. That is, neural network 100 can be "view-supervised" rather than operating completely unsupervised. This view-supervised approach can be compared with other view-supervised models as follows... Figure 7 As shown, and discussed in more detail below. In a view-supervised system, during decoding, a novel viewpoint (i.e., a viewpoint not provided to the system during training) can be provided to the decoder to generate a view from an unseen viewpoint.

[0081] Figure 5 This is a flowchart of an exemplary method for training a video object representation neural network system, which can be arranged according to a video object representation neural network system 100.

[0082] In the first step 502, a video sequence (such as video sequence 104) is obtained. The video sequence comprises a sequence of T image frames, one image frame at each of a series of times t. In step 504, each image frame in the video sequence is processed (e.g., by feature extraction network 106) to generate a set of S spatial codes for each of the T image frames, thereby generating a set of S×T spatial codes for the video sequence.

[0083] In step 506, the S×T spatially encoded groups are processed by a self-attention neural network by applying a self-attention mechanism (e.g., via self-attention network 108) to the layer input (e.g., layer input 107) comprising a group of S×T spatial codes, to generate an output (e.g., output 109) comprising a group of transformed spatial codes. Each transformed spatial code corresponds to an image frame and an image frame region in the obtained video sequence. Applying a self-attention mechanism to the layer input includes applying the self-attention mechanism to the spatial codes of different image frames in the obtained video sequence.

[0084] In step 508, for each of the set of object latent variables O, one or more values ​​for the object latent distribution used to parameterize the object latent variables are determined by aggregating the spatially encoded set of the transformation over the time of the image frame.

[0085] In the case where a video object representation neural network system is to infer frame latent variables, in optional step 510, for each of a set of frame latent variables F, one or more values ​​of the frame latent distribution used to parameterize the frame latent variables can be determined by aggregating the set of spatially encoded transforms over the region of the image frame. Alternatively, as described above, the encoder can alternatively be directly conditioned on the viewpoint.

[0086] In step 512, the value of each of the group of object latent variables is determined by sampling from the object latent distribution of the object latent variables.

[0087] In the case where the video object representation neural network system is to infer frame latent variables, in optional step 514, the value of each of the set of frame latent variables can be determined by sampling from the frame latent distribution of the frame latent variables. Alternatively, as described above, the encoder may not infer frame latent variables, in which case the frame latent distribution is not inferred.

[0088] In step 516, a sequence of decoded image frames is generated. That is, the values ​​of the determined object latent variables (and, where appropriate, the values ​​of any determined frame latent variables) are processed (e.g., by decoder 120) to generate a sequence of decoded image frames (e.g., output 124).

[0089] In this implementation, the system is trained by backpropagating the gradient of the objective function through an image frame decoder neural network, a self-attention neural network, and a feature extraction neural network. Backpropagation via the sampling step may involve a so-called reparameterization technique (Kingma et al., arXiv1312.6114).

[0090] In step 518, the system is trained by adjusting the values ​​of at least the parameters of the self-attention neural network to depend on i) a measure of the difference between the sequence of T image frames and the sequence of decoded image frames, ii) the difference between the latent distribution of each object and the prior object latent distribution, and optionally (i.e., in the case of generating frame latent variables) iii) the difference between the latent distribution of each frame and the prior frame latent distribution to optimize the objective function.

[0091] Training may include backpropagation. Specifically, training may include backpropagating the gradient of the objective function through a decoder neural network (e.g., decoder 120 generating output 124) and a self-attention neural network (e.g., self-attention network 108) that generates the sequence of decoded image frames. The gradient of the objective function may further be backpropagated through a feature extraction neural network (e.g., feature extraction network 106) that generates the spatial encoding of this set of networks. However, it should be understood that the networks in system 100 (such as, for example, the feature extraction neural network) may be trained separately.

[0092] The objective function (to minimize) can be a negative lower bound, for example, the evidence lower bound (ELBO) as defined in equation (3).

[0093]

[0094] Where, α, β o and β f These are representative weights, where q represents encoder 102, p represents decoder 120, and D... KL It is a measure of distributional dissimilarity, such as the Kullback-Leibler divergence.

[0095] In other words, the log-likelihood of data can be determined by the number of decoded pixels (T). d H d W d Normalization is achieved using 1 / T to allow decoding of fewer than all input pixels. Advantageously, this helps scale the decoder size without reducing the learned signal due to the pervasive correlation between neighboring pixels. It has been found that normalization via 1 / T... d H d W d Normalization helps ensure consistent learning dynamics regardless of how many pixels are selected for decoding. α can be set to 1, but it can also be adjusted, for example, if β... o and β f The proportion is too small and therefore numerically unstable. The example implementation uses β. o =β f .

[0096] In step 518, the training system may further include adjusting the values ​​of the parameters of the feature extraction neural network (such as feature extraction neural network 106). In step 518, the training system may also, or alternatively, include adjusting the values ​​of the parameters of the decoder neural network (such as decoder neural network 120).

[0097] Once trained, either the decoder or encoder portion of the system can be used independently. For example, the encoder portion can be used to obtain one or more attributes of one or more objects depicted in a video sequence from object latent variables or from values ​​used to parameterize the object latent distribution, such as the mean of the distribution. Attributes such as object size, color, texture, position in 1D, 2D, or 3D, or motion can be obtained. Similarly, once trained, a video object representation neural network system, more specifically, only the encoder portion, can be used to obtain one or more global attributes of frames in a video sequence from frame latent variables or from values ​​used to parameterize the frame latent distribution, such as the position or orientation of the (changing) viewpoint of the video sequence, for example, pose.

[0098] Figure 6 This is a flowchart depicting an example process for obtaining one or more attributes of one or more objects depicted in a video sequence. In step 602, a video sequence (such as video sequence 104) is obtained. The video sequence comprises a sequence of T image frames, with one image frame at each point in a series of time t.

[0099] In step 604, for example, each image frame in the video sequence is processed by the trained feature extraction network 106 to generate a set of S spatial codes for each of the T image frames, thereby generating a set of S×T spatial codes for the video sequence.

[0100] In step 606, a trained self-attention neural network processes the S×T spatially encoded groups by applying a self-attention mechanism to the layer input (such as layer input 107), which includes a group of S×T spatial codes, to generate an output (such as output 109) that includes a group of transformed spatial codes. Each transformed spatial code corresponds to an image frame and an image frame region in the obtained video sequence. Applying the self-attention mechanism to the layer input includes applying the self-attention mechanism to the spatial codes of different image frames in the obtained video sequence.

[0101] In step 608, for each of the set of object latent variables O, one or more values ​​for the object latent distribution used to parameterize the object latent variables are determined by aggregating the spatially encoded group of the transformation over the time of the image frame.

[0102] In step 610, the value of each of the group of object latent variables is determined by sampling from the object latent distribution of the object latent variables.

[0103] In step 612, the value of each of the determined latent variables for the objects is used to obtain one or more attributes of one or more objects depicted in the video sequence.

[0104] In the case where the video object representation neural network system is to infer frame latent variables, in optional step 614, the value of each of the set of frame latent variables can be determined by sampling from the frame latent distribution of the frame latent variables. Alternatively, as described above, the encoder may not infer frame latent variables, in which case the frame latent distribution is not inferred.

[0105] It should be understood that, although Figure 6 It is described based on determining object latent variables, but the method may alternatively or additionally include determining frame latent variables to determine one or more global attributes of frames in a video sequence.

[0106] In another example, only the decoder portion of the system might be needed—the part of the system that takes the video sequence input and is configured to determine the potential distribution of objects (and optionally frames). The encoder portion may optionally include parts configured to sample from these distributions (in other implementations, the information may be derived, for example, from the average of these distributions). In some applications, once trained, only the trained decoder portion of the system might be needed, for example, for video sequences used to generate 2D or 3D images. Figure 7This is a flowchart describing an example process of generating video sequences using a trained video generation neural network system. In step 702, the value of each of a set of object latent variables is determined by sampling from the corresponding prior object latent distributions of the object latent variables. Following step 702, steps 704, 706, and 708 are performed for each pixel of each generated image frame and for each time step of each generated image frame. In step 704, the trained decoder neural network processes the determined values ​​of the object latent variables, information about the specified pixel location, and information about the specified time step to determine parameters for the pixel distribution of each object latent variable for the pixel and for the time step. In step 706, the trained decoder neural network combines the pixel distributions of each object latent variable to determine a combined pixel distribution. The processing in step 706 may further include processing the determined values ​​of the object latent variables, information about the specified pixel location, and information about the specified time step to determine a mixed weight for each object latent variable for the pixel and for the time step, and combining the pixel distributions of each object latent variable weighted by the corresponding mixed weights to determine the combined pixel distribution. In step 708, the trained decoder neural network samples from the combined pixel distribution to determine the pixel value at that time step.

[0107] An example of a view-supervised video object representation neural network was experimentally compared with similar models GQN (EslamiSMA et al., Neural scene representation and rendering, Science, June 15, 2018; 360(6394): 1204-1210. doi: 10.1126 / science.aar6170.PMID:29903970) and NeRF (arXiv:2003.08934).

[0108] Figure 8 This illustrates a view-supervised version of a video object representation neural network (in... Figure 8A comparison of scene representation and view interpolation capabilities among SIMONe, NeRF-VAE, and GQN. All models partially observe a procedurally generated “Playroom” dataset from a given sequence of frames (4 of the 16 input frames fed to the model are visualized). The novel view is decoded on a circular trajectory around the room, with yaws linearly spaced in [-π, π]. It can be seen that NeRF-VAE preserves very little object structure, obscuring almost all objects in the scene. However, NeRF does understand the geometry of the generated room and is able to infer wall colors. GQN produces a more detailed reconstruction but overfits to a particular view and cannot interpolate smoothly. GQN creates an illusion of content. On the other hand, SIMONe produces a clear reconstruction of fine object structure. Even when it observes objects at varying distances, it correctly positions and resizes them with a completely new view. This makes SIMONe a strong choice over NeRF-VAE and GQN-style models when prioritizing capturing scene structure in diverse examples. SIMONe also segments the scene.

[0109] It should be understood that a video object representation neural network can be provided, which processes the input sequence to determine and output only object latent variables or only frame latent variables.

[0110] In another experiment, an unsupervised video object representation neural network (inferring both object latent variables and frame latent variables) was compared with other viewpoint-unsupervised approaches. Table 1 shows the comparisons with MONet (arXiv:1901.11390), Slot Attention (SA) (arXiv:2006.15055), and video models (S-IODINE). Figure 1 The video objects described represent the segmentation performance of the neural network (labeled SIMONe in Table 1) using the adjusted Rand exponent ARI-F based on the foreground object. Static and video ARI-F scores are calculated separately. For static ARI-F, the model is evaluated for each still image. For video ARI-F, the model is evaluated across space and time, treating the complete trajectory of the object as a single class. Therefore, video ARI-F penalizes models that cannot stably track objects. This is especially true for slot attention, as it tends to output objects in an arbitrary order for each frame in the sequence.

[0111]

[0112] Table 1

[0113] Table 2 provides two example possible architectures for System 100. It will be understood that the architectures depicted in Table 2 are not intended to be restrictive and are provided only as illustrations of possible architectures. In Table 2, c is the number of channels, k is the kernel size, s is the stride, and Pos.MPL is a multilayer perceptron used for localization coding. MLP([m, n]) refers to a two-layer MLP, each with m and n units respectively; GRU is a gated recurrent unit.

[0114]

[0115] Table 2: Exemplary Architecture

[0116] In the experiments, different approaches to achieving better pixel reconstruction were explored using object latent variables and frame latent variables. In the initial experiments, the sampled object latent variable o k and frame latent variable f t The samples are copied and spatially tiled across all pixels and time points before being fed to the decoder (in the experiment, a convolutional decoder). However, it was found that independent samples can be taken to seed the first stage of the decoder before applying the convolutional decoder stack. Furthermore, it was found beneficial to perform independent sampling across time when appropriate. In summary, samples are taken (independently for all pixels I = 64.64 and time frame T). k,i,t ~q(o k |X) and (independently for all slots K and all pixels I = 64.64) sample f k,i,t ~q(f t |X). This was found to improve performance, especially in the early stages of training.

[0117] I don't want to be bound by theory, which may be due to replicating the same samples across all spatial / temporal dimensions in the sequence. k and f t The gradient bias involved is reduced by using multiple independent samples as regularization terms.

[0118] A (trained) video object representation neural network system, more specifically, only the encoder part, can be used to obtain one or more attributes of one or more objects depicted in a video sequence from object latent variables, or from values ​​used to parameterize the object latent distribution, such as the mean of the distribution. Attributes such as object size, color, texture, position in 1D, 2D, or 3D, and motion can be obtained. Similarly, a (trained) video object representation neural network system, more specifically, only the encoder part, can be used to obtain one or more global attributes of frames in a video sequence from frame latent variables, or from values ​​used to parameterize the frame latent distribution, such as the pose at a specific frame in the video sequence.

[0119] A (trained) video object representation neural network system, more specifically, only the encoder part, can be used to classify images using defined latent variables. For example, one or more defined object latent variables, or values ​​used to parameterize the object latent distribution, can be fed as input to one or more classifier neural network layers. Many image classifiers exist, and any suitable model can be used. More generally, the video object representation neural network system can be configured to generate any suitable neural network output representing entities. For example, the neural network output can be a classification output (e.g., defining a score for each of a set of possible categories), a regression output, a sequence output (i.e., including a sequence of output elements), a segmentation output, or a combination thereof.

[0120] The identified latent variables (of a trained system) can be used as input to another system, for example, when performing machine learning tasks on network inputs. Example tasks can include feature-based retrieval, clustering, near-repetition detection, validation, feature matching, domain adaptation, video-based weakly supervised learning; and for videos, such as object tracking across video frames, gesture recognition of gestures performed by entities depicted in a video.

[0121] A (trained) video object representation neural network system, more specifically, only the decoder portion, can be used to generate new video sequences, including a new sequence of decoded image frames. This can be achieved by determining the value of each of the set of object latent variables by sampling from a prior object latent distribution of object latent variables, determining the value of each of the set of frame latent variables by sampling from a prior frame latent distribution of frame latent variables (which may be the same as the prior object latent distribution), and processing the determined values, specifically using the image frame decoder neural network as described above, to generate a new sequence of decoded image frames. Optionally, the image frame decoder neural network can be conditional on the conditional input of the image frame decoder neural network as described above, i.e., image frames can be generated based on the conditional input.

[0122] The (trained) encoder portion of a video object representation neural network system can be used to determine the values ​​of the set of object latent variables and frame latent variables, and then one or more of these can be modified and fed to the decoder of the video object representation neural network to generate a modified version of the input video sequence.

[0123] In some implementations, once trained, a video object representation neural network system, more specifically, only the encoder portion, can be used to control an agent to perform specific tasks while interacting with an environment. The agent can be a mechanical agent such as a robot or autonomous vehicle, and the environment can be a real-world environment. This usage can include, for example, obtaining observations characterizing the current state of the environment over consecutive time steps, including observations of a video sequence, such as by obtaining consecutive image frames of the video sequence. The trained video object representation neural network system can then be used to process the observed video sequence to obtain values ​​for parameterizing the latent distribution of the objects and for parameterizing the latent distribution of the frames, and optionally, latent variables of the objects and frames. These can then be processed by a control system to generate a control output, which in turn processes the data from the observations. In addition to the latent variables or their distributions, the observations can also be provided to the control system. The control system can include, for example, an action selection policy neural network in a reinforcement learning system, and the control output can include the action selection policy output. An action to be performed by the agent can then be selected in response to the observation. This action can be implemented by the agent using the control output, or the agent can be provided with control outputs to implement the action.

[0124] This can include providing motion data to the agent to perform actions and / or control signals or data, such as electronic control data, like motor control data. That is, control outputs can include motion data, control signals, or similar. For example, control outputs can include control signals for controlling the physical behavior of a mechanical agent (e.g., a robot), such as the torque of a robot joint, or higher-level control commands; or control signals for controlling an autonomous vehicle, such as controlling the torque of vehicle components, or higher-level control commands. Selected actions can define, for example, the position, velocity, or force / torque / acceleration data of one or more joints of a robot or a component of another mechanical agent.

[0125] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, when operated, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0126] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagated signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0127] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0128] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple collaborative files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected via a data communication network.

[0129] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0130] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0131] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. These processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0132] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few examples.

[0133] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0134] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user as a response.

[0135] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production, namely inference, workloads.

[0136] Machine learning frameworks can be used to implement and deploy machine learning models, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0137] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0138] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0139] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0140] Similarly, although the operations are described in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0141] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for training a video object representation neural network system, the video object representation neural network system including a self-attention neural network, the self-attention neural network including one or more self-attention layers and having a plurality of self-attention neural network parameters, each self-attention layer being configured to apply a self-attention mechanism on the layer input, the method comprising: Obtain a video sequence consisting of T image frames, one image frame at each time point in a series of time points; Each image frame is processed to generate a set of S spatial codes for each of the T image frames, thereby obtaining a set of S×T spatial codes for the video sequence, wherein each spatial code of the image frame includes the coding of the corresponding region of the image frame; The S×T spatial codes are processed by the self-attention neural network by applying the self-attention mechanism to the layer input, which includes the group of S×T spatial codes, to generate a group of transformed spatial codes, each of which corresponds to an image frame time and an image frame region. The application of the self-attention mechanism to the layer input includes applying the self-attention mechanism to the spatial codes of different image frames. For each of a set of object latent variables, one or more values ​​for parameterizing the object latent distribution of that object latent variable are determined by aggregating the set of spatially encoded transformations over time in the image frame. The value of each of the group of object latent variables is determined by sampling from the object latent distribution of the object latent variables; Process the values ​​of the determined object's latent variables to generate a decoded sequence of image frames; and The system is trained by adjusting at least the values ​​of the parameters of the self-attention neural network to optimize the objective function, which depends on i) a measure of the difference between the sequence of T image frames and the sequence of decoded image frames, and ii) the difference between the latent distribution of each object and the prior object latent distribution.

2. The method according to claim 1, further comprising: By aggregating groups of spatially encoded transforms over regions of an image frame, one or more values ​​of the frame latent distribution used to parameterize each of the set of frame latent variables are determined. The value of each of the set of frame latent variables is determined by sampling from the frame latent distribution of the frame latent variables; The process of generating the decoded image frame sequence also includes processing the values ​​of the determined frame latent variables to generate the decoded image frame sequence.

3. The method according to claim 2, wherein, Training the system includes at least adjusting the values ​​of the parameters of the self-attention neural network to optimize the objective function based on the difference between the latent distribution of each frame and the latent distribution of prior frames.

4. The method according to any one of claims 1-3, wherein, Each spatial code includes a feature map of a corresponding region of an image frame, wherein processing each image frame to generate the set of S spatial codes for each of T image frames includes processing each image frame using a feature extraction neural network with multiple feature extraction neural network parameters to generate a feature map of each region of the image frame; and wherein training the system includes adjusting the values ​​of the feature extraction neural network parameters.

5. The method according to any one of claims 1-3, comprising using an image frame decoder neural network having a plurality of image frame decoder neural network parameters to process the values ​​of determined object latent variables to generate the decoded image frame sequence; and in, Training the system also includes adjusting the values ​​of the parameters of the image frame decoder neural network.

6. The method according to claim 5, wherein, The decoded image frame sequence includes image frames for each of a series of decoded image frame time steps, and wherein generating the decoded image frame sequence using an image frame decoder neural network includes, for each pixel of each decoded image frame and for each decoded image frame time step: The image frame decoder neural network processes the determined values ​​of object latent variables, information about specified pixel locations, and information about specified time steps to determine the parameters of the pixel distribution for each object latent variable for the pixel and the time step. Combine the pixel distributions of the latent variables for each object to determine the combined pixel distribution; and Sample from the combined pixel distribution to determine values ​​for the pixel and for the time step.

7. The method of claim 6, further comprising: for each pixel of each decoded image frame and for each decoded image frame time step: The image frame decoder neural network is used to process the determined values ​​of object latent variables, information about specified pixel locations, and information about specified time steps to determine a mixed weight for each object latent variable for the pixel and the time step; and wherein Combining the pixel distributions of each object's latent variables to determine the combined pixel distribution includes combining the pixel distributions of each object's latent variables weighted by their respective blending weights.

8. The method according to claim 5, further comprising: Obtain conditional input, which defines one or more objects or object attributes of one or more objects in the video sequence, or defines the viewpoint position or orientation; and The conditional input is provided to the self-attention neural network and the image frame decoder neural network.

9. The method according to any one of claims 1-3, wherein, Processing the group of S×T spatial codes using a self-attention neural network includes using one of the one or more self-attention layers to process the layer input comprising the group of S×T spatial codes to generate a query and a key-value pair for each spatial code, and applying the query to the key-value pair using the self-attention mechanism to determine the group of spatial codes to be transformed; and wherein the parameters of the self-attention neural network include parameters for the learned transformation applied to the layer input to generate the query and the key-value pair.

10. The method according to any one of claims 1-3, wherein, The S×T spatially encoded groups are processed using a self-attention neural network, including: The S×T spatially encoded groups are provided as layer inputs to the first self-attention layer of the self-attention neural network to generate the output of the first self-attention layer; and The output of the first self-attention layer is provided to the second self-attention layer of the self-attention neural network to generate a set of spatial codes for the transformation; including The number of spatial codes is reduced from S×T to K×T, where K is the number of latent variables of the object and K is less than S.

11. The method according to claim 10, wherein, The reduction includes applying a pooling operation to the output of the first self-attention layer.

12. The method according to any one of claims 1-3, wherein, Training the system includes backpropagating the gradient of the objective function by generating the image frame decoder neural network that generates the sequence of decoded image frames, the self-attention neural network, and the feature extraction neural network that generates the set of spatial codes.

13. The method according to any one of claims 1-3, further comprising using a trained video object representation neural network system to obtain one or more attributes of one or more objects depicted in the video sequence from latent variables of the objects or from values ​​used to parameterize the latent distribution of the objects.

14. The method of claim 2, further comprising using a trained video object representation neural network system to obtain the position or orientation of the viewpoint of the video sequence from the frame latent variables or from values ​​used to parameterize the frame latent distribution.

15. The method according to any one of claims 1-3, further comprising using a trained video object representation neural network system to generate a new video sequence including a decoded sequence of new image frames in such a way as: The value of each of the set of object latent variables is determined by sampling from the prior object latent distribution of the object latent variables; The values ​​of the determined object's latent variables are processed to generate a new sequence of decoded image frames.

16. The method of claim 15, further comprising: The value of each of the set of frame latent variables is determined by sampling from the prior frame latent distribution of the frame latent variables; and The values ​​of the determined object latent variables and the determined frame latent variables are processed to generate a new sequence of decoded image frames.

17. The method according to any one of claims 1-3, further comprising using a trained video object representation neural network system to generate a modified video sequence including a decoded modified image frame sequence in such a way that: Obtain an input video sequence that includes the input image frame sequence; Process each input image frame to obtain a spatially encoded group of the input video sequence; A self-attention neural network is used to process the spatially encoded group of the input video sequence to generate a transformed spatially encoded group; The values ​​of the latent variables of the group of objects are determined by sampling from the latent distribution of objects determined according to the spatial encoding of the transformation; Modify the values ​​of the identified set of latent variables to obtain a modified set of latent variables; and The latent variables of this set of modifications are processed to generate the decoded modified image frame sequence.

18. The method of claim 17, further comprising determining the value of a set of frame latent variables by sampling from a set of frame latent distributions determined according to the spatial coding of the transform; Modify the values ​​of the identified latent variables for that set of frames to obtain the modified latent variables for that set.

19. The method of any one of claims 1-3, further comprising using a trained video object representation neural network system to control an agent to perform tasks while interacting with an environment, the method comprising: Obtain observations characterizing the current state of the environment, including a sequence of observed video images; The observed video sequence is processed using a trained video object representation neural network system to obtain values ​​for parameterizing the latent distribution of the object; The control system processes the data from the observations, including processing data from values ​​used to parameterize the potential distribution of the object, to generate a control output; and Using the control output in response to the observation, select the action to be performed by the agent.

20. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method according to any one of claims 1-19.

21. A system for training a video object representation neural network system, comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation according to any one of claims 1-19.

22. A computer-implemented video processing neural network system, comprising: The trained feature extraction neural network is configured as follows: Receive a video sequence comprising a sequence of T image frames, one image frame at each time point in a series of times, and Each image frame is processed to generate a set of S spatial codes for each of T image frames, thereby obtaining an S×T set of spatial codes for the video sequence, wherein each spatial code includes the encoding of a corresponding region of the image frame; and A trained self-attention neural network includes one or more self-attention layers, each configured to apply a self-attention mechanism to its input, wherein the self-attention neural network is configured as follows: By applying the self-attention mechanism to the layer input comprising a group of S×T spatial codes, a self-attention neural network processes the group of S×T spatial codes to generate a group of transformed spatial codes, each transformed spatial code corresponding to an image frame time and an image frame region, wherein applying the self-attention mechanism to the layer input includes applying the self-attention mechanism to the spatial codes of different image frames; and The video processing neural network system is configured as follows: For each of a set of object latent variables, one or more values ​​of the object latent distribution used to parameterize the object latent variable are determined by aggregating the spatially encoded set of the transform over time in the image frame; and one or more attributes of one or more objects depicted in the video sequence are determined from the values ​​used to parameterize the object latent distribution.

23. A computer-implemented video generation neural network system, configured as follows: The value of each of a set of object latent variables is determined by sampling from the corresponding prior object latent distributions of the object latent variables; The system includes: A trained image frame decoder neural network is used to process the values ​​of determined object latent variables to generate a video sequence comprising a sequence of image frames generated at consecutive time steps, wherein the image frame decoder neural network is configured to, for each pixel of each generated image frame and for each time step of each generated image frame: The values ​​of determined object latent variables, information about specified pixel locations, and information about specified time steps are processed to determine parameters of the pixel distribution for each object latent variable for the pixel and for the time step. Combine the pixel distributions of the latent variables for each object to determine the combined pixel distribution; and Samples are taken from the combined pixel distribution to determine values ​​for the pixel and for the time step.

24. The computer-implemented video generation neural network system according to claim 23, wherein, The image frame decoder neural network is also configured to, for each pixel of each decoded image frame and for each decoded image frame time step: The values ​​of determined object latent variables, information about specified pixel locations, and information about specified time steps are processed to determine a mixed weight for each object latent variable for the pixel and for the time step. and The pixel distribution of each object's latent variable, weighted by its corresponding blending weights, is combined to determine the combined pixel distribution.

Images