Unsupervised learning of object representation from video sequence using attention over spatio-temporal domain

Abstract

To provide unsupervised learning of an object representation from a video sequence using attention over a spatio-temporal domain.SOLUTION: A computer-implemented video generation neural network system configured to determine a value of each object latent variable in a set of object latent variables by sampling from the respective previous object latent distributions for the object latent variables comprises a trained image frame decoder neural network configured to: process the determined values of the object latent variables for each pixel of each generated image frame and each generated image frame time step, to determine parameters of pixel distributions for each of the object latent variables; and combine the pixel distributions for each object latent variable to determine a combined pixel distribution, and to determine the values of pixels and time steps by sampling from the combined pixel distribution.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This document relates to training machine language models to characterize global time-varying elements such as objects and viewpoints in video frames. [Background technology]

[0002] A machine learning model receives inputs and generates an output, e.g., a predicted output, based on the received inputs and the values ​​of the parameters of the model.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] arXiv:2010.11929 [Non-Patent Document 2] Kingma et al., arXiv1312.6114 [Non-Patent Document 3] Eslami SMA et al., "Neural scene representation and rendering.", Science., June 15, 2018, 360(6394), pp. 1204-1210, doi:10.1126 / science.aar6170, PMID:29903970 [Non-Patent Document 4] arXiv:2003.08934 [Non-Patent Document 5] arXiv:1901.11390 [Non-Patent Document 6] arXiv:2006.15055 Summary of the Invention [Means for solving the problem]

[0005] This specification describes a system and method, generally implemented as one or more computer programs on one or more computers at one or more physical locations, for training a video object representation neural network system. The encoder portion of the trained system may be used to determine characteristics of different objects in a video sequence, or to determine a viewpoint position for the video sequence, or both. The decoder portion of the trained system may be used to generate new video sequences. The encoder and decoder may be used together to modify a video sequence.

[0006] In one aspect, a computer-implemented method is provided for training a video object representation neural network system that includes a self-attention neural network, e.g., a transformer neural network layer, with one or more self-attention layers, each configured to apply a self-attention mechanism to layer inputs and having a plurality of self-attention neural network parameters.

[0007] The method includes acquiring a video sequence including T image frames, one for each time in succession, and processing each of the image frames to generate a set of S spatial encodings for each of the T image frames. The method thereby acquires a group of S×T spatial encodings for the video sequence, each spatial encoding including an encoding of a respective region or patch of the image frame. The video may be acquired, for example, from a camera or LIDAR sensor, i.e. the image frames may include a point cloud.

[0008] The method further includes processing the group of S×T spatial encodings using an attention neural network, in particular by applying a self-attention mechanism to the layer inputs including the group of S×T spatial encodings to generate a group of transformed spatial encodings, possibly after a further self-attention processing step. Each of the transformed spatial encodings corresponds to an image frame time and an image frame region. The group of transformed spatial encodings may have the same resolution or may have different, e.g. lower, resolutions, i.e. there may be S×T or less transformed spatial encodings. In an implementation, applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism via the spatial encodings of the different image frames, i.e. to image frames from different times in successive times.

[0009] The method further comprises determining, for the object latent variable, typically each vector in the set of vectors, one or more values ​​that parameterize an object latent distribution of the object latent variable, e.g., the mean or standard deviation (or variance) of the distribution. This is done by aggregating a group of transformed spatial encodings over the time of the image frames. The method may also determine, for each frame latent variable in the set of frame latent variables, one or more values ​​that parameterize a frame latent distribution of the frame latent variable by aggregating a group of transformed spatial encodings over the region of the image frames. The method determines a value for each object latent variable in the set of object latent variables by sampling from the object latent distribution for the object latent variable. The method may also determine a value for each frame latent variable in the set of frame latent variables by sampling from the frame latent distribution for the frame latent variable.

[0010] The method processes the determined values ​​of the object latent variables (and processes the determined values ​​of the determined frame latent variables) to generate a decoded sequence of image frames. There are many types of decoders capable of generating images from a set of latent variables, any of which may be used. Exemplary decoders are described below.

[0011] The method may then include training the system by adjusting values ​​of at least the self-attention neural network parameters to optimize an objective function in response to: i) a measure of difference between the sequence of T image frames and the decoded sequence of image frames; ii) a difference between each of the object latent distributions and a previous object latent distribution. Once the objective function has been determined, it may be optimized in response to iii) a difference between each of the frame latent distributions and a previous frame latent distribution. The previous object latent distribution and the previous frame latent distribution may be the same distribution, e.g., a unit Gaussian distribution.

[0012] Implementations of the above-described methods provide a trained video object representation neural network system. In some applications, after training, only the encoder portion of the system may be needed, i.e., from the video sequence input to the portion configured to determine these distributions, possibly including the portion configured to sample from the object latent distribution and (possibly) the frame latent distribution (in other implementations, information may be derived, for example, from the average of these distributions). In some applications, after training, only the trained decoder portion of the system may be needed, for example, to generate a video sequence of 2D or 3D images, as described below.

[0013] An implementation of the trained system provides object information in a set of object latent variables. For example, in an implementation, these may be substantially disentangled, i.e., different object latent variables correspond to different variance factors of objects in a scene and object properties such as size, color, position in each dimension, and motion. Similarly, in an implementation, different frame latent variables may disentangle different global properties of frames of a video sequence, such as viewpoint position in each dimension. Further advantages of the method and system are described below.

[0014] In an implementation, each spatial encoding includes a feature map, i.e., a map of visual features of a respective region or "patch" of the image frame. Processing the image frames to generate the set of spatial encodings may use a feature extraction neural network, e.g., a convolutional neural network, having a number of feature extraction neural network parameters to generate the feature map for each region of the image frame. Training the system may include tuning values ​​of the feature extraction neural network parameters.

[0015] In an implementation, the determined values ​​of the object latent variables (and possibly the frame latent variables) are processed using an image frame decoder neural network having a plurality of image frame decoder neural network parameters to generate a decoded sequence of image frames, where training the system includes adjusting values ​​of the image frame decoder neural network parameters.

[0016] In some implementations, the decoded sequence of image frames includes an image frame for each time step of the successive decoded image frame time steps. When generating the decoded sequence of image frames, an image frame decoder neural network is used to generate a parameter of a pixel distribution for each of the object latent variables for each decoded image frame pixel and each decoded image frame time step. Specifically, the image frame decoder neural network processes the determined value of the object (and possibly the frame latent variable), as well as the information specifying the pixel's location and the information specifying the time step, to determine a parameter of a pixel distribution (μ) for each of the object latent variables. The pixel distributions for each of the object latent variables are then combined to determine a joint pixel distribution, which is then sampled to determine the pixel and time step values. For example, in one implementation, the frame latent variable (for decoding the image frame) is concatenated with each of the object latent variables to provide an input, more specifically, a set of inputs, to the image frame decoder neural network.

[0017] The step of generating a decoded sequence of image frames may include determining a mixture weight for each of the object latent variables, each pixel, and each time step. The pixel distributions for each of the object latent variables may then be combined according to the respective mixture weights, possibly normalized. In such an implementation, the per-pixel mixture weights for each object latent variable provide a soft segmentation mask for the object corresponding to the object latent variable, e.g., segmenting each image frame into regions representing different objects in that image frame.

[0018] Any suitable measure of the difference between the sequence of image frames and the decoded sequence of image frames may be used, for example the cross-entropy loss, the squared error loss, or the Huber loss. Any suitable measure of the difference between the distributions may be used, for example the KL divergence or the Jensen-Shannon divergence or another measure of the distance between the distributions.

[0019] In some implementations, the method may include obtaining conditioning inputs that define one or more objects or object properties for one or more objects in the video sequence, or that define a viewpoint position or orientation, e.g., a pose. The conditioning inputs may be provided to the self-attention neural network and the image frame decoder neural network during training. In this manner, the system may be trained to generate video image sequences conditioned on conditioning inputs that define, for example, objects or object properties to include in the generated video image sequence, or arbitrarily varying viewpoints for the generated video sequence.

[0020] As mentioned above, the self-attention layer may be a Transformer Neural Network layer. An application of the Transformer Neural Network architecture to a computer version is described in arXiv:2010.11929.

[0021] In an implementation, the step of processing the group of S×T spatial encodings using an attention neural network may include appending location information to each of the S×T spatial encodings that define a respective region of the image frame and image frame time. The location information may include, for example, a location embedding, i.e., an embedding of the image frame region and time. The embedding may include a vector of the same dimensionality as the spatial encoding, and the embedding may be appended to each spatial encoding. The embedding may be learned or predefined.

[0022] In an implementation, the step of processing the group of S×T spatial encodings using an attention neural network includes processing a layer input including the group of S×T spatial encodings using one of one or more self-attention layers to generate a query and a key-value pair for each of the spatial encodings. A self-attention mechanism may then be used to, for example, apply each of the queries to the keys, more specifically to each of the key-value pairs, to determine the group of transformed spatial encodings. The self-attention may include masked self-attention, in which case not all of the queries may be applied. The self-attention mechanism may be a multi-head attention mechanism. In an implementation, the self-attention neural network parameters comprise parameters of a learned transformation applied to the layer input to generate the query and the key-value pair.

[0023] Generally, the attention mechanism maps a set of query and key-value pairs to an output, where the query, key, and value are all vectors. The output is calculated as a weighted sum of the values, and the weight assigned to each value is calculated by a compatibility function of the query having the corresponding key. The exact self-attention mechanism applied depends on the configuration of the attention neural network. For example, each attention layer can apply a dot product attention mechanism, such as a scaled dot product attention mechanism. In the scaled dot product attention, for a given query, the attention layer calculates the dot product of the query using all the keys, divides each of the dot products by a scale factor, such as the square root of the dimensions of the query and key, and then applies the softmax function over the scaled dot products to obtain the weights for the values. The attention layer then calculates the weighted sum of the values according to these weights.

[0024] In some implementations, the step of processing a group of S×T spatial encodings using an attention neural network includes providing the group of S×T spatial encodings as a layer input to a first self-attention layer of the attention neural network to generate a first self-attention layer output, and providing the first self-attention layer output to a second self-attention layer of the attention neural network to generate a group of transformed spatial encodings. In an implementation, this is a step of reducing the number of spatial encodings from S×T spatial encodings to K×T spatial encodings, where K < S is the number of object latent variables. Thus, in an implementation, there may be more feature maps than object latent variables, or the number of object latent variables and feature maps may be the same. Reducing the number in such a way may be done, for example, by pooling in the horizontal and / or vertical directions (of the tiled patches), and optionally normalizing, after the first self-attention layer or after the second self-attention layer.

[0025] In another aspect, there is provided a computer-implemented video processing neural network system comprising a feature extraction neural network configured (trained) to receive a video sequence including a sequence of T image frames, one for each time in succession in time, and to process each of the image frames to generate a set of S spatial encodings for each of the T image frames, thereby obtaining a group of S×T spatial encodings for the video sequence, where each spatial encoding may include an encoding of a respective region of the image frame.

[0026] The computer-implemented video processing neural network system also includes a (trained) self-attention neural network comprising one or more self-attention layers, each configured to apply a self-attention mechanism to a layer input. In an implementation, the self-attention neural network is configured to process the group of S×T spatial encodings using the attention neural network by applying the self-attention mechanism to the layer inputs comprising the group of S×T spatial encodings to generate a group of transformed spatial encodings, each corresponding to an image frame time and an image frame region. Applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism to the spatial encodings of the respective different image frames.

[0027] The video processing neural network system is configured to determine, for each object latent variable in the set of object latent variables, one or more values ​​that parameterize an object latent distribution of the object latent variable by aggregating a group of spatial encodings transformed over time of the image frames, and optionally to determine a value for each object latent variable in the set of object latent variables by sampling from the object latent distribution for the object latent variable.

[0028] The video processing neural network system is also configured to determine, for each frame latent variable in the set of frame latent variables, one or more values ​​that parameterize a frame latent distribution for the frame latent variable by aggregating groups of transformed spatial encodings across regions of the image frame, and optionally to determine values ​​for each frame latent variable in the set of frame latent variables by sampling from the frame latent distribution for the frame latent variable.

[0029] The video processing neural network system is also configured to: i) determine one or more characteristics of one or more objects shown in the video sequence from values ​​that parameterize the object latent distributions, i.e., from the object latent variables or values ​​that parameterize the object latent distributions; and / or ii) determine a viewpoint position or orientation for the video sequence from values ​​that parameterize the frame latent distributions, i.e., from the frame latent variables or values ​​that parameterize the frame latent distributions.

[0030] In another aspect, a computer-implemented video generating neural network system is provided that is configured to determine a value for each object latent variable in a set of object latent variables by sampling from a respective prior object latent distribution for the object latent variable (and optionally determine a value for each frame latent variable in the set of frame latent variables by sampling from a respective prior frame latent distribution for the frame latent variable).

[0031] The system comprises a (trained) image frame decoder neural network that processes the determined values ​​of the object latent variables (and possibly determined values ​​of the frame latent variables) to generate a video sequence comprising a sequence of generated image frames at successive time steps. The image frame decoder neural network is configured to, for each pixel of each generated image frame and each generated image frame time step, process the determined values ​​of the object latent variables (and possibly determined values ​​of the frame latent variables), information specifying a location of the pixel, and information specifying the time step to determine, for the pixel and time step, parameters of a pixel distribution for each of the object latent variables, combine the pixel distributions for each of the object latent variables to determine a joint pixel distribution, and sample from the joint pixel distribution to determine a value for the pixel and time step.

[0032] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0033] The video object representation neural network system, specifically the encoder portion of the system including the feature extraction neural network and the self-attention neural network, can process video sequences of 3D scenes and disentangle and identify, i.e., characterize, multiple objects simultaneously. The system can do this even when the viewpoint, e.g., camera pose, changes without any information about the viewpoint or camera pose being explicitly given. Implementations of the system can process scenes with multiple objects and complex backgrounds.

[0034] Thus, the system can determine an allocentric representation, i.e., a set of object latent variables or their distributions, that is important for many tasks. For example, a system that is not aware of the viewpoint may confuse the distance of an object from the viewpoint and the size of the object; embodiments of the system can eliminate this problem.

[0035] In an embodiment of the system, the set of object latent variables or their distributions provide a stable representation of objects in the scene with respect to the camera pose. In an implementation of the system, this is achieved using spatiotemporal attention across different image frames of the video sequence in combination with a factorized latent space that imposes a separation of static, i.e., substantially time-invariant, object attributes from global or dynamic properties of the video sequence, such as the camera pose. This also enables the system to determine static object motion properties, such as object trajectory summaries.

[0036] The system also determines representations of global, possibly time-varying, scene properties such as viewpoint in a set of frame latent variables or their distributions. The ability to determine viewpoint allows the system to be used, for example, to determine the position of a mobile mechanical agent equipped with a camera or other video capture device.

[0037] Some implementations of the system also have a simpler architecture than some prior systems, which have more complex architectures but less functionality than the present system.

[0038] Implementations of the system can also be used, for example, to generate video sequences of existing scenes as if the scene were from a particular viewpoint, or to generate video sequences of new or modified scenes that include one or more objects. This can be done, for example, by determining values ​​of latent variables from an existing scene, or sampling these values ​​from a prior distribution, and then providing the values ​​to the video object representation neural network system, specifically the decoder portion of the system. This can be used, for example, for viewpoint interpolation or extrapolation, or for planning, for example, when to control a mechanical agent, such as a robot, to perform a task. Unlike some prior systems that can do this for a particular previously trained scene, embodiments of the system can train on different scenes, and then do this for a new, previously unseen scene.

[0039] For example, an encoder of an untrained or trained video object representation neural network system may be used in a reinforcement learning system that controls an agent to perform a task along with or in place of images from the environment in which the agent operates, allowing the reinforcement learning system to learn faster and more effectively, or a decoder of an untrained or trained video object representation neural network system may be used in a model-based reinforcement learning system to predict future states of the environment in learning an action selection policy for controlling the agent.

[0040] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0041] [Figure 1] FIG. 1 illustrates a video object representation neural network system. [Diagram 2] FIG. 2 illustrates an encoder neural network of the video object representation neural network system. [Diagram 3] FIG. 2 illustrates an image frame decoder neural network of the video object representation neural network system configured to decode latent variables into pixel values. [Figure 4] 2 is a schematic diagram of the decoding process performed by a decoder of the video object representation neural network system. [Diagram 5] 4 is a flow diagram of an example process for training a video object neural network using the system. [Figure 6] 1 is a flow diagram of a process for obtaining object characteristics using the trained encoder portion of a video object representation neural network system. [Figure 7] 1 is a flow diagram of a process for generating a sequence of images using the trained decoder portion of a video object representation neural network system. [Figure 8] FIG. 1 illustrates a comparison of a view synthesis task performed by a video object representation neural network system and two other neural network models. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] Like reference numbers and designations in the various drawings indicate like elements.

[0043] FIG. 1 illustrates a schematic of a video object representation neural network system 100 that may be implemented by one or more computers at one or more locations. The video object representation neural network system 100 comprises an inference network (encoder 102) trained to infer latent variables from an input sequence of images (frames, dimensions H×W pixels) and a generative model (decoder 120) trained to decode the latent variables generated by the encoder 102 to generate pixel values ​​(i.e., images). The encoder 102 uses spatiotemporal attention-based inference that enables stable object tracking. The decoder 120 can combine object latent variables of a given sequence with frame latent variables of a different sequence to generate consistent renderings of the same scene content (i.e., objects and their properties, relative repositioning, and segmentation assignments) from different viewpoints overall.

[0044] Specifically, the encoder 102 processes an input image sequence and generates K object (or slot) latent variables that represent the content shown in the image sequence.

[0045]

number

[0046] In the implementation, the object latent variables are invariant across all frames in the input 104. That is, the distribution of the object latent variables is expected to be constant over time and summarize information across the entire sequence.

[0047] The encoder 102 also generates T frame latent variables, including a frame latent variable for each frame t in the input sequence.

[0048]

number

[0049] The encoder 102 outputs a set of latent variables, where the frame latent variables represent the time-varying global / view content across the input image sequence. Importantly, the encoder 102 decomposes and factorizes the scene sequence into object latent variables and frame latent variables in an unsupervised manner.

[0050] By factorizing the latent space into K object latent variables and T frame latent variables, the number of latent variables required to represent a video sequence can be reduced from K*T to K+T. The assumptions on the latent variables can assume independence between the object latent variables, the frame latent variables, and the feature dimensions, e.g., p(O,F)=Π k N(o k |O,I)Π t N(f t |0,I), where N(0,I) denotes the unit Gaussian.

[0051] The encoder 102 outputs T image frames x1 to x t A sequence X of frames (each having dimensions H×W pixels) is received as input 104. The sequence may be a sequence of images of a scene taken from multiple viewpoints. As described above, the input 104 is processed to determine corresponding object (or slot) latent variables O and frame latent variables F. The Gaussian approximation posterior probabilities q(O,F|X) are computed by the encoder 104 ε φ (X), where φ denotes the parameters of the encoder. The encoder 104 outputs values ​​that define the mean and variance (e.g., standard deviation) on a logarithmic scale for all latent variables, for example, given an input sequence 104.

[0052] More specifically, a sequence of frames 104 is processed to obtain an output 107 that includes a set of S spatial encodings for each of the T image frames. Each of the S spatial encodings in output 107 is associated with a particular image frame x and is an encoding of a respective region of the image frame. S may be equal to H×W, i.e., each region may be an individual pixel.

[0053] For example, processing the sequence of frames 104 to generate a set of S spatial encodings for each of the T image frames may include processing each of the image frames through a feature extraction neural network 106, which has a number of feature extraction neural network parameters and is trained to receive an image frame as an input and provide a set of S spatial encodings for the image frame as an output. For example, the feature extraction neural network 106 may comprise a convolutional neural network having one or more convolutional neural network layers trained to output S spatial feature maps at each time step. Each feature map may include C channels. S may be greater than the number of object latent variables K.

[0054] The output 107 of the S×T spatial encodings is provided as an input to a self-attention neural network 108 that includes one or more self-attention layers. The attention neural network 108 is trained to process the group of S×T spatial encodings by applying a self-attention mechanism to generate an output 109 that includes a group of K transformed spatial encodings. For example, the group of S×T spatial encodings may be provided as an input to a first layer of one or more self-attention layers as layer inputs and processed by the layer to generate queries and key-value pairs for each of the spatial encodings. The self-attention mechanism may then apply queries to the key-value pairs to determine the group of transformed spatial encodings. It will be appreciated that the self-attention neural network parameters are parameters of the learned transformations applied to the layer inputs to generate the queries and key-value pairs.

[0055] Each transformed spatial encoding corresponds to an image frame time t and an image frame region. The output 109 of the attention neural network 108 is

[0056]

number

[0057] It may be shown as follows.

[0058] The output 109 is provided to a latent distribution generator 110. The latent distribution generator 110 is configured to aggregate the output 109 along separate axes to obtain T frame posterior probability parameters and K object posterior probability parameters. Specifically, for each object latent variable in the set of K object latent variables O, one or more values ​​that parameterize an object latent distribution of the object latent variable are determined by aggregating the output 109 of the attention neural network 108 over the time of the image frame. Similarly, for each frame latent variable in the set of T frame latent variables F, one or more values ​​that parameterize a frame latent distribution of the frame latent variable are determined by aggregating the output 109 of the attention neural network 108 over the region of the image frame.

[0059] The distribution generator may comprise one or more neural networks, such as a multi-layer perceptron (MLP). For example, a first MLP O processes the output 109 to receive as input the aggregated output 109 over time of the image and generate an object latent distribution

[0060]

number

[0061] Similarly, the second MLP, mlp fprocesses the output 109 to receive as input the aggregated output 109 over a region of the image and calculates a frame latent distribution

[0062]

number

[0063] It can be trained to output

[0064] The latent distribution generator 110 determines a value for each object latent variable in the set of object latent variables by sampling from the object latent distribution for the object latent variable, and generates a set of K object latent variables.

[0065]

number

[0066] where:

[0067]

number

[0068] denotes the object latent mean,

[0069]

number

[0070] The latent distribution generator 110 also determines a value for each frame latent variable in the set of frame latent variables by sampling from the frame latent distribution for the frame latent variable, and generates a set of T frame latent variables

[0071]

number

[0072] where:

[0073]

number

[0074] denotes the frame latent mean,

[0075]

number

[0076] denotes the frame latency standard deviation.

[0077] The attention neural network 108 may comprise a first one or more self-attention layers T1 and a second one or more self-attention layers T2. The first one or more self-attention layers T1 and the second one or more self-attention layers T2 may comprise a transformer neural network subsystem. In general, the transformer network subsystem may be a subsystem characterized by having successive self-attention neural network layers. The self-attention neural network layers have an attention layer input for each element of the input and are configured to apply an attention mechanism to the attention layer inputs to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used.

[0078] In general, the attention layer is configured to apply a learned query linear transformation to each hidden state at each input position in the current input sequence to generate a respective query for each input position, apply a learned key linear transformation to (i) each hidden state at each input position and (ii) each hidden state at each slot in the episodic memory and compressed memory to generate a respective key for each input position and each memory slot, and apply a learned value linear transformation to (i) each hidden state at each input position and (ii) each hidden state at each slot in the episodic memory and compressed memory to generate a respective value for each input position and each memory slot. The attention layer then uses these queries, keys, and values ​​to apply the attention mechanism described above to determine an output sequence for the layer input sequence. The output sequence generally includes a respective attended vector for each hidden state input at each input position. In general, the queries, keys, and values ​​are all vectors. In this specification, the term "learned" means that the actions or values ​​have been adjusted during the training of the system, for example, by backpropagation of the gradient of the objective function.

[0079] In some implementations, the attention layer uses multi-head attention to enable the attention layer to jointly attend to information from different representation subspaces at different locations.

[0080] To implement multi-head attention, the attention layer applies h different attention mechanisms in parallel, in other words, the attention layer includes h different attention sublayers, and each attention sublayer in the same attention layer receives the same original query, original key, and original value.

[0081] Each attention sublayer is configured to transform the original query, keys, and values ​​using a learned linear transformation, and then apply the attention mechanism to the transformed query, keys, and values. Each attention sublayer will generally learn a different transformation from each other attention sublayer in the same attention layer.

[0082] Specifically, each attention sublayer is configured to apply the learned query linear transformation to each original query to generate a layer-specific query for each original query, the learned key linear transformation to each original key to generate a layer-specific key for each original key, and the learned value linear transformation to each original value to generate a layer-specific value for each original value. The attention sublayer then applies the attention mechanism described above using these layer-specific queries, keys, and values ​​to generate initial outputs for the attention sublayer.

[0083] T1 may receive S×T spatial encodings as input and output S×T transformed spatial encodings. Each of the S×T spatial encodings takes note of all other spatial encodings of the S×T spatial encodings. If S×T is greater than K, the S×T transformed spatial encodings may be reduced to K×T transformed spatial encodings. For example, the S×T transformed spatial encodings output from T1 may be input to a pooling layer that performs a pooling operation on the output. The output (or reduced output) of T1 is provided as input to T2, and T2 is trained on the output 109. The output of T2 may include the same number of slots as are input to T2. For example, if the output of T1 includes S slots before pooling and K slots after pooling, the output of T2 may include K slots.

[0084] Each of T1 and T2 may use absolute position embedding. The position embedding may be three-dimensional and indicate the spatial-temporal position of each slot. In an alternative implementation, T1 and T2 may use relative position embedding.

[0085] The decoder 120 comprises a plurality of decoder neural network parameters and is trained to process inputs including object latent variables O and frame latent variables F, and to produce a generated image sequence as output.

[0086] Images are decoded / generated by "interrogating" the decoder 120. The decoder 120 receives an input 122 including object latent variables and frame latent variables (e.g., sampled latent variables 112, 114) and processes the input 122 to generate an output 124 including a decoded sequence of image frames for one or more time steps. For example, the output 124 may include an image frame for each time step of successive decoded image frame time steps.

[0087] There are many types of decoders that can generate an image from a set of latent variables, and any of these decoders may be used. In one example, the decoder 120 is a neural network, which may be a multi-layer perceptron or a convolutional neural network. In addition to the object latent variables and the frame latent variables, the input 122 may further include the region of the image to be generated and the time step during decoding.

[0088] The decoded regions may be, for example, individual pixels. Decoding the sequence of image frames may include, for each pixel of each decoded image frame and each decoded image frame time step, the decoder processing an input including a value of the object latent variable, a value of the frame latent variable, information specifying a location of the pixel, and information specifying the time step. The decoder determines, for the pixel and time step, parameters of a pixel distribution for each of the object latent variables in the input. The decoder may further combine the pixel distributions for each of the object latent variables to determine a joint pixel distribution, and may sample from the joint pixel distribution to determine a value for the pixel and time step.

[0089] For example, each pixel may be modeled as a Gaussian mixture model with K components. If all regions and all frames are assumed to be independent, then for any RGB pixel x t,i The mixture weights for (1≦i≦HW) may capture which object k "explains" that pixel. The corresponding mixture weights

[0090]

number

[0091] and the pixel reconstruction mean μ k,t,i is calculated using the decoder 120 at a particular time step t and at a particular pixel position l i may be calculated for every slot k in

[0092]

number

[0093] In this case, constraining the decoder 120 to operate on individual pixels may advantageously save memory and computation since a subset of pixels is used as a training target rather than the complete image. After the pixels are decoded, the mixture weights are calculated by, for example, taking the softmax of the mask logits over the K slots.

[0094]

number

[0095] can be obtained

[0096]

number

[0097] .

[0098]

number

[0099] Equation (2) specifies the overall pixel likelihood, where σ x is a scalar hyperparameter.

[0100] In generating the input 124 for time t, the object latent variables O are converted to frame latent variables f for a time step during decoding, as shown diagrammatically in FIG. t FIG. 4 shows a schematic diagram of the process of decoding a single image frame for time t. In FIG. 4, the frame latent variable f t is the frame latent variable O1 . . . kThe concatenated latent variables are sampled for each location li to render each pixel, generating a pixel mixture model. The frame latent variables concatenated with the object latent variables may be inferred from the same input sequence from which the object latent variables were inferred. Alternatively, the frame latent variables may be obtained from a different input sequence. In this way, global / view contents (such as lighting and camera pose) from one input sequence may be transferred to a different input sequence.

[0101] In an exemplary implementation, the system 100 may be used to generate novel representations. For example, the system 100 may be trained to generate video image sequences conditioned on conditioning inputs that may define one or more objects or object characteristics for one or more objects in the video sequence 104. For example, the conditioning inputs may define the objects to include in the generated video image sequence 124 and / or may define characteristics of the objects. Alternatively or additionally, the conditioning inputs may define a static or changing viewpoint position or orientation. Preconditions may be learned for objects in a scene. For example, a neural network (e.g., a recurrent neural network) may be used to learn preconditions for objects in a scene and successive frame latencies.

[0102] Additionally, while the exemplary implementations described above determine both object latent variables and frame latent variables, it should be understood that determining and / or outputting both object latent variables and frame latent variables is not required. For example, the encoder 102 and decoder 120 may be directly conditioned on the ground truth viewpoint / camera pose rather than encoding / inferring the frame latent variables. That is, the neural network 100 may be "view-supervised" rather than operating completely unsupervised. A comparison of such a view-supervised approach with other view-supervised models is shown in FIG. 7 and described in more detail below. In a view-supervised system, during decoding, the decoder may be provided with a novel viewpoint (i.e., a viewpoint not provided to the system during training) to generate a view from an unseen viewpoint.

[0103] FIG. 5 is a flow diagram of an exemplary method for training a video object representation neural network system, which may be configured in accordance with the video object representation neural network system 100.

[0104] In a first step 502, a video sequence (such as video sequence 104) is obtained. The video sequence includes a sequence of T image frames, one at each successive time t. In step 504, each image frame in the video sequence is processed (e.g., by feature extraction network 106) to generate a set of S spatial encodings for each of the T image frames, thereby generating a group of S×T spatial encodings for the video sequence.

[0105] In step 506, the group of S×T spatial encodings is processed by the self-attention neural network by applying a self-attention mechanism (e.g., by the self-attention network 108) to layer inputs (e.g., layer input 107) that include the group of S×T spatial encodings to generate an output (e.g., output 109) that includes the group of transformed spatial encodings. Each transformed spatial encoding corresponds to an image frame and an image frame region in the captured video sequence. Applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism to the spatial encodings of different image frames in each of the captured video sequences.

[0106] In step 508, for each object latent variable in the set of object latent variables O, one or more values ​​that parameterize an object latent distribution of the object latent variable are determined by aggregating a group of transformed spatial encodings across time of the image frames.

[0107] If the video object representation neural network system infers frame latent variables, then in optional step 510, for each frame latent variable in the set of frame latent variables F, one or more values ​​that parameterize a frame latent distribution of the frame latent variable by aggregating a group of transformed spatial encodings over a region of the image frame may be determined. Alternatively, as described above, the encoder may instead condition directly on the viewpoint.

[0108] In step 512, a value for each object latent variable in the set of object latent variables is determined by sampling from the object latent distribution for the object latent variable.

[0109] If the video object representation neural network system infers frame latent variables, then in optional step 514, a value for each frame latent variable in the set of frame latent variables may be determined by sampling from a frame latent distribution for the frame latent variable. Alternatively, as described above, the encoder may not infer frame latent variables, in which case a frame latent distribution is not inferred.

[0110] At step 516, a decoded sequence of image frames is generated, i.e., the determined values ​​of the object latent variables (and optionally any determined values ​​of the frame latent variables) are processed (e.g., by decoder 120) to generate a decoded sequence of image frames (e.g., output 124).

[0111] In an implementation, the system is trained by backpropagating the gradient of the objective function through an image frame decoder neural network, a self-attention neural network, and a feature extraction neural network. The backpropagation through the sampling step may involve the so-called re-parameterization trick (Kingma et al., arXiv1312.6114).

[0112] In step 518, the system is trained by adjusting the values ​​of at least the self-attention neural network parameters in response to i) a measure of the difference between the sequence of T image frames and the decoded sequence of image frames, ii) the difference between each of the object latent distributions and the previous object latent distribution, and possibly (i.e., if frame latent variables are generated) iii) the difference between each of the frame latent distributions and the previous frame latent distribution, to optimize an objective function.

[0113] The training may include backpropagation. Specifically, the training may include backpropagating the gradient of the objective function through a decoder neural network (e.g., decoder 120 generating output 124) and a self-attention neural network (e.g., self-attention network 108) that generate a decoded sequence of image frames. The gradient of the objective function may be further backpropagated through a feature extraction neural network (e.g., feature extraction network 106) that generates a set of spatial encodings. However, it will be appreciated that the networks (e.g., feature extraction neural networks, etc.) in the system 100 may be trained separately.

[0114] The objective function (which is minimized) may be a negative lower bound, for example the lower bound of the evidence (ELBO) as defined in equation (3).

[0115]

number

[0116] In the above formula, α, β o , and β f are the respective weights, q represents the encoder 102, p represents the decoder 120, and D KL is a metric of distribution difference such as the Kullback-Leibler divergence.

[0117] That is, the data log-likelihood is calculated for the decoded pixel (T d H d W d ), making it possible to decode fewer than all input pixels. Advantageously, this helps scale the size of the decoder without reducing the training signal due to general correlation between adjacent pixels. d H d W dWe find that normalizing by α makes it easier to make the learning dynamics consistent regardless of the choice of how many pixels to decode. We can set α to 1, but we can also normalize by, say, β o and β f may be adjusted if the scale of is too small to be numerically stable. o =β f is used.

[0118] Training the system in step 518 may also include adjusting values ​​of feature extraction neural network parameters of a feature extraction neural network, such as feature extraction neural network 106. Training the system in step 518 may also include adjusting values ​​of decoder neural network parameters of a decoder neural network, such as decoder neural network 120.

[0119] After training, either the decoder or the encoder part of the system may be used independently. For example, the encoder part of the system may be used to obtain one or more characteristics of one or more objects shown in a video sequence from object latent variables or values ​​parameterizing the object latent distribution, e.g., the mean of the distribution. Characteristics such as object size, color, texture, position in 1D, 2D, or 3D, or motion may be obtained. Similarly, after training, the video object representation neural network system, more specifically, only the encoder part may be used to obtain one or more global characteristics of the frames of a video sequence, e.g., the (changing) viewpoint position or orientation, e.g., pose, for the video sequence, from frame latent variables or values ​​parameterizing the frame latent distribution.

[0120] 6 is a flow chart illustrating an example process for acquiring one or more characteristics of one or more objects shown in a video sequence. At step 602, a video sequence (such as video sequence 104) is acquired. The video sequence includes a sequence of T image frames, one at each successive time t.

[0121] In step 604, each image frame in the video sequence is processed, for example by the trained feature extraction network 106, to generate a set of S spatial encodings for each of the T image frames, thereby generating a group of S×T spatial encodings for the video sequence.

[0122] In step 606, the group of S×T spatial encodings is processed by the trained self-attention neural network by applying a self-attention mechanism (e.g., by self-attention network 108) to layer inputs (e.g., layer input 107) that include the group of S×T spatial encodings to generate outputs (e.g., output 109) that include the group of transformed spatial encodings. Each transformed spatial encoding corresponds to an image frame and an image frame region in the captured video sequence. Applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism to spatial encodings of different image frames of the captured video sequence.

[0123] In step 608, for each object latent variable in the set of object latent variables O, one or more values ​​that parameterize an object latent distribution of the object latent variable are determined by aggregating a group of transformed spatial encodings across the time of the image frames.

[0124] In step 610, a value for each object latent variable in the set of object latent variables is determined by sampling from the object latent distribution for the object latent variable.

[0125] In step 612, one or more characteristics of one or more objects shown in the video sequence are obtained using the determined values ​​for each object latent variable in the set of object latent variables.

[0126] If the video object representation neural network system infers frame latent variables, then in optional step 614, a value for each frame latent variable in the set of frame latent variables may be determined by sampling from a frame latent distribution for the frame latent variable. Alternatively, as described above, the encoder may not infer frame latent variables, in which case a frame latent distribution is not inferred.

[0127] Although FIG. 6 is depicted with respect to determining object latent variables, it will be appreciated that the method may alternatively or additionally include determining frame latent variables to determine one or more global properties of frames of the video sequence.

[0128] In another example, only the decoder portion of the system may be needed, which is the portion of the system from the video sequence input to the portion configured to determine the object (and possibly frame) latent distributions. The encoder portion may possibly include a portion configured to sample from these distributions (in other implementations, information may be derived, for example, from the average of these distributions). In some applications, after training, for example, to generate a video sequence of 2D or 3D images, only the trained decoder portion of the system may be needed. FIG. 7 is a flow diagram illustrating an example process for generating a video sequence using a trained video generation neural network system. In step 702, a value of each object latent variable in a set of object latent variables is determined by sampling from a respective previous object latent distribution for the object latent variable. After step 702, steps 704, 706, and 708 are performed for each pixel of each generated image frame and each generated image frame time step. In step 704, the trained decoder neural network processes the determined value of the object latent variable, the information specifying the location of the pixel, and the information specifying the time step to determine a parameter of the pixel distribution for each of the object latent variables for the pixel and the time step. At step 706, the trained decoder neural network combines the pixel distributions for each of the object latent variables to determine a joint pixel distribution. Processing at step 706 may further include processing the determined values ​​of the object latent variables, the information specifying the location of the pixel, and the information specifying the time step to determine a mixture weight for each of the object latent variables for the pixel and time step, and combining the pixel distributions for each object latent variable weighted by the respective mixture weight to determine a joint pixel distribution. At step 708, the trained decoder neural network samples from the joint pixel distribution to determine a value of the pixel at the time step.

[0129] We experimentally compared an example of a view-supervised video object representation neural network to similar models GQN (Eslami SMA et al., "Neural scene representation and rendering." Science., 15 Jun 2018, 360(6394), pp. 1204-1210, doi:10.1126 / science.aar6170. PMID:29903970) and NeRF (arXiv:2003.08934).

[0130] Figure 8 shows a comparison of scene representation and view interpolation capabilities between NeRF-VAE, a view-supervised version of the video object representation neural network (denoted as SIMONe in Figure 8), and GQN. All models observe a partial “Playroom” dataset, procedurally generated from a given sequence of frames (four of the 16 input frames fed to the model are visualized). Novel views are decoded on a circular trajectory in the room, linearly spaced in yaw in [-π,π]. We can see that NeRF-VAE does not preserve most object structures, and nearly all objects in the scene are blurred. However, NeRF is able to understand the shape of the generated room and infer the color of the walls. GQN produces a more detailed reconstruction, but it overlearns certain views and does not interpolate smoothly. GQN hallucinates the content. SIMONe, on the other hand, produces a sharp reconstruction of fine object structures. SIMONe correctly places and sizes objects in completely novel views, whether they are observed from afar or very close up. This makes SIMONe a strong alternative to NeRF-VAE and GQN-based models when capturing scene structure across diverse examples is a priority. SIMONe also segments the scene.

[0131] It will be appreciated that a video object representation neural network may be provided that processes an input sequence to determine and output only object latent variables or only frame latent variables.

[0132] In another experiment, we compared our unsupervised video object representation neural network (which infers both object and frame latent variables) with other viewpoint-unaware unsupervised methods. Table 1 shows the segmentation performance (in terms of adjusted Rand exponents for foreground objects) of the video object representation neural network described with reference to Figure 1 (denoted as SIMONe in Table 2) compared with MONet (arXiv:1901.11390), Slotted Attention (SA) (arXiv:2006.15055), and a video model (S-IODINE). We calculated static (S)ARI-F and video (S)ARI-F scores separately. For static ARI-F, we evaluated the model for each still image. For video ARI-F, we evaluated the model across space and time, considering the entire trajectory of an object as a single class. Video ARI-F therefore penalizes models that cannot stably track objects. This is especially true for slotted attention. The reason is that slotted attention tends to output objects in an arbitrary order for each frame in the sequence.

[0133] [Table 1]

[0134] Table 2 shows two exemplary possible architectures for the system 100. It will be appreciated that the architectures shown in Table 1 are not intended to be limiting, but are merely provided as examples of possible architectures. In Table 1, c is the number of channels, k is the kernel size, s is the stride, and Pos. MPL is a multi-layer perceptron applied to positional encoding. MLP([m, n]) refers to a two-layer MLP with m and n units in each layer, respectively, and GRU is a gated recurrent unit.

[0135] [Table 2]

[0136] In our experiments, we investigated different ways of using object latent variables and frame latent variables to enable better pixel reconstruction. In our initial experiments, we used sampled object latent variables. Ok and the frame latent variable f t was spatially copied and tiled across all pixels and time points before being provided to the decoder (in our experiments, a convolutional decoder). However, it turns out that it is possible to take independent samples to seed the first stage of the decoder before applying the convolutional decoder stack. In addition, it turns out that it is advantageous to sample independently over time where appropriate. In essence, we sample Ok,i,t ~q( Ok |X), (independently for all slots K and all pixels I = 64 64) f k,i,t We took ~q(ft|X) as the optimal training parameter, which we found improved performance, especially early in the training.

[0137] Without wishing to be bound by theory, the above assumes that the same sample is taken across all space / time in the sequence. Ok and f t This can be attributed to the reduction in gradient bias associated with copying the . Taking multiple independent samples acts as a regularizer.

[0138] The (trained) video object representation neural network system, more specifically, only the encoder portion, may be used to obtain one or more characteristics of one or more objects shown in a video sequence from object latent variables or values ​​parameterizing the object latent distribution, e.g., the mean of the distribution. Characteristics such as object size, color, texture, position in 1D, 2D, or 3D, motion, etc. may be obtained. Similarly, the (trained) video object representation neural network system, more specifically, only the encoder portion, may be used to obtain one or more global characteristics of the frames of a video sequence, e.g., the pose at a particular frame of a video sequence, from frame latent variables or values ​​parameterizing the frame latent distribution.

[0139] The (trained) video object representation neural network system, more specifically, only the encoder portion, may be used for image classification using the determined latent variables. For example, one or more of the determined object latent variables or values ​​parameterizing the object latent distribution may be provided as inputs to one or more classifier neural network layers. There are many image classifiers, and any suitable model may be used. More generally, the video object representation neural network system may be configured to generate any suitable neural network output that characterizes an entity. For example, the neural network output may be a classification output (e.g., defining a score for each category in a set of possible categories), a regression output, a sequence output (i.e., including a sequence of output elements), a segmentation output, or a combination thereof.

[0140] The determined latent variables (of the trained system) can provide input to another system, for example, for use in performing a machine learning task on the network input. Example tasks may include feature-based search, clustering, near-identical scene detection, verification, feature point matching, domain adaptation, video-based weakly supervised learning, and for videos, for example, object tracking across video frames, gesture recognition of gestures performed by an entity shown in the video.

[0141] The (trained) video object representation neural network system, more specifically only the decoder portion, may be used to generate a new video sequence including a decoded sequence of new image frames. This may be achieved by determining a value for each object latent variable in the set of object latent variables by sampling from a previous object latent distribution for the object latent variables, determining a value for each frame latent variable in the set of frame latent variables by sampling from a previous frame latent distribution for the frame latent variables (which may be the same as the previous object latent distribution), and processing the determined values ​​using, specifically, an image frame decoder neural network as described above to generate a decoded sequence of new image frames. In some cases, the image frame decoder neural network may be conditioned on a conditioning input to the image frame decoder neural network as described above, i.e., generate image frames in response to such conditioning input.

[0142] The (trained) encoder portion of the video object representation neural network system may be used to determine values ​​for a set of object latent variables and frame latent variables, and one or more of these values ​​may then be modified and provided to a decoder of the video object representation neural network to generate a modified version of the input video sequence.

[0143] In some implementations, after training, the video object representation neural network system, more specifically only the encoder portion, may be used to control an agent to perform a specific task while interacting with an environment. The agent may be a mechanical agent, such as a robot or an autonomous vehicle, and the environment may be a real-world environment. Such a method of use may include, for example, characterizing a current state of the environment by, for example, acquiring successive image frames of a video sequence for successive time steps, and acquiring observational information including an observational video sequence. The trained video object representation neural network system may then be used to process the observational video sequence to obtain values ​​parameterizing the object latent distribution and values ​​parameterizing the frame latent distribution, and possibly object latent variables and frame latent variables. A control system may then be used to process the above values ​​to generate a control output, whereby the control system may process data from the observational information. The observational information may be provided to the control system in addition to the latent variables or their distributions. The control system may comprise an action selection policy neural network, for example in a reinforcement learning system, and the control output may include an action selection policy output. An action to be performed by the agent may then be selected in response to the observational information. The action may be performed by the agent using the control output, or the control output may be provided for the agent to perform the action.

[0144] This may include providing data such as action data and / or control signals or electronic control data, e.g., motor control data, for the agent to perform an action. That is, the control output may include action data, control signals, etc. For example, the control output may include control signals for controlling the physical behavior of a mechanical agent, e.g., a robot, e.g., torques or higher level control commands for the joints of the robot, or control signals for controlling an autonomous vehicle, e.g., torques or higher level control commands for controlling elements of the vehicle. The action selected may define, for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent.

[0145] The term "configured" is used herein in connection with systems and computer program components. Configuring one or more computer systems to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system that, when operated, causes the system to perform such operation or action. Configuring one or more computer programs to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform such operation or action.

[0146] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information that is transmitted to an appropriate receiver device for execution by the data processing apparatus.

[0147] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or any combination of one or more of these.

[0148] A computer program may be referred to or written as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages. A computer program may be deployed in any form, including a stand-alone program, or a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored in part of a file that holds other programs or data, for example, a markup language document, a single file dedicated to the program in question, or one or more scripts stored in multiple cooperating files, for example, files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer or multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0149] As used herein, the term "database" is used broadly to refer to any collection of data, which may not be structured in any particular way, or at all, and which may be stored in one or more locations on a storage device. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0150] Similarly, the term "engine" is used herein broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed in one or more locations on one or more computers. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer.

[0151] The processes and logic flows described herein may be implemented by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows may also be implemented by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0152] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from such mass storage devices, transfer data to such mass storage devices, or both. However, a computer may not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few examples.

[0153] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and DC ROM disks and DVD-ROM disks.

[0154] To enable interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, for enabling the user to provide input to the computer. Other types of devices can also be used to enable interaction with the user, e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer can interact with the user by sending documents and receiving documents from a device used by the user, e.g., sending a web page to a web browser on the user's device in response to a request received from the web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0155] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the common computationally intensive parts of machine learning training or prediction, i.e., inference, workload.

[0156] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0157] An embodiment of the subject matter described herein may be implemented in a computing system including a back-end component, e.g., as a data server, or in a computing system including a middleware component, e.g., an application server, or in a computing system including a front-end component, e.g., a client computer having a graphical user interface, web browser, or app that allows a user to interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0158] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying the data to a user interacting with the device acting as a client and receiving user input from such a user. Data generated at the user device, e.g., a result of a user interaction, can be received at the server from the device.

[0159] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of the present invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features are described above as acting in some combinations and may even be claimed as such from the outset, in some cases, one or more features of a claimed combination may be deleted from the combination, and the claimed combination may be moved to a subcombination or a variation of the subcombination.

[0160] Similarly, although operations are illustrated in the figures and claimed in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in the order in which they are performed, to achieve desired results, nor should it be understood that all illustrated operations need to be performed. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged as multiple software products.

[0161] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or even sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0162] 100 Video Object Representation Neural Network System 102 Encoder 104 Input, Encoder, Input Sequence, Frame, Video Sequence 106 Feature Extraction Neural Network 107 Output 108 Self-Attention Neural Network 109 Output 110 Latent distribution generator 112 output, sampled latent variables 114 output, sampled latent variables 120 Decoder, Decoder Neural Network 122 Input 124 Outputs, Inputs, and Video Image Sequences T1, T2 self-attention layers

Claims

1. 1. A computer-implemented method for training a video object representation neural network system, the video object representation neural network system including a self-attention neural network comprising one or more self-attention layers, each self-attention layer configured to apply a self-attention mechanism to a layer input, the self-attention neural network having a plurality of self-attention neural network parameters, the method comprising: acquiring a video sequence comprising a sequence of T image frames, one at each successive time; processing each of the image frames to generate a set of S spatial encodings for each of the T image frames, thereby obtaining a group of S×T spatial encodings for the video sequence, each spatial encoding of an image frame including an encoding of a respective region of the image frame; processing the group of S×T spatial encodings using the self-attention neural network by applying the self-attention mechanism to layer inputs comprising the group of S×T spatial encodings to generate a group of transformed spatial encodings, each corresponding to an image frame time and an image frame region, where applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism across the spatial encodings of different image frames; determining, for each object latent variable in a set of object latent variables, one or more values ​​that parameterize an object latent distribution for that object latent variable by aggregating the group of transformed spatial encodings over time of the image frames; determining a value for each object latent variable in the set of object latent variables by sampling from the object latent distribution of the object latent variable; processing the determined values ​​of the object latent variables to generate a decoded sequence of image frames; training the system by adjusting values ​​of at least the self-attention neural network parameters to optimize an objective function in response to: i) a measure of difference between the sequence of T image frames and the decoded sequence of image frames; and ii) a difference between each of the object latent distributions and a previous object latent distribution. A method comprising:

2. determining, for each frame latent variable in a set of frame latent variables, one or more values ​​that parameterize a frame latent distribution for the frame latent variable by aggregating the group of transformed spatial encodings over a region of the image frame; determining a value for each frame latent variable in the set of frame latent variables by sampling from the frame latent distribution for the frame latent variable; Further comprising:

2. The method of claim 1, wherein generating a decoded sequence of image frames further comprises processing the determined values ​​of the frame latent variables to generate the decoded sequence of image frames.

3. 3. The method of claim 2, wherein training the system comprises adjusting values ​​of at least the self-attention neural network parameters to optimize the objective function in response to a difference between each of the frame latent distributions and a previous frame latent distribution.

4. 4. The method of claim 1, wherein each spatial encoding comprises a feature map for the respective region of the image frame, and wherein processing each of the image frames to generate the set of S spatial encodings for each of the T image frames comprises processing each of the image frames using a feature extraction neural network having a plurality of feature extraction neural network parameters to generate the feature map for each region of the image frame, and wherein training the system comprises adjusting values ​​of the feature extraction neural network parameters.

5. processing the determined values ​​of the object latent variables using an image frame decoder neural network having a plurality of image frame decoder neural network parameters to generate a decoded sequence of the image frames; The method of claim 1 , wherein the step of training the system further comprises adjusting values ​​of the image frame decoder neural network parameters.

6. The decoded sequence of image frames includes an image frame for each successive decoded image frame time step, and generating the decoded sequence of image frames using an image frame decoder neural network includes, for each pixel of each decoded image frame and each decoded image frame time step: processing the determined values ​​of the object latent variables, the information specifying the pixel locations, and the information specifying the time steps using the image frame decoder neural network to determine, for the pixels and the time steps, parameters of pixel distributions for each of the object latent variables; combining the pixel distributions for each of the object latent variables to determine a joint pixel distribution; sampling from the combined pixel distribution to determine values ​​for the pixel and the time step; 6. The method of claim 5, comprising:

7. For each pixel of each decoded image frame and each decoded image frame time step: using the image frame decoder neural network to process the determined values ​​of the object latent variables, the information specifying the pixel location, and the information specifying the time step to determine mixture weights for each of the object latent variables for the pixel and the time step. Further comprising:

7. The method of claim 6, wherein combining the pixel distributions for each of the object latent variables to determine a joint pixel distribution comprises combining the pixel distributions for each object latent variable weighted by a respective mixture weight.

8. obtaining conditioning input defining one or more objects, or object properties for one or more objects in the video sequence, or defining a viewpoint position or orientation; providing said conditioning inputs to said self-attention neural network and to said image frame decoder neural network; 8. The method of claim 5, 6, or 7, further comprising:

9. 9. The method of claim 1, wherein the step of processing the group of S×T spatial encodings using the attention neural network comprises processing the layer input including the group of S×T spatial encodings using one of the one or more self-attention layers to generate a query and a key-value pair for each of the spatial encodings, and applying the query to the key-value pairs using the self-attention mechanism to determine the group of transformed spatial encodings, wherein the self-attention neural network parameters comprise parameters of a learned transformation applied to the layer input to generate the query and the key-value pairs.

10. The step of processing the group of S×T spatial encodings using the attention neural network comprises: providing the group of S×T spatial encodings as the layer inputs to a first self-attention layer of the attention neural network to generate a first self-attention layer output; providing the first self-attention layer output to a second self-attention layer of the attention neural network to generate the group of transformed spatial encodings; reducing the number of spatial encodings from S×T spatial encodings to K×T spatial encodings, where K is the number of object latent variables and K is less than S; 10. The method of any one of claims 1 to 9, comprising:

11. The method of claim 10 , wherein the reducing step comprises applying a pooling operation to the first self-attention layer output.

12. 12. The method of claim 1, wherein training the system comprises backpropagating a gradient of the objective function through an image frame decoder neural network that generates a decoded sequence of the image frames, the self-attention neural network, and a feature extraction neural network that generates the set of spatial encodings.

13. 13. The method of claim 1, further comprising: using the trained video object representation neural network system to obtain one or more properties of one or more objects shown in the video sequence from the object latent variables or the values ​​that parameterize the object latent distributions.

14. 14. A method according to claim 2, further comprising the step of obtaining a viewpoint position or orientation for the video sequence from the values ​​parameterizing frame latent variables or frame latent distributions using the trained video object representation neural network system.

15. determining a value for each object latent variable in the set of object latent variables by sampling from a prior object latent distribution for the object latent variable; processing the determined values ​​of the object latent variables to generate a decoded sequence of new image frames. The method of claim 1 , further comprising: generating a new video sequence comprising the decoded sequence of the new image frames using the trained video object representation neural network system by:

16. determining a value for each frame latent variable in the set of frame latent variables by sampling from a previous frame latent distribution for the frame latent variable; processing the determined values ​​of the object latent variables and the determined values ​​of the frame latent variables to generate a decoded sequence of the new image frames; The method of claim 15, dependent on claim 2, further comprising:

17. Obtaining an input video sequence comprising a sequence of input image frames; processing each of the input image frames to obtain a group of spatial encodings for the input video sequence; processing the group of spatial encodings for the input video sequence using the attention neural network to generate a group of transformed spatial encodings; determining values ​​for the set of object latent variables by sampling from object latent distributions determined from the group of transformed spatial encodings; revising the determined values ​​for the set of object latent variables to obtain a revised set of latent variables; processing the modified set of latent variables to generate a decoded sequence of modified image frames; generating a modified video sequence comprising a decoded sequence of the modified image frames using the trained video object representation neural network system by 13. The method of any one of claims 1 to 12, further comprising:

18. determining values ​​for the set of frame latent variables by sampling from frame latent distributions determined from the group of transformed spatial encodings; modifying the determined values ​​for the set of frame latent variables to obtain a modified set of latent variables; The method of claim 17, dependent on claim 2, further comprising:

19. using the trained video object representation neural network system to control an agent to perform a task while interacting with an environment, the method comprising: acquiring observational information characterizing a current state of the environment, the observational information including an observed video sequence; processing the observed video sequence using the trained video object representation neural network system to obtain the values ​​that parameterize the object latent distributions; processing data from the observations, including processing data from the values ​​that parameterize the object latent distributions using a control system, to generate a control output; selecting an action to be performed by said agent in response to said observation using said control output; 11. The method of any one of claims 1 to 10, comprising:

20. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 19.

21. 20. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform each of the operations of the methods of any one of claims 1 to 19.

22. 1. A computer implemented video processing neural network system comprising: A trained feature extraction neural network, receiving a video sequence comprising a sequence of T image frames, one at each successive time; processing each of the image frames to generate a set of S spatial encodings for each of the T image frames, thereby obtaining a group of S×T spatial encodings for the video sequence, each spatial encoding including an encoding of a respective region of the image frame; a trained feature extraction neural network configured to: A trained self-attention neural network including one or more self-attention layers, each self-attention layer configured to apply a self-attention mechanism to a layer input, processing the group of S×T spatial encodings using the attention neural network by applying the self-attention mechanism to layer inputs including the group of S×T spatial encodings to generate a group of transformed spatial encodings, each corresponding to an image frame time and an image frame region, where applying the self-attention mechanism to the layer inputs includes applying the self-attention mechanism across the spatial encodings of different image frames; A self-attention neural network configured to Equipped with The video processing neural network system comprises: for each object latent variable in a set of object latent variables, determining one or more values ​​that parameterize an object latent distribution of the object latent variable by aggregating a group of the transformed spatial encodings over time of the image frames; and determining one or more properties of one or more objects shown in the video sequence from the values ​​that parameterize the object latent distribution.

1. A computer-implemented video processing neural network system configured to:

23. 1. A computer implemented video generation neural network system, comprising: determining a value for each object latent variable in the set of object latent variables by sampling from a respective prior object latent distribution for said object latent variable; It is configured as follows: The system comprises: a trained image frame decoder neural network that processes the determined values ​​of the object latent variables to generate a video sequence comprising a sequence of generated image frames at successive time steps, wherein for each pixel of each generated image frame and each generated image frame time step: processing the determined values ​​of the object latent variables, the information specifying the pixel locations, and the information specifying the time steps to determine, for the pixels and the time steps, parameters of a pixel distribution for each of the object latent variables; combining the pixel distributions for each of the object latent variables to determine a joint pixel distribution; Sampling from the combined pixel distribution to determine values ​​for the pixel and the time step. A trained image frame decoder neural network, configured as follows:

1. A computer-implemented video generation neural network system comprising:

24. The image frame decoder neural network calculates, for each pixel of each decoded image frame and for each decoded image frame time step: processing the determined values ​​of the object latent variables, the information specifying the pixel location, and the information specifying the time step to determine mixture weights for each of the object latent variables for the pixel and the time step; combining the pixel distributions for each object latent variable weighted by the respective mixture weights to determine the combined pixel distribution.

24. The computer-implemented video generation neural network system of claim 23, further configured to:

Images