Animating an image using point locus

Abstract

A method, system, and apparatus for animating an image using point loci, including a computer program encoded on a computer storage medium.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application Nos. 63 / 450,951, filed March 8, 2023, 63 / 452,405, filed March 15, 2023, and 63 / 548,824, filed February 1, 2024. The disclosures of the prior applications are considered part of the disclosure of this application and are incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to using neural networks to process inputs that include images.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current value of its respective parameter set. Summary of the Invention

[0004] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that animates images using a generative neural network system.

[0005] "Animating" an input image refers to generating a video that represents an animation of the input image, e.g., a video that depicts how the scene depicted in the input image changes over time. Examples of changes over time include the movement of objects within the scene, and changes in lighting and other image characteristics.

[0006] This specification also describes techniques for training point tracking neural networks.

[0007] A point tracking neural network is a neural network that processes an input video to generate a network output that includes, for each query point in a set of one or more query points in a given frame of the input video, the predicted spatial location of each of the query points in other video frames in the sequence.

[0008] After training the point tracking neural network, it can be used for any of a variety of purposes.

[0009] As an example, a point tracking neural network may be used to generate training data for training a generative neural network in a generative neural network system that animates images.

[0010] Other uses of point tracking neural networks are described below.

[0011] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0012] Animating still images is generally a difficult problem, both because video modeling is typically extremely computationally expensive and because it is difficult to predict realistic motion from a single still image.

[0013] To address these challenges, the described system decomposes the difficult task of animating a still image into two parts: (i) first generating a set of point loci from the input image, and (ii) then generating a video from the point loci and the input image. That is, the system first generates a set of point loci that are a dense, explicit representation of the surface motion of objects in the scene depicted in the input image, and then generates pixels for frames in the video from the point loci and the input image.

[0014] Thus, the described system can use point loci to direct attention to appropriate locations within the input image when generating pixels for a frame in order to generate a consistent appearance throughout the video. Specifically, the point loci define the motion that should be present in the video, ensuring physical plausibility of the video because the video is generated conditional on the already generated point loci.

[0015] Additionally, by first generating point trajectories, the system decomposes the video generation problem into two computationally efficient steps, ensuring that the video generated by the system shows realistic motion of the objects depicted in the input images.

[0016] Furthermore, animating still images without additional information is an ill-posed problem because there may be many possible realistic future trajectories for any given object depicted in the still image. By utilizing a diffusion neural network to generate point trajectories from still images, the system can effectively sample from a space of realistic surface motion trajectories to ensure that the final video produced represents a realistic sample from a highly multimodal distribution of object motion. Furthermore, because the system uses a diffusion neural network to generate point trajectories, the system can effectively sample multiple plausible sets of point trajectories given the same input image, thereby enabling the system to generate multiple different plausible videos from the same input image, each representing a different sample from the distribution of object motion.

[0017] This specification also describes techniques for training a point-tracking neural network that effectively utilize unlabeled data to improve the training of the neural network through unsupervised learning. As a particular example, a system can use the described unsupervised learning techniques to fine-tune a pre-trained point-tracking neural network that was trained through supervised learning. That is, the system can use the described unsupervised learning techniques to leverage unlabeled data and improve the performance of the pre-trained neural network. For example, the neural network may have been pre-trained on a dataset that includes synthetic video sequences, e.g., only or primarily synthetic video sequences, and the system may then further train ("fine-tune") the neural network through unsupervised learning on fine-tuning data that includes unlabeled real-world video sequences. This can improve the point-tracking neural network's ability to generalize to diverse real-world videos, i.e., its ability to perform point-tracking tasks that require processing real-world video sequences, even when labeled real-world data is unavailable or limited. Specifically, while ground truth pixel-level trajectories can be easily generated when generating synthetic videos, obtaining accurate pixel-level trajectory labels for real-world videos can be difficult or impossible. The described techniques enable the system to incorporate unlabeled real-world videos to improve the performance of a trained point tracking neural network.

[0018] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0019] [Figure 1A] Here is an example that creates a video that animates an input image. [Figure 1B] FIG. 1 is a diagram of an exemplary image animation system. [Figure 2] FIG. 1 is a flow diagram of an example process for generating a set of point loci from an input image. [Figure 3] FIG. 1 is a flow diagram of an example process for generating a video from an input image and a set of point loci. [Figure 4] 1 illustrates an exemplary architecture of a point tracking neural network. [Figure 5] An example of training a point tracking neural network is given. DETAILED DESCRIPTION OF THE INVENTION

[0020] Like reference numbers and designations in the various drawings refer to like elements.

[0021] FIG. 1A shows an example of generating a video 102 that animates an input image 104 .

[0022] As shown in the example of FIG. 1A, an image animation system implemented as a computer program on one or more computers at one or more locations receives an input image 104 .

[0023] For example, the system can receive an input image 104 from a user of the system.

[0024] The system then "animates" the input images 104 by generating a video 102 that represents an animation of the input images 104 that depicts how the scene depicted in the input images 104 changes over time.

[0025] That is, even if the input image 104 is a "still" image at a single point in time, the system generates a video 102 that includes respective video frames at each of multiple time steps (starting with the input image 104 as the first frame at the first time step in the video) and represents a realistic estimate of how the scene depicted in the input image 104 changes over time.

[0026] Specifically, the system breaks down the task of animating a video into two steps.

[0027] First, the system processes an input image 104 to generate a set of point loci 106 .

[0028] Each point locus 106 corresponds to a different point in the input image 104 and, for each time step in the video, includes the predicted spatial location (generated by the system) of the corresponding point in the video frame at that time step in the video.

[0029] Each point identifies a point in a corresponding one of the video frames, i.e., a respective spatial location, i.e., a respective pixel, in a corresponding one of the video frames. Thus, each point in a given trajectory 106 can be represented as a point (x, y, t), where x, y are the spatial coordinates of the point and t is the index of the corresponding video frame in the video 102.

[0030] 1A as dotted curves, with points at future times represented as points on the curve, with points closer to the tip of the curve being further away in time from the input image. While only three point loci are shown in FIG. 1 for ease of illustration, in practice the system may generate many more point loci so that the loci represent a dense representation of the future motion of points in the input image 104. For example, the system may generate a respective point locus for each grid cell in a grid, e.g., an 8-pixel by 8-pixel grid overlaid on the input image.

[0031] Optionally, the point trajectory 106 may also include a respective occlusion score for each of the frames, which represents the likelihood that the corresponding point is occluded in the video frame at the time step.

[0032] For example, the system can use a generative neural network to generate point loci 106 from the input image 104 .

[0033] This is explained in more detail below.

[0034] The system then processes the input images 104 and point loci 106 to generate the video 102 , ie, generates frames in the video 102 from the input images 104 and point loci 106 .

[0035] For example, the system can use other generative neural networks to generate the video 102 from the input images 104 and point loci 106 .

[0036] 1A, in the example of FIG. 1A, the video 102 may show the person raising their right arm in a future frame because point locus 106 indicates that a point on the depicted person's right arm is likely to move away from their body. Similarly, the video may show the person kicking their left leg in a future frame because point locus 106 indicates that a point on the depicted person's left leg is likely to move up and away from their body.

[0037] Thus, the system decomposes the difficult task of animating a still image by first generating a dense explicit representation of the surface motion of objects in the scene depicted in the input 104, i.e., point loci 106, and then generating the pixels of frames in a video from the point loci 106 and the input images 104.

[0038] Thus, the system can use the locus 106 to direct attention to appropriate locations in the input image 104 when generating pixels for a frame in order to produce a consistent appearance throughout the video. Specifically, the point locus 106 defines the motion that should be present in the video 102, ensuring physical plausibility.

[0039] 1B is a diagram of an exemplary image animation system 100. Image animation system 100 is an example of a system in which the systems, components, and techniques described below may be implemented, implemented as a computer program on one or more computers at one or more locations.

[0040] As described above, the system 100 generates a video 102 that animates the input images 104, i.e., a video 102 that represents an animation of the input images 104 that depicts how the scene depicted in the input images 104 changes over time.

[0041] Specifically, the system 100 receives an input image 104. For example, the system may receive the input image 104 as input from a user.

[0042] The system 100 processes a first input derived from the input image 104 using a first generative neural network 110 to generate a respective point locus 106 for each of one or more points in the input image 104. The first generative neural network 110 is also referred to herein as a "trajectory model."

[0043] For example, the points can be pixels randomly sampled from the input image 104, can be pixels on a grid overlaid on the input image 104, or can be points specified by the user who provided the input image 104 (or another user).

[0044] Each point locus 106 includes, for each of a plurality of time steps in the video 102, a predicted spatial location (generated by the system 100) of a corresponding point in a video frame at that time step in the video 102.

[0045] Optionally, point locus 106 may also include a respective occlusion score for each of the frames, which estimates the likelihood that the corresponding point will be occluded in the video frame at the time step.

[0046] For example, the first generating neural network 110 can be a diffusion neural network that generates each point trajectory from a corresponding noisy trajectory conditioned on a first input. This diffusion neural network is referred to as a “trajectory” diffusion neural network. As used herein, “noisy” refers to values ​​sampled from a “noisy distribution,” e.g., a Gaussian distribution or other suitable distribution. For a given point trajectory, the noisy trajectory thus includes, for each spatial location within the given point trajectory, a corresponding spatial location sampled from the corresponding noise distribution.

[0047] The trajectory diffusion neural network may generally have any suitable neural network architecture that enables the trajectory diffusion neural network to map inputs that include noisy trajectories to denoising outputs that define updates to the noisy trajectories.

[0048] As an example, the trajectory spreading neural network may be a two-dimensional convolutional neural network, such as a U-Net or other neural network with a convolutional architecture. Optionally, the trajectory spreading neural network may include one or more self-attention layers as part of the convolutional architecture.

[0049] Generating a set of trajectories using a trajectory diffusion neural network is described in more detail below with reference to FIG.

[0050] The system 100 then generates each of the video frames in the video 102 using a second generative neural network 120 and based on the input image 104 and the one or more point loci 106. The second generative neural network 120 is also referred to herein as a "pixel model."

[0051] For example, the second generative neural network 120 can be a diffusion neural network that generates each video frame in the video (i.e., generates one or more intensity values ​​for each of the pixels in the video frame) from a corresponding noisy video frame conditioned on the input image 104 and the trajectory 106. This diffusion neural network is referred to as a "pixel" diffusion neural network.

[0052] The pixel diffusion neural network may generally have any suitable neural network architecture that enables the pixel diffusion neural network to map an input to an output image.

[0053] As an example, the pixel diffusion neural network may be a convolutional neural network, such as a U-Net or other neural network with a convolutional architecture. Optionally, the pixel diffusion neural network may include one or more self-attention layers as part of the convolutional architecture.

[0054] Generating the video 102 is described in more detail below with reference to FIG.

[0055] Thus, the system 100 uses two different generative neural networks: one that generates the point trajectories 106, and one that generates the video 102 given the point trajectories 106. The output from the trajectory model therefore dictates the motion that should be present in the video 102 generated by the pixel model, ensuring physical plausibility of the generated video 102.

[0056] Once the video 102 is generated, the system 100 may use the video for any of a variety of purposes. For example, the system 100 may store the video 102 or provide the video 102 for presentation to a user, such as the user who submitted the input image 104.

[0057] Before using the first and second generating neural networks 110 and 120 to generate video, the system 100 or other training system trains the neural networks 110 and 120 .

[0058] For example, the training system may train the first generative neural network 110 with a training data set that includes: (i) a plurality of video sequences, where the video sequences are sequences of video frames, and (ii) for each of the video sequences, a respective point trajectory for each of one or more points in a first frame in the video sequence.

[0059] That is, the training system may generate training data from a set of video sequences for training the first generative neural network 110. For example, the system may generate training examples, each corresponding to one of the video sequences, each including a first frame in the corresponding video sequence as a training input image and a respective point locus for a point in the first frame as a target output.

[0060] The training system may then train the first generating neural network 110 on the training examples using an objective function appropriate for the type of neural network being used. For example, if the first generating neural network 110 is a diffusion neural network, the objective function may be a score-matching objective function.

[0061] As another example, the training system may train a second generative neural network 120 with the same training data set.

[0062] That is, the training system may generate training data from a set of video sequences for training the second generative neural network 120. For example, the system may generate training examples, each corresponding to one of the video sequences, each including a first frame in the corresponding video sequence as a training input image, respective point loci for points in the first frame in the corresponding video sequence as a set of target point loci, and the corresponding video sequence as a target output.

[0063] The training system may then train the second generating neural network 120 on the training examples using an objective function appropriate for the type of neural network being used. For example, if the second generating neural network 120 is a diffusion neural network, the objective function may be a score-matching objective function.

[0064] In some cases, the training system may generate point trajectories for at least some of the video sequences by processing the video sequences using a point tracking neural network.

[0065] A point tracking neural network is a neural network that processes an input video to generate a network output that includes, for each query point in a set of one or more query points in a given frame of the input video, a predicted spatial location of each of the query points in other video frames in the sequence, and optionally an occlusion estimate for the query point.

[0066] That is, because a large amount of densely labeled video may not be available for use in training the generative neural network, the training system may use a point tracking neural network to predict point trajectories for at least a portion of the video sequence, and then use the point tracking neural network's predictions to train the first and second generative neural networks 110 and 120.

[0067] The point tracking neural network may generally have any suitable architecture and may be trained using any suitable technique.

[0068] Some examples of point tracking neural network architectures are described below with reference to FIG.

[0069] Another example of a point tracking neural network architecture is described in Doersch, et al, TAP-Vid: A Benchmark for Tracking Any Point in a Video, arXiv:2211.03726.

[0070] An example of training a point tracking neural network is described below with reference to FIG.

[0071] Other examples of training point-tracking neural networks are described in Doersch, et al., TAP-Vid: A Benchmark for Tracking Any Point in a Video, arXiv:2211.03726, and Doersch, et al., TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement, arXiv:2306.08637.

[0072] 2 is a flow diagram of an exemplary process 200 for generating a set of point loci. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an image animation system, such as image animation system 100 depicted in FIG. 1B, can be suitably programmed in accordance with this specification to perform process 200.

[0073] The system receives an input image (step 202).

[0074] The system processes the input image using an image encoder neural network to generate an encoded representation of the input image (step 204).

[0075] Generally, the encoded representation includes a respective feature vector for each of multiple spatial regions in the input image. For example, if the input image is an H×W image, the encoded representation may be an H / k×W / k map of feature vectors. For example, k may be equal to 4, 8, or 16.

[0076] The image encoder neural network may generally have any suitable neural network architecture for encoding an input image, for example, the image encoder neural network may be a convolutional neural network or a vision transformer neural network.

[0077] In some cases, the system uses a pre-trained image encoder neural network that has already been trained to generate representations of images in the representation learning task. In some other cases, the system jointly trains the image encoder neural network with the first generator neural network.

[0078] For each point locus in the set, the system generates a corresponding noisy locus (step 206).

[0079] Specifically, for a given point locus, the noisy locus includes a corresponding value for each value in the given point locus. To generate the noisy locus, the system samples each of these values ​​from a corresponding "noise distribution," e.g., a Gaussian or other suitable distribution.

[0080] Thus, as described above, each point trajectory includes, for each time step in the video, (i) the predicted spatial location of the corresponding point in the video frame at the time step, and optionally (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded in the video frame at the time step.

[0081] The predicted spatial location within the video frame may be an absolute spatial location, i.e., a location expressed as absolute coordinates in the image coordinate system, or a relative spatial location, i.e., a location expressed as coordinates relative to a corresponding point in the input image.

[0082] Therefore, the system samples the predicted spatial location and, if included, the occlusion score for each time step from the respective noise distribution, so that the trajectory contains noisy coordinates, i.e., noisy predicted spatial location, and noisy occlusion estimates for each time step.

[0083] Optionally, the noisy trajectory may also contain additional values ​​that assist the first generating neural network in effectively utilizing the information contained in the noisy trajectory.

[0084] For example, the noisy coordinates can be extended with position encoding, i.e., the noisy trajectory may include, for each set of noisy coordinates, a position encoding of the noisy coordinates. As a particular example, the position encoding may be Fourier position encoding, which encodes the noisy coordinates using a fixed number of Fourier features.

[0085] As another example, the noisy trajectory may also include "relative" noisy coordinates in addition to "absolute" noisy coordinates, for example, coordinates of the noisy predicted spatial position in a coordinate system centered on the corresponding point in the input image.

[0086] Next, the system generates each point locus in the set from the corresponding noisy locus and the coded representation of the input image using a first generative neural network (step 208).

[0087] Specifically, in the example of FIG. 2, the first generating neural network is a spreading neural network (trajectory spreading neural network).

[0088] To generate the point trajectories, the system performs a series of de-diffusion iterations using a trajectory diffusion neural network.

[0089] At each despreading iteration, the system processes the input of the despreading iteration, including the noisy point trajectory at the time of the despreading iteration, using a trajectory diffusion neural network and conditioned on the coded representation of the input image, to generate a denoised output that defines an update to the noisy point trajectory.

[0090] For example, the denoising output may be, for each noisy point locus, a prediction of the corresponding actual (unknown) point locus. As another example, the denoising output may be, for each noisy point locus, a prediction of the noise that was added to the corresponding actual point locus to generate the noisy point locus. As mentioned above, when the input includes both absolute and relative coordinates, in some implementations the denoising output predicts the relative coordinates, while in some other implementations the denoising output predicts the absolute coordinates.

[0091] The system may condition the trajectory spreading neural network on the coded representation of the input image in any of a variety of ways.

[0092] For example, the system may include the coded representation as part of the input for the iteration, eg, concatenated with the noisy point trajectory.

[0093] As another example, a trajectory diffusion neural network may include one or more conditioning layers that receive as input an encoded representation of an input image.

[0094] As an example, each conditioning layer may be a cross-attention layer that cross-attends to a coded representation of the input image.

[0095] As another example, each conditioning layer may be a conditioned group normalization layer. That is, after the group normalization layer performs mean subtraction and variance normalization within each group to produce a normalized output Z, the group normalization layer typically applies scale and shift operations. To create a "conditioned" group normalization layer, these are replaced by linear projections of the conditioned inputs. For example, the system can resize the encoded representation so that its spatial dimensions are the same size as Z, and then apply respective learned transforms, e.g., two 1x1 convolutional layers, to create scales and multipliers the same size as Z. These scales and multipliers can be applied in place of the scale and shift operations.

[0096] For each de-spreading iteration, the system then uses the de-noising output to update the noisy trajectory. For example, the system may generate an estimate of the updated noisy trajectory from the de-noising output and then apply a diffusion sampler, such as a Denoising Diffusion Probabilistic Model (DDPM) sampler, a Denoising Diffusion Implicit Model (DDIM) sampler, or other suitable sampler, to the de-noising output to generate the updated noisy trajectory. If the de-noising output is a prediction of the corresponding actual (unknown) point trajectory for each noisy point trajectory, the system may use the de-noising output directly as the estimate. If the de-noising output is a prediction of the noise added to the corresponding actual point trajectory to generate the noisy point trajectory for each noisy point trajectory, the system may determine the estimate from the current noisy point trajectory, the de-noising output, and the noise level of the current de-spreading iteration. Optionally, after the final de-spreading iteration, the system may refrain from using the diffusion sampler and instead use the estimate as the updated noisy trajectory.

[0097] The system then uses the updated noisy trajectory after the last despreading iteration to generate the final point trajectory.

[0098] If the point trajectory includes an occlusion estimate, the system may apply a smoothing operation to make the occlusion continuous during the de-diffusion iteration because the diffusion neural network operates in a continuous space. The system may then back-calculate a binary occlusion estimate from the noisy occlusion estimate after the final de-diffusion iteration.

[0099] for example,

number

number

number

number

number

number

number

number

[0100] Thus, the system iteratively "denoises" the noisy trajectory to generate the final point trajectory.

[0101] As mentioned above, the trajectory spreading neural network may generally have any suitable neural network architecture that enables the trajectory spreading neural network to map inputs that include noisy trajectories to denoising outputs that define updates to the noisy trajectories.

[0102] As an example, the trajectory spreading neural network may be a two-dimensional convolutional neural network, such as a U-Net or other neural network with a convolutional architecture. Optionally, the trajectory spreading neural network may include one or more self-attention layers as part of the convolutional architecture.

[0103] 3 is a flow diagram of an exemplary process 300 for generating a video from an input image and a set of point loci. For convenience, process 300 will be described as being performed by one or more computer systems located at one or more locations. For example, an image animation system, such as image animation system 100 depicted in FIG. 1B, can be suitably programmed in accordance with this specification to perform process 300.

[0104] The system receives an input image and a set of point loci (step 302).

[0105] For each video frame in the video, the system generates a corresponding noisy video frame (step 304).

[0106] Specifically, for a given video frame, the noisy video frame includes intensity values ​​corresponding to each intensity value in the given video frame. To generate the noisy video frame, the system samples each of these values ​​from a corresponding "noise distribution," e.g., a Gaussian distribution or other suitable distribution.

[0107] Next, the system generates each video frame from the corresponding noisy video frame, the input image, and the point locus using a second generative neural network (step 306).

[0108] Specifically, in the example of FIG. 3, the second generating neural network is a diffusion neural network (pixel diffusion neural network).

[0109] To generate the video frames, the system uses a pixel diffusion neural network to perform a series of despreading iterations, i.e., in each despreading iteration, for each video frame at each time step, the system updates a corresponding noisy video frame.

[0110] At each despreading iteration, and for each video frame, the system processes the input of the despreading iteration, including the corresponding noisy video frame at the time of the despreading iteration, using a pixel diffusion neural network and conditioned on the input image and point trajectory, to generate a denoised output that defines an update to the noisy video frame.

[0111] For example, the denoising output may be a prediction of the corresponding actual (unknown) video frame.

[0112] As another example, the denoising output may be a prediction of the noise that was added to the corresponding actual video frame to generate the noisy video frame.

[0113] The system may condition the pixel spreading neural network on the input image and point trajectories in any of a variety of ways.

[0114] As an example, the system may include, as input to the diffusion neural network for a given video frame at a given time, a version of the input image warped according to one or more point loci to represent the image, i.e., the system may generate a warped version of the input image warped according to one or more point loci, i.e., a version that is a representation of how the input image would look at that time if points were to move according to the point loci.

[0115] The system may generate the warped version of the input image in any of a variety of ways.

[0116] As an example, the system may use patch-based warping. For a given frame t, the trajectory at time t specifies where each patch in the input image should appear. The system may then construct a new image in which each local patch is placed in its correct position, using (for example) bilinear interpolation to obtain sub-pixel accuracy. However, this may result in gaps between patches in certain situations. In some implementations, to account for this, the system may actually warp a larger patch around each point. When multiple patches appear that cover the same output pixel, the system may weight them inversely proportional to their distance from the track center.

[0117] Optionally, to address the fact that aliasing can occur when multiple patches overlap, the system may perform warping by warping each patch multiple times, with the difference between the warps being how the system calculates the blending weights. The system may then include all of the warped versions of the input image in the input to the diffusion neural network. For example, p i,j,t Let be the position of the trajectory starting from point i,j in the original image at time t according to the point trajectory for point i,j. In the original warping,

number

number

number

number

[0118] The system may then include this warped version of the input image along with a noisy version of the video frame, for example, by concatenating the two images.

[0119] As another example, the system may condition a diffusion neural network on warped versions of features extracted from the input image, i.e., on features of the input image warped according to one or more point loci to represent the image.

[0120] For example, the features can be feature vectors in an encoded representation of the input image, which can be, for example, the same encoded representation described above with reference to the steps, or a different encoded representation generated by a separate, separately trained image encoder neural network.

[0121] To warp these features for time step t, the system can use the position of each feature at time step t according to a set of point trajectories. The system can then use (for example) bilinear interpolation to place the feature at the appropriate location within the grid of "warped" features. Optionally, the system can track the number of features placed within any particular grid cell (specifically, the sum of the bilinear interpolation weights) and normalize by this sum; if the sum is less than 0.5, the system can divide by 0.5.

[0122] The system may condition the diffusion neural network on the warped version of the features in any suitable manner, for example, using one of the conditioning techniques described above with reference to FIG. 2.

[0123] In some implementations, for at least a subset of frames, the system also includes temporal context from other frames in the video in the input to the frame's diffusion neural network. For example, for a given frame, the input at a given despreading iteration may include a current (noisy) version of one or more previous frames in the video at the time of the despreading iteration. Alternatively or additionally, for a given frame, the input at a given despreading iteration may include a current (noisy) version of one or more subsequent frames in the video at the time of the despreading iteration.

[0124] For each despreading iteration, the system then uses the denoising output of the video frame to update the corresponding noisy video frame for each video frame. For example, the system may generate an estimate of the updated noisy video frame from the denoising output and then apply a diffusion sampler, such as a Denoising Diffusion Probabilistic Model (DDPM) sampler, a Denoising Diffusion Implicit Model (DDIM) sampler, or other suitable sampler, to the estimate to generate the updated noisy video frame. If the denoising output is a prediction of the corresponding actual (unknown) frame, the system may use the denoising output directly as the estimate. If the denoising output is a prediction of noise added to the corresponding actual frame to generate the noisy frame, the system may determine the estimate from the current noisy frame, the denoising output, and the noise level of the current despreading iteration. Optionally, after the last despreading iteration, the system may refrain from using the diffusion sampler and instead use the estimate as the updated noisy frame.

[0125] The system then uses the updated noisy video frame after the last despreading iteration to generate a video frame in the video.

[0126] FIG. 4 shows exemplary architectures 410, 420, and 430 of point tracking neural networks.

[0127] A point tracking neural network is a neural network that processes inputs that include (i) a video that includes multiple video frames, and (ii) a set of one or more query points.

[0128] Each query point identifies a point in a corresponding one of the video frames, ie a respective spatial location, ie a respective pixel, in a corresponding one of the multiple video frames.

[0129] The point tracking neural network 400 processes a set of one or more query points and a video sequence, i.e., intensity values ​​of pixels of video frames within the video sequence, to generate a network output that includes, for each query point, a predicted spatial location of each of the query points in other video frames within the sequence, i.e., video frames other than the query point's corresponding video frame.

[0130] Depending on the architecture of the neural network or the design of the processing pipeline, the network output may also include the predicted spatial location of the corresponding video frame, although in some of these cases the system may ignore the predicted spatial location of the corresponding video frame (because the actual location in the corresponding video frame is provided as input to the system).

[0131] That is, given a query point in one of the video frames, the point tracking neural network may generate predictions of the spatial location of the query point in other video frames within the video.

[0132] The predicted location of a given query point in a given other video frame is a prediction of the location of the portion of the scene depicted at the given query point in the corresponding video frame. For example, at a given query point, if the corresponding video frame depicts a particular point on the surface of an object in the scene, then the predicted location of the given query point identifies the predicted location of the same particular point on the surface of the object in the given other video frame.

[0133] In some implementations, the point tracking neural network also generates, for each query point, a respective occlusion score for the query point relative to each of the other video frames in the sequence. The occlusion score for a given query point in a given video frame represents the likelihood that the query point is occluded in the given video frame, i.e., the likelihood that the portion of the scene depicted at the query point in the corresponding video frame is occluded in the given video frame.

[0134] Specifically, in the exemplary architecture 410, a point tracking neural network is configured to generate query features for a given video sequence and corresponding points in a given frame within the video sequence.

[0135] The neural network is then configured to use the query features to generate a cost volume comprising a respective cost map for each of a plurality of frames in the video sequence.

[0136] For example, a neural network may process a sequence of w × h video frames, and for video frame t q Query points in (i q ,j q ,t q ) can be received.

[0137] The neural network processes video frames in the video sequence using a visual backbone neural network to generate a feature grid including a respective visual feature, i.e., feature vector, for each of a plurality of spatial locations in each of the video frames. Typically, each of the spatial locations corresponds to a different region of the video frame. For example, the feature grid can be a w / 8×h / 8 grid of d-dimensional visual features, where each visual feature corresponds to a grid of 8×8 pixels from the corresponding video frame.

[0138] The visual backbone neural network may have any suitable architecture that enables the neural network to map a video sequence to a feature grid. In the example of Figure 3, the visual backbone neural network is a 3D convolutional neural network (ConvNet), such as TSM-ResNet-18 or other suitable convolutional neural network. In other examples, the visual backbone neural network may be a different type of neural network, such as a vision transformer neural network.

[0139] The neural network may then generate extracted features for the query point from the spatial location of the query point in the corresponding video frame and the respective visual features for one or more of the multiple spatial locations in the corresponding video frame.

[0140] For example, the neural network can compute the corresponding video frame t q For each spatial location of the query in (i q ,j q ) can be generated by performing an interpolation, e.g., a bilinear interpolation, of the visual features of a set of spatial locations within a local neighborhood of

[0141] The neural network then generates a cost volume from the feature grid and the extracted features of the query points. For example, the cost volume can have a respective cost value for each spatial location in each of the video frames. That is, the cost volume includes an h' x w' x 1 grid of cost values ​​for each video frame in the sequence.

[0142] To calculate a cost value for a given spatial location in a given video frame, the system calculates the dot product between the extracted features and the visual features for the given spatial location in the given video frame.

[0143] The neural network is then configured to generate, for each of the plurality of frames, an initial location of a corresponding point in the frame using the cost map for the frame, and an initial occlusion estimate for the corresponding point in the frame.

[0144] In general, to generate the predicted positions, the neural network may process the cost volume for each video frame using a decoder neural network to generate a respective score for each spatial position within the video frame.

[0145] For each of a plurality of video frames other than the corresponding video frame, the neural network may then generate an initial predicted location from the respective scores for the spatial location within the video frame.

[0146] If the neural network also predicts occlusion, as in the example of FIG. 4, the neural network may process the cost volume using a decoder neural network to generate, for each video frame other than the corresponding video frame, a predicted position of each query point within the video frame and an occlusion score for each query point within the video frame.

[0147] For example, the neural network can perform this processing independently for each of the video frames, i.e., for a given video frame, the neural network processes an h′×w′×1 portion of the cost volume for the given video frame using the decoder neural network to generate a predicted location of the query point within the given video frame and an occlusion score for the query point within the video frame.

[0148] For example, the decoder neural network may include a set of shared layers and respective branches for position inference and occlusion inference.

[0149] The decoder neural network processes a portion of the cost volume for a given video frame using a shared layer, e.g., a convolutional layer followed by a rectified linear unit (ReLU) activation function layer, to generate a shared output.

[0150] For the occlusion inference branch, the decoder neural network processes the shared output using layers in the occlusion inference branch to generate a single occlusion score ("logit") for a given video frame. For example, the occlusion inference branch includes a first set of layers that collapses the shared output into a vector (e.g., using spatial average pooling), followed by a second set of layers that regress occlusion logits for a given video frame from the single vector. For example, the second set of layers may include a linear layer, a Leaky ReLU, and another linear layer that generates a single logit. A Leaky ReLU may be a ReLU activation function that has a gradient for negative values ​​(e.g., a relatively small gradient, such as a gradient of 0.01) (as opposed to a flat gradient for negative values).

[0151] For the location inference branch, the neural network may apply a set of layers, e.g., a convolutional (Conv) layer with a single output followed by a spatial softmax, to generate a respective score for each spatial location in the video frame.

[0152] The neural network may then compute a "soft argmax" to calculate a position from each score.

[0153] To calculate the soft argmax, the neural network may identify the spatial location with the highest score, i.e., the "argmax" location, according to the scores of each of the spatial locations. Then, the neural network may identify each spatial location within a fixed-size window B of the spatial location with the highest score, i.e., the argmax location, and then determine a predicted location by calculating a weighted average.

[0154] That is, the neural network determines the predicted location by computing a weighted sum of spatial locations within a fixed-size window of argmax spatial locations, with the weight for each spatial location being calculated based on, e.g., equal to or directly proportional to, the ratio between the spatial location's score and the sum of the scores of the spatial locations within the fixed-size window.

[0155] In the exemplary architecture 410, the neural network uses the initial position and the initial occlusion estimate as the final output of the neural network.

[0156] In this example, the training system may train a neural network with the following loss function for a given query point for each frame t in a given training video:

number

number

number

number

number

number

number

[0157] In the example architectures 420 and 430, once the initial estimates are generated, the neural network 400 uses the initial point trajectory, which includes the initial positions and initial occlusion estimates for multiple frames, to generate point trajectories for the corresponding points by refining the initial positions, refining the initial occlusion estimates for multiple frames, or both.

[0158] Specifically, in the example of FIG. 4, the neural network refines the initial position and initial occlusion estimates for multiple frames by performing one or more refinement iterations.

[0159] In each refinement iteration, the neural network 400 generates, for each frame from the current point trajectory at the time of the refinement iteration, a set of local score maps that capture the similarity between the query features and features in the neighborhood of the predicted location of the current point trajectory in the frame. For the first refinement iteration, the current point trajectory is the initial point trajectory. For subsequent refinement iterations, the current point trajectory is the updated point trajectory after the previous refinement iteration.

[0160] The neural network then processes inputs including the set of local score maps using a refinement neural network to generate updates to the current point trajectory. As a particular example, the inputs to the refinement neural network may include the set of local score maps, the query features, and the current point trajectory.

[0161] For example, in the exemplary architecture 430, the refinement neural network is a depthwise mixing neural network that propagates information between frames using depthwise convolutional layers, each of which may have, for example, a temporal receptive field that extends across multiple (e.g., all or a subset) frames.

[0162] In this example, the training system can train a neural network with a loss that is the sum of the above losses calculated for the initial prediction and the prediction after all refinement iterations.

[0163] In some implementations, the neural network may also generate an uncertainty estimate for each predicted spatial location that represents the neural network's uncertainty in the prediction. For example, the neural network may output additional logits in the occlusion path, generating the uncertainty estimate u t When a neural network utilizes refinement, it also refines the uncertainty estimates at each refinement iteration.

[0164] In this example, the loss above is an additional term

number

number

number

[0165] Next, a specific example of architecture 430 will be described.

[0166] Given the estimated position, occlusion, and uncertainty for each frame, the goal of each iteration of the refinement procedure is to adjust the estimate to be closer to the ground truth.

number

[0167] For example, these can be computed with pyramids of different resolutions, so that for a given trajectory, they have the shape (H' x W' x L), where H' = W' = 7 is the size of the local neighborhood and L is the number of levels in the spatial pyramid. For example, different pyramid levels can be computed by spatially pooling a feature volume F containing features for spatial locations within frames of a video. This set of similarities is post-processed in a refinement network to predict refined position, occlusion, and uncertainty estimates.

[0168] However, unlike initialization, the neural network includes "local score maps" for multiple frames simultaneously as input to post-processing. As a specific example, the neural network can include the current position estimate, raw query features, and the (flattened) local score maps in a tensor of shape T×(C+K+4), where C is the number of channels in the query features, K=H′·W′·L is the number of values ​​in the flattened local score maps, and 4 is an additional dimension for position, occlusion, and uncertainty.

[0169] The output of this refinement network at the ith iteration is the residual

number

[0170] ΔF q,t,i is of shape T×C, and therefore after the first iteration, slightly different "query features" are used at each frame when computing the new local score maps.

[0171] These position and score maps are updated

number

[0172] FIG. 5 shows an example 500 of training a point-tracking neural network, for example, a point-tracking neural network having one of architectures 410, 420, or 430, or a point-tracking neural network having a different network architecture.

[0173] In the example of Figure 5, the system "fine-tunes" a "pre-trained" point-tracking neural network through unsupervised learning.

[0174] As a particular example, the system may fine-tune a pre-trained point tracking neural network that includes synthetic video sequences, e.g., that was trained through supervised learning on a dataset that includes only or primarily synthetic video sequences, through unsupervised learning on fine-tuning data that includes unlabeled real-world video sequences, thereby improving the point tracking neural network's ability to generalize to a variety of real-world videos, i.e., to perform point tracking tasks that require processing real-world video sequences.

[0175] Specifically, the system receives data specifying a trained point tracking neural network 510 .

[0176] The teacher neural network 510 is configured to receive a teacher point tracking input including (i) a video sequence including a plurality of video frames and (ii) a query point within one of the plurality of video frames, and to generate, for each other video frame of the plurality of video frames, a teacher point tracking output including a location of the query point within the other video frame and an occlusion estimate for the query point within the other video frame.

[0177] Next, the system trains the student point tracking neural network 520 through unsupervised learning.

[0178] Similar to the teacher neural network 510, the student point tracking neural network 520 is configured to receive a student point tracking input including (i) a video sequence including a plurality of video frames and (ii) a query point within one of the plurality of video frames, and to generate, for each other video frame of the plurality of video frames, a student point tracking output including a location of the query point within the other video frame and an occlusion estimate for the query point within the other video frame.

[0179] Specifically, prior to training, the supervised point tracking neural network 510 is pre-trained. In the example of Figure 5, the supervised point tracking neural network is pre-trained on, for example, a dataset of synthetic video sequences through supervised learning to yield pre-trained network parameters 502.

[0180] For example, the system or other training system may have pre-trained the teacher point tracking neural network 510 with one of the loss functions above.

[0181] For example, the teacher neural network may have architecture 410, 420, or 430, or a different point tracking neural network architecture.

[0182] 5, the teacher and student point tracking neural networks 510 and 520 may have the same architecture, or more generally, the student point tracking neural network 520 may include all of the same parameters as the teacher neural network 510, and optionally additional parameters. For example, the student 520 may include, as part of its backbone, one or more additional convolutional residual layers, each initialized to represent the identity transformation.

[0183] To take advantage of this, before training the student point tracking neural network 520, the system uses the teacher point tracking neural network 510 to initialize the parameters of the student point tracking neural network 520, i.e., to become the pre-trained network parameters 502.

[0184] Next, the system trains the student point tracking neural network 520 over multiple training steps.

[0185] In each training step, the system receives a set of one or more training video sequences from a training dataset that includes multiple training video sequences. As shown in the example of Figure 5, the videos in the training dataset may be real-world videos that include real-world video frames.

[0186] For each training video sequence, the system applies one or more transformations to the training video sequence to generate a transformed video sequence ("degraded real frame"). For example, the one or more transformations include spatial transformations, image degradation, or both.

[0187] As a concrete example, given an input video, the system can create a second view of the input video by resizing each frame to a smaller resolution (e.g., varying linearly over time over the course of training) and overlaying them on a black background at random positions (also varying linearly over time). This can be computed as a frame-by-frame axis-wise affine transformation Φ to coordinates and applied to the pixels.

[0188] Optionally, the system can further degrade this view by applying random JPEG degradation to make the task more difficult before pasting it onto a black background. Both actions result in the loss of texture information, so the network needs to learn high-level, possibly semantic, cues (e.g., the tip of the cat's top left ear) to accurately track points, rather than low-level texture matching.

[0189] The system then selects one or more supervised query points within the training video sequence. For example, the system may sample each supervised query point by uniformly randomly sampling both the position and time-step index within the training video sequence.

[0190] For each training query point, the system generates a training point tracking output for the training query point by processing the video sequence using a training point tracking neural network 510.

[0191] The system identifies corresponding student query points in the transformed video sequence, i.e., student query points in the transformed video sequence that correspond to the teacher query points.

[0192] Specifically, the system may use the teacher point tracking output to identify initial student query points and then transform the initial student query points to match one or more transformations to generate the student query points, i.e., the system may also apply Φ to the student query coordinates to generate the final student query point coordinates.

[0193] For example, to identify the initial student query points, the system can randomly sample points from the trajectories generated for the teacher query points. This can force the trajectories to form an equivalence class by training the student neural network 520 to generate identical trajectories regardless of which point is used as the query.

[0194] The system generates student point tracking outputs for the student query points by processing the transformed video sequence using a student point tracking neural network 520.

[0195] The system then trains a student point tracking neural network for each training video sequence and for each teacher query point using a loss function that measures the difference between (i) the student point tracking output for the corresponding student query point and (ii) the teacher point tracking output for the teacher query point.

[0196] For example, the system can generate pseudo labels from the teacher point tracking outputs for each training video sequence and for each teacher query point. The loss function can then measure the error between the pseudo labels and the student point tracking outputs.

[0197] For example, when training using the above loss, the system can generate pseudo-labels for a given video frame by (i) setting the ground truth location in the given video frame to the spatial prediction in the teacher point tracking output, (ii) setting the ground truth occlusion score to 1 only if the teacher occlusion estimate exceeds a threshold, e.g., zero, and, optionally, (iii) setting the ground truth uncertainty to 1 only if the distance between the student prediction and the teacher prediction for the frame exceeds a threshold distance.

[0198] The system can then train a student point tracking neural network with the loss function above, but using the pseudo-labels instead of the ground truth outputs.

[0199] Note, however, that if the teacher does not track points correctly, the student's query may be a different real-world point than the teacher, resulting in an erroneous training signal. To account for this, in some implementations, the system may determine whether to mask out the teacher query point from the loss function by enforcing cycle consistency between the student point tracking output and the teacher point tracking output for each teacher query point, and then decide to mask out teacher query points that are not cycle consistent. "Masking out" a teacher query point refers to removing or setting to zero all quantities that depend on the teacher query point when calculating the loss function.

[0200] Furthermore, for points closer in time to the student's query frame, the teacher's prediction may be less accurate than the student's prediction. To account for this, in some implementations, the system may determine, for each teacher query point, whether to mask out the teacher query point's prediction for any training video frame from the loss function by applying a proximity mask. That is, the system may apply a proximity mask to mask out any training video frames that are closer to the student's query frame than the teacher's query frame from the loss function. "Masking out" a training video frame for a teacher query point refers to removing or setting to zero all quantities that depend on the training video frame and the teacher query point when calculating the loss function.

[0201] Optionally, during unsupervised training, the system may continue training the point-tracking neural network through supervised learning, e.g., to help avoid catastrophic forgetting. In these cases, during some or all of the training steps, the system also trains a student neural network through supervised learning on a set of labeled video sequences.

[0202] During some or all of the training steps, the system may use the student point tracking neural network 520 to update the teacher point tracking neural network 510. For example, as shown in Figure 5, at specified points during training, the system may update the parameters of the teacher point tracking neural network 510 to be the exponential moving average (EMA) of the parameters of the student point tracking neural network 520. That is, rather than fixing the teacher point tracking neural network 510 during the training process, the system continues to update the teacher neural network 510 during training to improve the quality of the training process.

[0203] The point tracking neural network described with reference to FIG. 4 or FIG. 5 may be used for any of a variety of purposes.

[0204] For example, a point tracking neural network can be used to generate training data for the generative neural network described above.

[0205] As another example, the predicted positions generated by a point-tracking neural network may be used to generate reward signals for training a robot or other agent through reinforcement learning; for example, if a task performed by the agent requires moving a point in a scene from one location to another, the distance between the predicted position of the point in the last video frame in the sequence and the target position can be used to generate a reward.

[0206] As another example, the predicted positions (and optionally occlusion scores) generated by the point tracking neural network may be provided as additional inputs to a policy neural network for controlling a robot or other agent interacting with the environment. In this example, the query point may be a point of interest in the last frame of a video sequence, and predictions for previous frames in the sequence may be provided as inputs to the policy neural network to provide signals regarding, for example, the recent movement of the agent or other objects in the environment.

[0207] As another example, the predicted positions (and optionally occlusion scores) generated by the point tracking neural network may be provided as input to a video understanding neural network, such as an action classification neural network or a topic classification neural network, along with the corresponding video sequence, to provide the video understanding neural network with additional information about motion in the scene.

[0208] As another example, the predicted positions (and optionally occlusion scores) generated by a point-tracking neural network can be used to enable imitation learning, i.e., imitation of motion rather than appearance.

[0209] As described above, the first generating neural network and / or the second generating neural network may be a diffusion neural network. In general, a diffusion neural network is a neural network configured to process, at any given time step, diffusion inputs including (i) a current noisy data item (such as a noisy trajectory, noisy coordinates, a noisy occlusion estimate, or a noisy video frame) and, optionally, (ii) data identifying the given time step, to generate a denoising output that defines an estimate of the noise component of the current noisy data item given the current time step. The noise component estimate is an estimate of the noise added to the original data item (such as a point trajectory or a video frame) to generate the current noisy data item. The denoising output may be, for example, an estimate of the noise component or an estimate of the original data item.

[0210] After training, the system or other inference system uses the trained diffusion neural network to generate output data items (such as point loci or video frames) over multiple time steps by performing a de-diffusion process that progressively de-noises the initial data items until a final output data item is reached, where some or all of the values ​​in the initial data items are noisy, i.e., sampled from an appropriate noise distribution.

[0211] That is, the initialized data item has the same dimensions as the final data item but has noisy values. For example, the system can initialize the data item, i.e., generate a first instance of the data item, by sampling each value in the data item from a corresponding noise distribution, e.g., a Gaussian distribution, or a different noise distribution. That is, the output data item includes multiple values, and the initial data item includes the same number of values, some or all of which are sampled from the corresponding noise distribution.

[0212] As mentioned above, the pixel diffusion neural network may generally have any suitable neural network architecture that enables the pixel diffusion neural network to map an input to an output image.

[0213] In some implementations, the architecture of the pixel diffusion neural network may be similar to the U-Net neural network architecture described by O. Ronneberger et al. in “U-Net: Convolutional Networks for Biomedical Image Segmentation,” arXiv:1505.04597. In particular examples, the pixel diffusion neural network may be implemented as a convolutional neural network including a downward analysis path and an upward synthesis path, each path including multiple neural network layers. The analysis path may include multiple downsampling, e.g., convolutional layers, and the synthesis path may include multiple upsampling, e.g., upconvolutional layers. In addition to the convolutional layers, the upsampling and / or downsampling may be implemented partially or fully by interpolation. The neural network may include shortcut skips or residual connections between equal-resolution layers in the analysis path and the synthesis path. In some implementations, at least one of the sets of one or more layers between the analysis path and the synthesis path includes a set of fully connected layers.

[0214] In some implementations, the first or second generative neural network may include one or more self-attention layers. In general, a self-attention layer may apply an attention mechanism to elements of an embedding (e.g., input data) to update each element of the embedding; for example, the input embedding is used to determine a query vector and a set of key-value vector pairs (query-key-value attention), and the updated embedding includes a weighted sum of values ​​weighted by a similarity function of the query for each respective key. There are many different attention mechanisms that may be used. For example, the attention mechanism may be a dot-product attention mechanism applied by applying the query vector to each key vector to determine a respective weight for each value vector, and then combining the multiple value vectors using the respective weights to determine the attention layer output for each element of the input sequence.

[0215] The systems and methods described herein can be used to predict how a physical system or environment at a particular time will evolve over one or more subsequent time steps. That is, input images may depict a scene including a physical environment, and corresponding video generated from the input images may include video frames that predict the physical environment at one or more time steps after the particular time. For example, the input images can include one or more objects, and each video frame can be a prediction of the position and / or configuration of each of the objects in the physical environment at the corresponding time step.

[0216] The video can be used as input to a control task. For example, the video can be used for model predictive control of an agent, such as a robot or machine agent. In such cases, the noisy trajectory may be conditioned with one or more actions that can be performed by the robot or machine agent, and the video is a prediction of the physical environment that would result if the robot or machine agent performed the one or more actions. Thus, the video can be used by a control system to control a machine agent, such as a robot, to perform a specific task by processing the predicted video using the control system to generate control signals that control the machine agent to perform the task according to the video. The input image can be acquired by one or more sensors in the physical environment, for example, one or more sensors of the robot or machine agent. The input image can depict a part of the robot or machine agent, such as a gripper hand or other device for manipulating an object. As another example, video frames, or embeddings of video frames, can be provided as additional input to a policy neural network used to select actions to be performed by a robot or other agent interacting with a physical environment to perform a specific task, such as navigation within the physical environment or identifying or manipulating one or more objects within the physical environment.

[0217] In some implementations, the systems and methods described herein may be used for human pose estimation or prediction. For example, an input image may depict a person posing at a particular time, and then each video frame may be a prediction of the pose the person will assume at a corresponding time after the particular time.

[0218] This specification uses the term "configured" in the context of systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when running, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action. With reference to a particular neural network, the neural network may be configured to perform a particular action by being trained to perform the particular action.

[0219] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and structural equivalents thereof, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagating signal, e.g., a mechanically generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus.

[0220] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0221] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or code portions. A computer program can be deployed to be executed on one computer or to be executed on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0222] As used herein, the term "database" is used broadly to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0223] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and run on the same computer or computers.

[0224] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0225] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or on other types of central processing units. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0226] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0227] To provide for user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0228] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units for handling common computationally intensive parts of machine learning training or production, such as, for example, inference, workloads.

[0229] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0230] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an application that enables a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0231] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example, to display the data to and receive user input from a user interacting with the device functioning as a client. Data generated at the user device, e.g., results of user interaction, can be received from the device by the server.

[0232] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of patentable subject matter, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0233] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0234] Specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0235] Aspects of the present disclosure may be as described in the following clauses.

[0236] Clause 1. A method implemented by one or more computers, comprising: receiving data specifying a trained point tracking neural network configured to receive (i) a video sequence including a plurality of video frames, and (ii) a trained point tracking input including a query point in one of the plurality of video frames, and to generate, for each other video frame of the plurality of video frames, a trained point tracking output including a location of the query point in the other video frame and an occlusion estimate of the query point in the other video frame; and (ii) a student point tracking input including the query point in one of the plurality of video frames; and generating, for each other video frame of the plurality of video frames, a student point tracking output including a position of the query point in the other video frame and an occlusion estimate of the query point in the other video frame; wherein the training includes, in each of a plurality of training steps: receiving a set of one or more training video sequences from a training dataset comprising a plurality of training video sequences; For each training video sequence, applying one or more transformations to the training video sequence to generate a transformed video sequence; selecting one or more supervised query points within the training video sequence; For each supervised query point, generating a training point tracking output for the training query points by processing the video sequence using the training point tracking neural network; identifying corresponding student query points within the transformed video sequence; generating student point tracking outputs for the student query points by processing the transformed video sequence using the student point tracking neural network; To train and and training the student point tracking neural network using a loss function that, for each training video sequence and for each teacher query point, measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point.

[0237] Clause 2. The method of clause 1, wherein prior to said training, said teacher point tracking neural network is pre-trained.

[0238] Clause 3. The method of clause 2, wherein the teacher point tracking neural network is pre-trained through supervised learning.

[0239] Clause 4. The method of clause 3, wherein the point tracking neural network is pre-trained through supervised learning on a dataset of synthetic video sequences.

[0240] Clause 5. The method of any one of clauses 1 to 4, wherein the teacher and student point tracking neural networks have the same architecture.

[0241] Clause 6. The method of clause 5, further comprising initializing parameters of the student point tracking neural network using the teacher point tracking neural network before training the student point tracking neural network.

[0242] Clause 7. The training comprises, in each of at least a subset of the training steps: 7. The method of any one of clauses 5 or 6, further comprising updating the teacher point tracking neural network using the student point tracking neural network.

[0243] Clause 8. Updating the teacher point tracking neural network using the student point tracking neural network includes: 8. The method of clause 7, comprising updating parameters of the teacher point tracking neural network to be an exponential moving average (EMA) of parameters of the student point tracking neural network.

[0244] Clause 9. The method of any one of clauses 1 to 8, wherein the one or more transformations include a spatial transformation.

[0245] Clause 10. The method of any one of clauses 1 to 9, wherein the one or more transformations include image degradation.

[0246] Clause 11. Identifying corresponding student query points within the transformed video sequence includes: identifying initial student query points using the teacher point tracking output; and transforming the initial student query points to match the one or more transformations to generate the student query points.

[0247] Clause 12. The method of any one of clauses 1 to 11, wherein the teacher point tracking output and the student point tracking output further include a respective uncertainty score for each predicted location.

[0248] Clause 13. Training the student point tracking neural network using a loss function that, for each training video sequence and for each teacher query point, measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point, comprises: 13. The method of any one of clauses 1 to 12, comprising generating pseudo labels from the teacher point tracking output for each training video sequence and for each teacher query point, wherein the loss function measures the error between the pseudo labels and the student point tracking output.

[0249] Clause 14. Training the student point tracking neural network using a loss function that, for each training video sequence and for each teacher query point, measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point, comprises: 14. The method of any one of clauses 1 to 13, comprising determining whether to mask out the teacher query points from the loss function by enforcing cycle consistency between the student point tracking output and the teacher point tracking output.

[0250] Clause 15. Training the student point tracking neural network using a loss function that, for each training video sequence and for each teacher query point, measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point, comprises: 15. The method of any one of clauses 1 to 14, comprising determining whether to mask out predictions of the teacher query points in any training videos from the loss function by applying a proximity mask.

[0251] Clause 16. The training comprises, in each of a plurality of training steps: 16. The method of any one of clauses 1-15, further comprising training the student neural network through supervised learning on a set of labeled video sequences.

[0252] Article 17. One or more computers; one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method described in any one of clauses 1 to 16.

[0253] Clause 18. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of clauses 1 to 16.

[0254] Clause 19. A method, executed by one or more computers, for generating a video that animates an input image over a number of time steps, comprising: receiving the input image; processing a first input derived from the input image using a first generative neural network to generate a respective point locus for each of one or more points in the input image, each point locus including, for each of the plurality of time steps in the video, a predicted spatial location of a corresponding point in a video frame at the time step in the video; generating each of the video frames in the video frames that animate the input image using a second generative neural network and based on the input image and the one or more point loci.

[0255] Clause 20. The method of clause 19, wherein each point trajectory includes, for each of the time steps, (i) the predicted spatial location of the corresponding point in the video frame at that time step, and (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded in the video frame at that time step.

[0256] Clause 21. The method of clause 19 or clause 20, further comprising processing the input image using an image encoder neural network to generate an encoded representation of the input image, wherein the first input comprises the encoded representation of the input image.

[0257] Clause 22. The method of any one of clauses 19 to 21, wherein the first generative neural network is a diffusion neural network that generates each point trajectory from a corresponding noisy trajectory conditioned on the first input.

[0258] Clause 23. The method of clause 22, wherein the first generative neural network is trained with a training data set including (i) a plurality of video sequences and (ii) for each of the video sequences, a respective point trajectory for each of one or more points in a first frame within the video sequence.

[0259] Clause 24. The method of clause 22 or clause 23, wherein the first generator neural network comprises a two-dimensional convolutional neural network.

[0260] Clause 25. The method of clause 24, wherein the two-dimensional convolutional neural network is a U-Net.

[0261] Clause 26. The method of clause 24 or clause 25, wherein the two-dimensional convolutional neural network includes one or more self-attention layers.

[0262] Clause 27. The method of any one of clauses 22 to 26, wherein each corresponding noisy trajectory comprises a concatenation of noisy coordinates and noisy occlusion estimates.

[0263] Clause 28. The method of clause 27, wherein the noisy coordinates are extended with a positional coding.

[0264] Clause 29. The method of clause 23, wherein, for at least a subset of the video sequence, one or more of the point loci are generated by processing input including the corresponding points in the first frame using a point tracking neural network.

[0265] Clause 30. The point tracking neural network performs, for each video sequence in the subset and for a corresponding point in the first frame in the video: generating query features for the corresponding points; generating a cost volume using the query features, the cost volume including a respective cost map for each of a plurality of frames in the video sequence; and for each of the plurality of frames, using the cost map for the frame to generate an initial position of the corresponding point in the frame and an initial occlusion estimate for the corresponding point in the frame.

[0266] Clause 31. The point tracking neural network comprises: 31. The method of clause 30, further configured to generate the point locus for the corresponding points by refining the initial positions, refining the initial occlusion estimates for the plurality of frames, or both, using an initial point locus that includes the initial positions and the initial occlusion estimates for the plurality of frames.

[0267] Clause 32. Refining the initial positions and the initial occlusion estimates of the plurality of frames using an initial point trajectory including the initial positions and the initial occlusion estimates of the plurality of frames includes, in each of one or more refinement iterations: generating, for each frame from the current point trajectory at the time of the refinement iteration, a set of local score maps capturing the similarity between the query features and features in the vicinity of the predicted position in the current point trajectory in the frame; and processing an input comprising the set of local score maps using a refinement neural network to generate updates to the current point trajectory.

[0268] Clause 33. The method of clause 32, wherein the refinement neural network is a depthwise mixing neural network that uses depthwise convolutional layers to propagate information between frames.

[0269] Clause 34. The method of clause 32 or clause 33, wherein the input includes the set of local score maps, the query features, and the current point trajectory.

[0270] Clause 35. A method according to any one of clauses 19 to 34, wherein the second generative neural network is a diffusion neural network that generates each frame from a corresponding noisy frame conditioned on conditioning inputs derived from the input image and the one or more point trajectories.

[0271] Clause 36. The method of clause 35, wherein for each image, the conditioning input comprises a warped version of the input image warped according to the one or more point loci to represent the image.

[0272] Clause 37. The method of clause 36, wherein for each image, the conditioning input comprises a warped version of features extracted from the input image warped according to the one or more point loci to represent the image.

[0273] Clause 38. The method of clause 36 or clause 37, wherein the diffusion neural network generates each frame over multiple despreading iterations, and for at least a subset of the frames, the input at a given despreading iteration includes a current version of one or more previous frames in the video at the time of the despreading iteration.

[0274] Article 39. One or more computers; one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective method operations of any one of clauses 19 to 38.

[0275] Clause 40. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of clauses 19 to 38.

Claims

1. 1. A method executed by one or more computers for generating a video that animates an input image over a number of time steps, comprising: receiving the input image; processing a first input derived from the input image using a first generative neural network to generate a respective point locus for each of one or more points in the input image, each point locus including, for each of the plurality of time steps in the video, a predicted spatial location of a corresponding point in a video frame at the time step in the video; generating each of the video frames in the video that animate the input image using a second generative neural network and based on the input image and the one or more point loci.

2. 2. The method of claim 1 , wherein each point trajectory includes, for each of the time steps, (i) the predicted spatial location of the corresponding point in the video frame at that time step, and (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded in the video frame at that time step.

3. 3. The method of claim 1 or claim 2, further comprising processing the input image using an image encoder neural network to generate an encoded representation of the input image, wherein the first input comprises the encoded representation of the input image.

4. 4. The method of claim 1, wherein the first generative neural network is a diffusion neural network that generates each point trajectory from a corresponding noisy trajectory conditioned on the first input.

5. 5. The method of claim 4, wherein the first generative neural network is trained with a training data set comprising: (i) a plurality of video sequences; and (ii) for each of the video sequences, a respective point locus for each of one or more points in a first frame within the video sequence.

6. The method of claim 4 or claim 5, wherein the first generator neural network comprises a two-dimensional convolutional neural network.

7. The method of claim 6, wherein the two-dimensional convolutional neural network is a U-Net.

8. The method of claim 6 or claim 7, wherein the two-dimensional convolutional neural network includes one or more self-attention layers.

9. The method of any one of claims 4 to 8, wherein each corresponding noisy trajectory comprises a concatenation of noisy coordinates and noisy occlusion estimates.

10. The method of claim 9 , wherein the noisy coordinates are extended with a positional encoding.

11. 6. The method of claim 5, wherein, for at least a subset of the video sequence, one or more of the point loci are generated by processing input including the corresponding points in the first frame using a point tracking neural network.

12. The point tracking neural network calculates, for each video sequence in the subset and for a corresponding point in the first frame in the video: generating query features for the corresponding points; generating a cost volume using the query features, the cost volume including a respective cost map for each of a plurality of frames in the video sequence; and for each of the plurality of frames, using the cost map for the frame to generate an initial position of the corresponding point in the frame and an initial occlusion estimate for the corresponding point in the frame.

13. The point tracking neural network 13. The method of claim 12, further comprising: generating the point locus for the corresponding points by refining the initial positions, refining the initial occlusion estimates for the plurality of frames, or both, using an initial point locus that includes the initial positions and the initial occlusion estimates for the plurality of frames.

14. Refining the initial positions and the initial occlusion estimates for the plurality of frames using an initial point trajectory including the initial positions and the initial occlusion estimates for the plurality of frames includes, in each of one or more refinement iterations: generating, for each frame from the current point trajectory at the time of the refinement iteration, a set of local score maps capturing similarities between features in the vicinity of the predicted position in the current point trajectory in the frame and the query features; and processing inputs comprising the set of local score maps using a refinement neural network to generate updates to the current point trajectory.

15. 15. The method of claim 14, wherein the refinement neural network is a depthwise mixing neural network that uses depthwise convolutional layers to propagate information between frames.

16. The method of claim 14 or claim 15, wherein the input comprises the set of local score maps, the query features, and the current point trajectory.

17. 17. The method of any one of claims 1 to 16, wherein the second generative neural network is a diffusion neural network that generates each frame from a corresponding noisy frame conditioned on conditioning inputs derived from the input image and the one or more point loci.

18. 18. The method of claim 17, wherein for each image, the conditioning input comprises a warped version of the input image warped according to the one or more point loci to represent the image.

19. 20. The method of claim 18, wherein for each image, the conditioning input comprises a warped version of features extracted from the input image warped according to the one or more point loci to represent the image.

20. 20. The method of claim 18 or claim 19, wherein the diffusion neural network generates each frame over multiple despreading iterations, and for at least a subset of the frames, the input at a given despreading iteration comprises a current version of one or more previous frames in the video at the time of the despreading iteration.

21. 21. The method of any one of claims 1 to 20, wherein the input images depict a real-world environment at a particular time, and each of the video frames is a prediction of the real-world environment at a corresponding time after the particular time.

22. 22. The method of claim 21, further comprising using a control system to control a robot or machine agent to perform a particular task by processing the video using the control system to generate one or more control signals that control the robot or machine agent to perform the task in accordance with the video.

23. 23. A method according to any preceding claim, wherein the input image depicts a person in a pose at a particular time, and each video frame is a prediction of a pose that the person will assume at a corresponding time after the particular time.

24. 1. A method, executed by one or more computers, for generating, for each of one or more video sequences, a respective point locus for each of one or more points in a first frame in the video sequence, the method comprising: generating one or more of the point loci by processing an input including the corresponding points in the first frame using a point tracking neural network, the point tracking neural network: For each video sequence, and for the corresponding point in the first frame in the video, generating query features for the corresponding points; generating a cost volume using the query features, the cost volume including a respective cost map for each of a plurality of frames in the video sequence; for each of the plurality of frames, generating an initial position of the corresponding point in the frame and an initial occlusion estimate for the corresponding point in the frame using the cost map for the frame; generating the point locus of the corresponding points by refining the initial positions, refining the initial occlusion estimates for the plurality of frames, or both, using an initial point locus that includes the initial positions and the initial occlusion estimates for the plurality of frames.

25. Refining the initial positions and the initial occlusion estimates for the plurality of frames using an initial point trajectory including the initial positions and the initial occlusion estimates for the plurality of frames includes, in each of one or more refinement iterations: generating, for each frame from the current point trajectory at the time of the refinement iteration, a set of local score maps capturing similarities between features in the vicinity of the predicted position in the current point trajectory in the frame and the query features; and processing inputs comprising the set of local score maps using a refinement neural network to generate updates to the current point trajectory.

26. 26. The method of claim 25, wherein the refinement neural network is a depthwise mixing neural network that uses depthwise convolutional layers to propagate information between frames.

27. The method of claim 25 or 26, wherein the input comprises the set of local score maps, the query features, and the current point trajectory.

28. 28. The method of any one of claims 24 to 27, wherein the method is for generating a training data set comprising: (i) the one or more video sequences; and (ii) for each of the video sequences, the respective point loci for each of one or more points in the first frame within the video sequence.

29. one or more computers; and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the method of any one of claims 1 to 28.

30. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 28.

Images