Image animation using dot trails

Abstract

A method, system, and apparatus for animating an image using point loci, including a computer program encoded on a computer storage medium.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Application No. 63 / 450,951 filed on 8 March 2023, No. 63 / 452,405 filed on 15 March 2023, and No. 63 / 548,824 filed on 1 February 2024. The disclosures of the prior applications are deemed to be part of the disclosures of this application and are incorporated into the disclosures of this application by reference. [Background technology]

[0002] This specification relates to processing inputs, including images, using neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from a given input according to the current values ​​of its respective set of parameters. [Overview of the Initiative]

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations, which animates images using a generative neural network system.

[0005] "Animating" an input image means generating a video that represents the animation of the input image, for example, a video that depicts how the scene depicted in the input image changes over time. Examples of changes over time include the movement of objects in the scene, as well as changes in lighting and other image properties.

[0006] This specification also describes techniques for training point-tracking neural networks.

[0007] A point-tracking neural network is a neural network that processes an input video to generate a network output that, for each query point in a given set of one or more query points within a given frame of the input video, contains the predicted spatial location of each query point in other video frames in the sequence.

[0008] After training a point-tracking neural network, it can be used for any of a variety of purposes.

[0009] For example, a point-tracking neural network can be used to generate training data for training a generative neural network within a generative neural network system that animates images.

[0010] Other use cases for point-tracking neural networks are described below.

[0011] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages:

[0012] Animating still images is generally a challenging problem, both because video modeling is typically computationally intensive and because it is difficult to predict realistic motion from a single still image.

[0013] To address these challenges, the system described breaks down the difficult task of animating a still image into two parts: (i) first generating a set of dot trajectories from the input image, and (ii) then generating a video from the dot trajectories and the input image. That is, the system first generates a set of dot trajectories, which are a dense explicit representation of the movement of the surfaces of objects in the scene depicted in the input image, and then generates the pixels of the frames in the video from the dot trajectories and the input image.

[0014] Therefore, the system described can use dot trajectories to direct attention to the correct positions in the input image when generating pixels for a frame in order to produce a consistent appearance throughout the video. Specifically, dot trajectories define the motion that should exist in the video and ensure the physical validity of the video because the video is generated conditionally on the dot trajectories that have already been generated.

[0015] In addition, by first generating point trajectories, the system decomposes the video generation problem into two computationally efficient steps, ensuring that the video generated by the system shows realistic movement of the objects depicted in the input image.

[0016] Furthermore, animating a still image without additional information is a poorly set problem because for any given object depicted in a still image, there can be many possible realistic future trajectories. By utilizing a diffuse neural network to generate point trajectories from a still image, the system can effectively sample from a space of realistic surface motion trajectories to ensure that the ultimately generated video represents a realistic sample from a highly multimodal distribution of the object's motion. Moreover, because the system uses a diffuse neural network to generate point trajectories, it can effectively sample multiple reasonable sets of point trajectories given the same input image, thereby enabling the system to generate multiple different reasonable videos from the same input image, each representing a different sample from the distribution of the object's motion.

[0017] This specification also describes techniques for training point-tracking neural networks that effectively utilize unlabeled data to improve the training of neural networks through unsupervised learning. In a specific example, a system can use the described unsupervised learning techniques to fine-tune a pre-trained point-tracking neural network that has been trained through supervised learning. That is, a system can use the described unsupervised learning techniques to leverage unlabeled data and improve the performance of a pre-trained neural network. For example, a neural network may have been trained on a dataset of synthetic video sequences, e.g., synthetic video sequences only, or mainly synthetic video sequences, and the system may then further train the neural network ("fine-tune") through unsupervised learning on fine-tuning data that includes unlabeled real-world video sequences. This can improve the point-tracking neural network's ability to generalize to a variety of real-world videos, i.e., its ability to perform point-tracking tasks that require processing real-world video sequences even when labeled real-world data is unavailable or limited. Specifically, while pixel-level ground truth trajectories can be easily generated when creating synthetic video, obtaining accurate pixel-level trajectory labels for real-world video can be difficult or impossible. The technique described allows the system to incorporate unlabeled real-world video to improve the performance of a trained point-tracking neural network.

[0018] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0019] [Figure 1A] An example of generating a video that animates an input image is shown. [Figure 1B] It is a diagram of an exemplary image animation system. [Figure 2] It is a flowchart of an exemplary process for generating a set of point trajectories from an input image. [Figure 3] It is a flowchart of an exemplary process for generating a video from an input image and a set of point trajectories. [Figure 4] An exemplary architecture of a point-tracking neural network is shown. [Figure 5] An example of training a point-tracking neural network is shown.

Mode for Carrying Out the Invention

[0020] Like reference numerals and symbols in the various drawings refer to like elements.

[0021] FIG. 1A shows an example of generating a video 102 that animates an input image 104. [[ID=B]]

[0022] As shown in the example of FIG. 1A, an image animation system implemented as a computer program on one or more computers at one or more locations receives an input image 104.

[0023] [[ID=B]] For example, the system can receive the input image 104 from a user of the system.

[0024] The system then "animates" the input image 104 by generating a video 102 that represents an animation of the input image 104 depicting how the scene depicted in the input image 104 changes over time.

[0025] In other words, even if the input image 104 is a "still" image at a single point in time, the system generates a video 102 that includes each video frame at each of multiple time steps (starting with the input image 104 as the first frame at the first time step in the video), representing a realistic estimate of how the scene depicted in the input image 104 changes over time.

[0026] Specifically, the system breaks down the task of animating a video into two steps.

[0027] First, the system processes the input image 104 to generate a set of point trajectories 106.

[0028] Each point trajectory 106 corresponds to a different point in the input image 104 and includes, for each time step in the video, the predicted spatial position (generated by the system) of the corresponding point in the video frame at that time step.

[0029] Each point is a point in a corresponding video frame, that is, a point that identifies each pixel, which is the spatial position of each corresponding video frame among multiple video frames. Thus, each point in a given trajectory 106 can be represented as a point (x, y, t), where x and y are the spatial coordinates of the point, and t is the index of the corresponding video frame in video 102.

[0030] In Figure 1A, point trajectories are represented as dotted curves, with points at future points in time represented as points on the curve, and points closer to the tip of the curve being further away from the input image in time. For ease of illustration, only three point trajectories are shown in Figure 1, but in reality, the system can generate more point trajectories so that the trajectories represent a denser representation of the future movement of points in the input image 104. For example, the system can generate a point trajectory for each grid cell in a grid, for example, an 8x8 grid superimposed on the input image.

[0031] Optionally, the point locus 106 may also include an occlusion score for each frame, representing the likelihood that the corresponding point is occluded within the video frame at a given time step.

[0032] For example, the system can use a generative neural network to generate a point trajectory 106 from an input image 104.

[0033] This will be explained in more detail below.

[0034] The system then processes the input image 104 and the dot trajectory 106 to generate video 102, that is, it generates frames in video 102 from the input image 104 and the dot trajectory 106.

[0035] For example, the system can use other generative neural networks to generate a video 102 from the input image 104 and the point trajectory 106.

[0036] Although not shown in Figure 1A, in the example in Figure 1A, the dot trajectory 106 indicates that the dot on the depicted person's right arm is likely to move away from the person's body, so video 102 may show the person raising their right arm in a future frame. Similarly, the dot trajectory 106 indicates that the dot on the depicted person's left leg is likely to move upward and away from the person's body, so video may show the person kicking their left leg up in a future frame.

[0037] Therefore, the system breaks down the difficult task of animating a still image by first generating a dense explicit representation of the movement of the surface of objects in the scene depicted in input 104, i.e., a dot trajectory 106, and then generating pixels for frames in the video from the dot trajectory 106 and the input image 104.

[0038] Therefore, the system can use the trajectory 106 to direct attention to the correct position within the input image 104 when generating the pixels of the frame in order to produce a consistent appearance throughout the video. Specifically, the point trajectory 106 defines the movement that should exist within the video 102, ensuring physical validity.

[0039] Figure 1B is a diagram of an exemplary image animation system 100. The image animation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below may be implemented.

[0040] As described above, the system 100 generates a video 102 that animates the input image 104, that is, a video 102 that represents an animation of the input image 104 that depicts how the scene depicted in the input image 104 changes over time.

[0041] Specifically, system 100 receives input image 104. For example, the system can receive input image 104 as input from a user.

[0042] System 100 processes a first input derived from the input image 104 using a first generative neural network 110 to generate a point trajectory 106 for each of one or more points in the input image 104. The first generative neural network 110 is also referred to herein as the “trajectory model”.

[0043] For example, a point can be a pixel randomly sampled from the input image 104, a pixel on a grid superimposed on the input image 104, or a point identified by the user (or another user) who provided the input image 104.

[0044] Each point trajectory 106 includes, for each of the multiple time steps in the video 102, the predicted spatial position (generated by system 100) of the corresponding point in the video frame at that time step in the video 102.

[0045] Optionally, the point trajectory 106 may also include the occlusion score for each frame. The occlusion score for a given frame estimates the likelihood that the corresponding point will be occluded within the video frame at a given time step.

[0046] For example, the first generative neural network 110 may be a spread neural network that generates each point locus from corresponding noisy locuses conditioned on a first input. This spread neural network is referred to as a “locus” spread neural network. As used herein, “noisy” refers to a “noise distribution,” e.g., values ​​sampled from a Gaussian distribution or other suitable distribution. For a given point locus, the noisy locus therefore includes, for each spatial position within the given point locus, corresponding spatial positions sampled from the corresponding noise distribution.

[0047] A spread trajectory neural network can generally have any suitable neural network architecture that allows it to map an input containing noisy trajectories to a denoising output that defines updates to the noisy trajectories.

[0048] As an example, a spread trajectory neural network may be a two-dimensional convolutional neural network, such as a U-Net or other neural network having a convolutional architecture. Optionally, the spread trajectory neural network may include one or more self-attention layers as part of its convolutional architecture.

[0049] The generation of trajectory sets using a trajectory diffusion neural network is explained in more detail below, with reference to Figure 2.

[0050] Next, the system 100 uses a second generative neural network 120 and, based on the input image 104 and one or more point trajectories 106, generates each of the video frames in the video 102. The second generative neural network 120 is also referred to herein as the “pixel model”.

[0051] For example, the second generative neural network 120 could be a spread neural network that generates each video frame in the video (i.e., generates one or more intensity values ​​for each pixel of the video frame) from corresponding noisy video frames conditioned on the input image 104 and trajectory 106. This spread neural network is referred to as a “pixel” spread neural network.

[0052] A spread pixel neural network can generally have any suitable neural network architecture that allows the spread pixel neural network to map the input to an output image.

[0053] As an example, a spread pixel neural network may be a convolutional neural network, such as a U-Net or other neural network having a convolutional architecture. Optionally, the spread pixel neural network may include one or more self-attention layers as part of its convolutional architecture.

[0054] The process of generating video 102 will be explained in more detail below, with reference to Figure 3.

[0055] Therefore, system 100 uses two different generative neural networks: one that generates point trajectories 106, and another that generates video 102 given point trajectories 106. Thus, the output from the trajectory model defines the motion that should be present in video 102 generated by the pixel model, ensuring the physical validity of the generated video 102.

[0056] Once video 102 is generated, system 100 may use the video for any of several purposes. For example, system 100 may store video 102 or provide video 102 for presentation to a user, for example, the user who submitted the input image 104.

[0057] Before using the first and second generative neural networks 110 and 120 to generate video, system 100 or another training system trains the neural networks 110 and 120.

[0058] For example, the training system can train a first generative neural network 110 with a training dataset that includes (i) a plurality of video sequences, where each video sequence is a sequence of video frames, and (ii) for each video sequence, the respective point trajectories for each of one or more points in the first frame within the video sequence.

[0059] In other words, the training system can generate training data for training a first generative neural network 110 from a set of video sequences. For example, the system can generate training examples, each corresponding to one of the video sequences, each containing a first frame in the corresponding video sequence as a training input image, and the respective point trajectories for points in the first frame as target outputs.

[0060] The training system can then train the first generative neural network 110 with training examples using an objective function appropriate to the type of neural network being used. For example, if the first generative neural network 110 is a diffusion neural network, the objective function can be a score matching objective function.

[0061] As another example, the training system may train a second generative neural network 120 on the same training dataset.

[0062] In other words, the training system can generate training data for training a second generative neural network 120 from a set of video sequences. For example, the system can generate training examples, each corresponding to one of the video sequences, each including a first frame in the corresponding video sequence as a training input image, the point trajectories for each point in the first frame of the corresponding video sequence as a set of target point trajectories, and the corresponding video sequence as the target output.

[0063] The training system can then train a second generative neural network 120 with training examples using an objective function appropriate to the type of neural network being used. For example, if the second generative neural network 120 is a diffusion neural network, the objective function could be a score-matching objective function.

[0064] In some cases, the training system may generate point trajectories for at least some of the video sequences by processing them using a point-tracking neural network.

[0065] A point-tracking neural network is a neural network that processes an input video to generate a network output that, for each query point in a given set of one or more query points within a given frame of the input video, includes the predicted spatial position of each query point in other video frames in the sequence, and optionally an occlusion estimate for the query point.

[0066] In other words, because a large amount of densely labeled video may not be available for use in training generative neural networks, the training system may use a point-tracking neural network to predict point trajectories for at least a portion of the video sequences, and then use the predictions of the point-tracking neural network to train the first and second generative neural networks 110 and 120.

[0067] Point-tracking neural networks can generally have any suitable architecture and can be trained using any suitable technique.

[0068] Some examples of point-tracking neural network architectures are described below, with reference to Figure 4.

[0069] Another example of a point-tracking neural network architecture is described in Doersch, et al, TAP-Vid: A Benchmark for Tracking Any Point in a Video, arXiv:2211.03726.

[0070] An example of training a point-tracking neural network is described below, with reference to Figure 5.

[0071] Other examples of training point-tracking neural networks are described in Doersch, et al, TAP-Vid: A Benchmark for Tracking Any Point in a Video, arXiv:2211.03726, and Doersch, et al, TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement, arXiv:2306.08637.

[0072] Figure 2 is a flowchart of an exemplary process 200 for generating a set of point trajectories. For convenience, the process 200 is described as being performed by one or more computer systems located in one or more locations. For example, an image animation system, e.g., the image animation system 100 depicted in Figure 1B, can be appropriately programmed according to this specification and perform the process 200.

[0073] The system receives the input image (step 202).

[0074] The system processes the input image using an image encoder neural network to generate an encoded representation of the input image (step 204).

[0075] Generally, an encoded representation contains feature vectors for each of several spatial regions within the input image. For example, if the input image is an H×W image, the encoded representation may be an H / k×W / k map of feature vectors, where k may be equal to 4, 8, or 16.

[0076] An image encoder neural network can generally have any suitable neural network architecture for encoding an input image. For example, an image encoder neural network may be a convolutional neural network or a vision transformer neural network.

[0077] In some cases, the system uses a pre-trained image encoder neural network that has already been trained to generate image representations in a representation learning task. In some other cases, the system jointly trains the image encoder neural network with a first generative neural network.

[0078] The system generates a corresponding noisy trajectory for each point trajectory in the set (step 206).

[0079] Specifically, for a given point locus, the noisy locus contains corresponding values ​​for each value within the given point locus. To generate the noisy locus, the system samples each of these values ​​from a corresponding "noise distribution," such as a Gaussian distribution or another suitable distribution.

[0080] Therefore, as described above, each point trajectory includes, for each time step in the video, (i) the predicted spatial position of the corresponding point in the video frame at the time step, and optionally, (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded in the video frame at the time step.

[0081] The predicted spatial position within a video frame can be either an absolute spatial position, i.e., a position expressed as an absolute coordinate in the image coordinate system, or a relative spatial position, i.e., a position expressed as a relative coordinate to a corresponding point in the input image.

[0082] Therefore, the system samples the predicted spatial position and, if included, the occlusion score for each time step from the respective noise distribution, and as a result, the trajectory includes the noisy coordinates, i.e., the noisy predicted spatial position and the noisy occlusion estimate for each time step.

[0083] Optionally, the noisy trajectory may also include additional values ​​to help the first generative neural network effectively utilize the information contained within the noisy trajectory.

[0084] For example, noisy coordinates can be extended with position coding. That is, a noisy trajectory may include a position coding of the noisy coordinates for each set of noisy coordinates. As a specific example, the position coding could be Fourier position coding, which encodes the noisy coordinates using a fixed number of Fourier features.

[0085] As another example, noisy trajectories may also include "relative" noisy coordinates, such as the coordinates of the noisy predicted spatial position in a coordinate system centered on a corresponding point in the input image, in addition to "absolute" noisy coordinates.

[0086] Next, the system uses a first generative neural network to generate the trajectory of each point in the set from the corresponding noisy trajectories and the encoded representation of the input image (step 208).

[0087] Specifically, in the example in Figure 2, the first generative neural network is a diffuse neural network (trajectory diffuse neural network).

[0088] To generate point trajectories, the system uses a trajectory diffusion neural network to perform a series of despread iterations.

[0089] In each despread iteration, the system processes the input of the despread iteration, which includes the noisy point trajectory at the time of the despread iteration, using a trajectory spread neural network and subject to an encoded representation of the input image, in order to generate a denoising output that defines an update to the noisy point trajectory.

[0090] For example, the denoising output may be a prediction of the corresponding actual (unknown) point trajectory for each noisy point trajectory. As another example, the denoising output may be a prediction of the noise added to the corresponding actual point trajectory to generate the noisy point trajectory for each noisy point trajectory. As described above, when the input includes both absolute and relative coordinates, in some embodiments the denoising output predicts relative coordinates, while in some other embodiments the denoising output predicts absolute coordinates.

[0091] The system can be conditioned on an encoded representation of an input image in any variety of ways.

[0092] For example, the system may include an encoded representation as part of the input for iteration, for instance, concatenated with a noisy point trajectory.

[0093] Another example is a trajectory diffusion neural network, which may include one or more conditioning layers that receive encoded representations of input images as input.

[0094] For example, each conditioning layer may be a cross-attention layer that cross-attends to an encoded representation of the input image.

[0095] As another example, each conditioning layer may be a conditional group normalization layer. That is, after a group normalization layer performs mean subtraction and variance normalization within each group to produce a normalized output Z, the group normalization layer typically applies scale and shift operations. To produce a "conditional" group normalization layer, these are replaced by a linear projection of the conditioning input. For example, the system could resize the encoded representation so that its spatial dimension is the same as Z, and then apply each trained transformation, e.g., two 1x1 convolutional layers, to create scale and multipliers of the same size as Z. These scale and multipliers can be applied instead of scale and shift operations.

[0096] For each dedespanic iteration, the system then uses the denoising output to update the noisy trajectories. For example, the system can generate an estimate of the updated noisy trajectories from the denoising output, and then apply a diffusion sampler, such as a DDPM (Denoising Diffusion Probabilistic Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another suitable sampler, to the denoising output to generate the updated noisy trajectories. If the denoising output is a prediction of the corresponding actual (unknown) point trajectory for each noisy point trajectory, the system can use the denoising output directly as the estimate. If the denoising output is a prediction of the noise added to the corresponding actual point trajectory to generate the noisy point trajectory for each noisy point trajectory, the system can determine the estimate from the current noisy point trajectory, the denoising output, and the noise level of the current dedespanic iteration. Optionally, after the last dedespanic iteration, the system may refrain from using a diffusion sampler and instead use the estimate as the updated noisy trajectories.

[0097] Next, the system uses the updated noisy trajectory after the last descendance iteration to generate the final point trajectory.

[0098] If the point trajectory includes occlusion estimates, the system can apply a smoothing operation to continuate the occlusion during the despreading iteration, since the diffusion neural network operates in continuous space. The system can then de-calculate a binary occlusion estimate from the noisy occlusion estimate after the last despreading iteration.

[0099] for example,

number

number

number

number

number

number

number

number

[0100] Therefore, the system iteratively "de-noises" the noisy trajectory in order to generate the final point trajectory.

[0101] As described above, a spread trajectory neural network can generally have any suitable neural network architecture that enables it to map an input containing noisy trajectories to a denoising output that defines the update to the noisy trajectories.

[0102] As an example, a spread trajectory neural network may be a two-dimensional convolutional neural network, such as a U-Net or other neural network having a convolutional architecture. Optionally, the spread trajectory neural network may include one or more self-attention layers as part of its convolutional architecture.

[0103] Figure 3 is a flowchart of an exemplary process 300 for generating a video from a set of input images and point trajectories. For convenience, the process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, an image animation system, e.g., the image animation system 100 depicted in Figure 1B, can be appropriately programmed according to this specification and perform the process 300.

[0104] The system receives the input image and the set of point trajectories (step 302).

[0105] The system generates a corresponding noisy video frame for each video frame in the video (step 304).

[0106] Specifically, for a given video frame, a noisy video frame contains intensity values ​​corresponding to each intensity value within the given video frame. To generate a noisy video frame, the system samples each of these values ​​from a corresponding "noise distribution," such as a Gaussian distribution or another suitable distribution.

[0107] Next, the system uses a second generative neural network to generate each video frame from the corresponding noisy video frame, input image, and point trajectory (step 306).

[0108] Specifically, in the example in Figure 3, the second generative neural network is a diffuse neural network (pixel diffuse neural network).

[0109] To generate video frames, the system uses a pixel spread neural network to perform a series of despread iterations. That is, in each despread iteration, the system updates the corresponding noisy video frame for each video frame at each time step.

[0110] For each despread iteration and for each video frame, the system uses a pixel spread neural network to process the input of the despread iteration, which includes the corresponding noisy video frame at the time of the despread iteration, conditionally on the input image and point trajectory, in order to generate a denoising output that defines the update to the noisy video frame.

[0111] For example, the denoising output may be a prediction of the corresponding actual (unknown) video frame.

[0112] As another example, the denoising output may be a prediction of the noise added to the corresponding actual video frame to generate a noisy video frame.

[0113] The system can condition a pixel spread neural network on the input image and point trajectory in one of several ways.

[0114] As an example, the system may include, as input to a spread neural network for a given video frame at a given time, a version of the input image warped along one or more point trajectories to represent the image. That is, the system may generate a warped version of the input image warped along one or more point trajectories, i.e., a version that represents how the input image would look at that time if the points were moving along the point trajectories.

[0115] The system can generate warped versions of the input image in various arbitrary ways.

[0116] As an example, the system may use patch-based warping. For a given frame t, the trajectory at time t determines where each patch in the input image should appear. The system may then construct a new image in which each local patch is placed in its correct position, using (for example) bilinear interpolation to obtain subpixel accuracy. However, this may result in gaps between patches in certain situations. In some embodiments, to account for this, the system may actually warp larger patches around each point. If multiple patches appear that cover the same output pixel, the system may weight them inversely proportional to their distance from the track center.

[0117] Optionally, to address the fact that aliasing can occur when multiple patches overlap, the system may perform warping by warping each patch multiple times, where the difference between warps is how the system calculates blending weights. The system may then include all warped versions of the input image as input to a diffuse neural network. For example, p i,j,t Let be the position of the locus starting from points i and j in the original image at time t, according to the locus of points i and j. In the original warping, any specific pixel

number

number

number

number

[0118] Next, the system may include this warped version of the input image, along with the noisy version of the video frame, for example, by concatenating two images.

[0119] As another example, the system could be conditioned on a diffuse neural network to warped versions of features extracted from an input image, i.e., features of the input image warped according to one or more point trajectories to represent the image.

[0120] For example, a feature can be a feature vector in an encoded representation of the input image. For example, the encoded representation can be the same encoded representation described above with reference to the steps, or it can be a different encoded representation generated by a separate, individually trained image encoder neural network.

[0121] To warp these features over a time step t, the system can use the position of each feature at time step t according to a set of point loci. The system can then use (for example) bilinear interpolation to place the features in the appropriate positions within a grid of “warped” features. Optionally, the system can track the number of features placed within any given grid cell (specifically, the sum of the bilinear interpolation weights) and normalize by this sum, although if the sum is less than 0.5, the system may divide by 0.5.

[0122] The system can be conditioned to a warped version of the features by any suitable method, for example, using one of the conditioning techniques described above with reference to Figure 2.

[0123] In some embodiments, for at least a subset of frames, the system includes the temporal context from other frames in the video in the input of the frames to the spread neural network. For example, for a given frame, the input in a given despread iteration may include the current (noisy) version of one or more preceding frames in the video at the time of the despread iteration. Alternatively, or in addition, for a given frame, the input in a given despread iteration may include the current (noisy) version of one or more subsequent frames in the video at the time of the despread iteration.

[0124] For each despreading iteration, the system then uses the denoising output of the video frame to update the corresponding noisy video frame for each video frame. For example, the system can generate an estimate of the updated noisy video frame from the denoising output, and then apply a spread sampler, such as a DDPM (Denoising Diffusion Probabilistic Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or other suitable sampler, to the estimate to generate the updated noisy video frame. If the denoising output is a prediction of the corresponding actual (unknown) frame, the system can use the denoising output directly as the estimate. If the denoising output is a prediction of the noise added to the corresponding actual frame to generate the noisy frame, the system can determine the estimate from the current noisy frame, the denoising output, and the noise level of the current despreading iteration. Optionally, after the last despreading iteration, the system may refrain from using a spread sampler and instead use the estimate as the updated noisy frame.

[0125] Next, the system uses the updated noisy video frame from the last despread iteration to generate video frames within the video.

[0126] Figure 4 shows exemplary architectures 410, 420, and 430 of point-tracking neural networks.

[0127] A point-tracking neural network is a neural network that processes inputs that include (i) a video containing multiple video frames, and (ii) a set of one or more query points.

[0128] Each query point identifies a point in a corresponding video frame, i.e., its spatial position in a corresponding video frame, i.e., its individual pixel.

[0129] The point-tracking neural network 400 processes one or more sets of query points and a video sequence, i.e., the pixel intensity values ​​of the video frames in the video sequence, to generate a network output for each query point that includes the predicted spatial position of each query point in other video frames in the sequence, i.e., video frames other than the video frame corresponding to the query point.

[0130] Depending on the architecture or processing pipeline design of the neural network, the network output may also include the predicted spatial position of the corresponding video frame. However, in some of these cases, the system may ignore the predicted spatial position of the corresponding video frame (because the actual position in the corresponding video frame is provided as input to the system).

[0131] In other words, given a query point in one of the video frames, a point-tracking neural network can generate predictions of the spatial location of the query point in other video frames within the video.

[0132] The predicted position of a given query point in another given video frame is a prediction of the position of the part of the scene that was depicted at the given query point in the corresponding video frame. For example, if the corresponding video frame depicts a specific point on the surface of an object in the scene at a given query point, the predicted position of the given query point identifies the predicted position of the same specific point on the surface of the object in another given video frame.

[0133] In some embodiments, the point-tracking neural network also generates an occlusion score for each query point for each of the other video frames in the sequence. The occlusion score for a given query point in a given video frame represents the likelihood that the query point is occluded in the given video frame, i.e., the likelihood that the portion of the scene depicted at the query point in the corresponding video frame is occluded in the given video frame.

[0134] Specifically, in the exemplary architecture 410, the point-tracking neural network is configured to generate query features for corresponding points for a given video sequence and corresponding points within a given frame in the video sequence.

[0135] The neural network is then configured to use query features to generate cost volumes, each containing its own cost map for each of the multiple frames in the video sequence.

[0136] For example, a neural network processes a sequence of w x h video frames, and the video frame t q Query point (i q Today J q ,t q ) can be received.

[0137] The neural network processes video frames in a video sequence using a visual backbone neural network to generate a feature grid containing the respective visual features, i.e., feature vectors, for each of the multiple spatial positions within each video frame. Generally, each spatial position corresponds to a different region of the video frame. For example, the feature grid could be a w / 8 × h / 8 grid of d-dimensional visual features, where each visual feature corresponds to an 8 × 8 pixel grid from the corresponding video frame.

[0138] A visual backbone neural network can have any suitable architecture that allows the neural network to map a video sequence to a feature grid. In the example in Figure 3, the visual backbone neural network is a 3D convolutional neural network (ConvNet), e.g., TSM-ResNet-18 or another suitable convolutional neural network. In other examples, the visual backbone neural network could be a different type of neural network, e.g., a vision transformer neural network.

[0139] Next, the neural network can generate extraction features for the query point from the spatial location of the query point within the corresponding video frame, and from the visual features of one or more of the spatial locations within the corresponding video frame.

[0140] For example, a neural network can process the corresponding video frame t q The spatial position of each query within (i q Today J q Extracted features ("query features") can be generated by interpolating the visual features of a set of spatial locations within the local neighborhood of ), for example by performing bilinear interpolation.

[0141] Next, the neural network generates a cost volume from the feature grid and the extracted query points. For example, the cost volume may have a cost value for each spatial position in each video frame. That is, the cost volume contains an h' × w' × 1 grid of cost values ​​for each video frame in the sequence.

[0142] To calculate a cost value for a given spatial location within a given video frame, the system calculates the inner product between the extracted feature and the visual feature for the given spatial location within the given video frame.

[0143] Next, the neural network is configured to generate, for each of multiple frames, the initial position of the corresponding point in the frame and the initial occlusion estimate of the corresponding point in the frame, using the frame's cost map.

[0144] Generally, to generate predicted positions, a neural network may use a decoder neural network to handle the cost volume for each video frame, generating a score for each spatial position within the video frame.

[0145] For each of the multiple video frames other than the corresponding video frame, the neural network can then generate an initial predicted position from the respective scores for the spatial position within the video frame.

[0146] If the neural network also predicts occlusion, as shown in the example in Figure 4, the neural network may use a decoder neural network to process the cost volume to generate the predicted position of each query point within the video frame and the occlusion score for each query point within the video frame for each video frame other than the corresponding video frame.

[0147] For example, a neural network can perform this process independently for each video frame. That is, for a given video frame, the neural network uses a decoder neural network to process h' × w' × 1 portion of the cost volume for that video frame in order to generate the predicted position of the query point within the given video frame and the occlusion score for the query point within the video frame.

[0148] For example, a decoder neural network may include a set of shared layers and their respective branches for positional and occlusion inference.

[0149] The decoder neural network processes the cost volume portion for a given video frame using a convolutional layer followed by a shared layer, such as a rectified linear unit (ReLU) activation function layer, to generate a shared output.

[0150] In the case of an occlusion inference branch, the decoder neural network processes the shared output using layers within the occlusion inference branch to generate a single occlusion score ("logit") for a given video frame. For example, an occlusion inference branch includes a first set of layers that fold the shared output into a vector (e.g., using spatial average pooling), followed by a second set of layers that regress the occlusion logit for a given video frame from the single vector. For example, the second set of layers could include linear layers, Leaky ReLU, and other linear layers that generate a single logit. Leaky ReLU can be a ReLU activation function that has a gradient for negative values ​​(e.g., a relatively small gradient such as a gradient of 0.01) (as opposed to a flat gradient for negative values).

[0151] For the position inference branch, the neural network may apply a set of layers, such as a convolutional (Conv) layer with a single output followed by a spatial softmax, to generate a score for each spatial position within a video frame.

[0152] Next, the neural network can calculate a "soft argmax" to determine the position from each score.

[0153] To calculate soft argmax, the neural network can identify the spatial location with the highest score, i.e., the "argmax" location, according to the score of each spatial location. The neural network can then identify each spatial location within a fixed-size window B of the spatial location with the highest score, i.e., the argmax location, and then determine the predicted location by calculating a weighted average.

[0154] That is, the neural network determines the predicted position by calculating a weighted sum of the spatial positions within a fixed-size window of the argmax spatial positions, and the weight for each spatial position is calculated based on, for example, being equal to or being directly proportional to the ratio between the score of the spatial position and the sum of the scores of the spatial positions within the fixed-size window.

[0155] In an exemplary architecture 410, the neural network uses the initial position and the initial occlusion estimate as the final output of the neural network.

[0156] In this example, the training system can train the neural network with the following loss function for a given query point for each frame t in a given training video.

Number

Number

Number

Number

Number

number

number

[0157] In exemplary architectures 420 and 430, once initial estimates are generated, the neural network 400 generates point loci for corresponding points by refining the initial positions, refining the initial occlusion estimates for multiple frames, or both, using the initial point loci, which includes initial position and initial occlusion estimates for multiple frames.

[0158] Specifically, in the example in Figure 4, the neural network refines the initial position and initial occlusion estimates for multiple frames by performing one or more refinement iterations.

[0159] In each refinement iteration, the neural network 400 generates a set of local score maps for each frame, capturing the similarity between neighboring features of the predicted position in the current point trajectory within the frame and the query features, based on the current point trajectory at the time of the refinement iteration. For the first refinement iteration, the current point trajectory is the initial point trajectory. For subsequent refinement iterations, the current point trajectory is the updated point trajectory after the preceding refinement iteration.

[0160] Next, the neural network processes the input, which includes a set of local score maps, using an elevating neural network to generate an update for the current point trajectory. In a specific example, the input to the elevating neural network may include a set of local score maps, query features, and the current point trajectory.

[0161] For example, in exemplary architecture 430, the refined neural network is a depth-based mixed neural network that propagates information between frames using depth-based convolutional layers. Each depth-based convolutional layer may have a time-receptive field that extends across multiple (e.g., all or a subset) frames.

[0162] In this example, the training system can train the neural network with a loss that is the sum of the losses calculated above for the initial prediction and the prediction after all refined iterations.

[0163] In some embodiments, the neural network may also generate uncertainty estimates for each predicted spatial position, representing the neural network's uncertainty in the prediction. For example, the neural network may output additional logits in the occlusion path, and the uncertainty estimate u t To generate this, additional logits are processed. When a neural network utilizes refinement, it also refines the uncertainty estimate in each refinement iteration.

[0164] In this example, the above loss is an additional term.

number

number

number

[0165] Next, we will explain a specific example of architecture 430.

[0166] Given the estimated position, occlusion, and uncertainty for each frame, the purpose of each iteration i of the refinement procedure is to update the estimates to be closer to ground truth.

number

[0167] For example, these can be computed using pyramids of different resolutions, so that for a given trajectory, they have the shape (H' × W' × L), where H' = W' = 7 is the size of the local neighborhood and L is the number of levels in the spatial pyramid. For example, different pyramid levels can be computed by spatially pooling a feature volume F containing features for spatial positions within a video frame. This set of similarities is then post-processed with an elevation network to predict refined position, occlusion, and uncertainty estimates.

[0168] However, unlike initialization, a neural network simultaneously contains "local score maps" for multiple frames as input to post-processing. As a specific example, a neural network can contain the current position estimate, raw query features, and (flattened) local score maps in a tensor of shape T × (C + K + 4), where C is the number of channels in the query features, K = H'·W'·L is the number of values ​​in the flattened local score map, and 4 is an additional dimension for position, occlusion, and uncertainty.

[0169] The output of this refined network at the i-th iteration is the residual.

number

[0170] ΔF q,t,i The shape is T×C, and therefore, after the first iteration, a slightly different "query feature" is used in each frame when calculating a new local score map.

[0171] These location and score maps have been updated.

number

[0172] Figure 5 shows an example of training a point-tracking neural network, for example, a point-tracking neural network having one of architectures 410, 420, or 430, or a point-tracking neural network having a different network architecture 500.

[0173] In the example in Figure 5, the system "fine-tunes" a "pre-trained" point-tracking neural network through unsupervised learning.

[0174] As a specific example, a system can fine-tune a pre-trained point-tracking neural network, trained through supervised learning on datasets containing synthetic video sequences, such as synthetic video sequences alone or primarily synthetic video sequences, through unsupervised learning on fine-tuning data containing unlabeled real-world video sequences. This can improve the point-tracking neural network's ability to generalize to a variety of real-world videos, i.e., its ability to perform point-tracking tasks that require processing real-world video sequences.

[0175] Specifically, the system receives data specifying the training point tracking neural network 510.

[0176] The training neural network 510 is configured to receive a training point tracking input that includes (i) a video sequence containing multiple video frames and (ii) a query point in one of the multiple video frames, and to generate a training point tracking output for each of the other video frames of the multiple video frames that includes the location of the query point in the other video frame and an occlusion estimate for the query point in the other video frame.

[0177] Next, the system trains a student point-tracking neural network 520 through unsupervised learning.

[0178] Similar to the teacher neural network 510, the student point tracking neural network 520 is configured to receive a student point tracking input that includes (i) a video sequence containing multiple video frames and (ii) a query point in one of the multiple video frames, and to generate a student point tracking output for each of the other video frames that includes the location of the query point in the other video frames and an occlusion estimate for the query point in the other video frames.

[0179] Specifically, prior to training, the marker-tracking neural network 510 is pre-trained. In the example in Figure 5, the marker-tracking neural network is pre-trained through supervised learning, for example, on a dataset of synthetic video sequences, to yield pre-trained network parameters 502.

[0180] For example, the system or another training system may pre-train the marker-tracking neural network 510 with one of the loss functions described above.

[0181] For example, the training neural network may have architecture 410, 420, or 430, or a different point-tracking neural network architecture.

[0182] Furthermore, in the example in Figure 5, the teacher and student point-tracking neural networks 510 and 520 have the same architecture, or more generally, the student point-tracking neural network 520 includes all the same parameters as the teacher neural network 510, and optionally includes additional parameters. For example, the student 520 may include one or more additional convolutional residual layers as part of the backbone, each initialized to represent an identity transformation.

[0183] To take advantage of this, before training the student point-tracking neural network 520, the system uses the teacher point-tracking neural network 510 to initialize the parameters of the student point-tracking neural network 520, i.e., to the pre-trained network parameters 502.

[0184] Next, the system trains the student point-tracking neural network 520 over several training steps.

[0185] In each training step, the system receives one or more sets of training video sequences from a training dataset containing multiple training video sequences. As shown in the example in Figure 5, the videos in the training dataset may be real-world videos containing real-world video frames.

[0186] For each training video sequence, the system applies one or more transformations to the training video sequence to generate a transformed video sequence ("degraded real frames"). For example, one or more transformations may include spatial transformation, image degradation, or both.

[0187] As a concrete example, given an input video, the system can create a second view of the input video by resizing each frame to a smaller resolution (for example, changing linearly over time throughout the training process) and overlaying them onto a black background at random positions (similarly changing linearly over time). This can be calculated as an affine transformation Φ along the frame-by-frame axis with respect to the coordinates and applied to the pixels.

[0188] Optionally, the system can further degrade this view by applying random JPEG degradation to make the task more difficult before pasting it onto a black background. Since both actions result in the loss of texture information, the network needs to learn high-level, perhaps semantic cues (e.g., the tip of the cat's upper left ear) rather than low-level texture matching to accurately track the point.

[0189] Next, the system selects one or more training query points within the training video sequence. For example, the system can sample each training query point by uniformly and randomly sampling both the position and time step index within the training video sequence.

[0190] For each teacher query point, the system generates a teacher-point tracking output for the teacher query point by processing the video sequence using the teacher-point tracking neural network 510.

[0191] The system identifies the corresponding student query points in the converted video sequence, i.e., the student query points in the converted video sequence that correspond to the teacher query points.

[0192] Specifically, the system can use the teacher point tracking output to identify initial student query points, and then transform these initial student query points to match one or more transformations in order to generate student query points. That is, the system can also apply Φ to the student query coordinates to generate the final student query point coordinates.

[0193] For example, to identify the initial student query point, the system can randomly sample points from the trajectories generated for the teacher query point. This can be achieved by training the student neural network 520 to generate identical trajectories regardless of which point is used as the query, thereby forcing each trajectory to form an equivalent class.

[0194] The system generates student point-tracking outputs for student query points by processing the transformed video sequence using the student point-tracking neural network 520.

[0195] The system then trains a student point tracking neural network for each training video sequence and for each teacher query point, using a loss function that measures the difference between (i) the student point tracking output for the corresponding student query point and (ii) the teacher point tracking output for the teacher query point.

[0196] For example, the system can generate pseudo-labels from the teacher point tracking output for each training video sequence and for each teacher query point. The loss function can then measure the error between the pseudo-labels and the student point tracking output.

[0197] For example, when training using the loss described above, the system can generate pseudo-labels for a given video frame by (i) setting the ground truth position in a given video frame as the spatial prediction in the teacher point tracking output, (ii) setting the ground truth occlusion score to 1 only if the teacher occlusion estimate exceeds a threshold, e.g., zero, and optionally, (iii) setting the ground truth uncertainty to 1 only if the distance between the student prediction and the teacher prediction for a frame exceeds a threshold distance.

[0198] Next, the system can train a student point-tracking neural network with the loss function described above, but using pseudo-labels instead of ground truth outputs.

[0199] However, it should be noted that if the teacher does not track points correctly, student queries may be real-world points different from those tracked by the teacher, resulting in erroneous training signals. In some embodiments, to account for this, the system may decide whether to mask out teacher query points from the loss function by applying cycle consistency between the student point tracking output and the teacher point tracking output for each teacher query point, and then decide to mask out teacher query points that lack cycle consistency. "Mask out" teacher query points means removing or setting to zero all quantities that depend on teacher query points when calculating the loss function.

[0200] Furthermore, for points that are temporally closer to the student's query frame, the teacher's prediction may be less accurate than the student's prediction. In some embodiments, to account for this, the system may decide whether to mask out the prediction of the teacher query point for any given training video frame from the loss function by applying a proximity mask for each teacher query point. That is, the system may apply a proximity mask to mask out any training video frame that is closer to the student's query frame than to the teacher's query frame from the loss function. “Masking out” training video frames for teacher query points means removing or setting to zero all quantities that depend on the training video frame and the teacher query point when calculating the loss function.

[0201] Optionally, during unsupervised training, the system may continue training the point-tracking neural network through supervised learning, for example, to help avoid catastrophic forgetting. In these cases, during some or all of the training steps, the system also trains the student neural network through supervised learning on a set of labeled video sequences.

[0202] During some or all of the training steps, the system may update the teacher point-tracking neural network 510 using the student point-tracking neural network 520. For example, as shown in Figure 5, the system may update the parameters of the teacher point-tracking neural network 510 at a specified point in time during training to the exponential moving average (EMA) of the parameters of the student point-tracking neural network 520. In other words, rather than fixing the teacher point-tracking neural network 510 during the training process, the system continuously updates the teacher neural network 510 during training to improve the quality of the training process.

[0203] The point-tracking neural network described with reference to Figure 4 or Figure 5 can be used for any of the various purposes.

[0204] For example, a point-tracking neural network can be used to generate training data for the generative neural network described above.

[0205] As another example, predicted positions generated by a point-tracking neural network can be used to generate reward signals for training a robot or other agent through reinforcement learning. For instance, if the task performed by the agent requires moving a point in a scene from one location to another, the distance between the predicted position of the point and the target position in the last video frame of the sequence can be used to generate the reward.

[0206] As another example, predicted locations (and optionally occlusion scores) generated by a point-tracking neural network can be provided as additional input to a policy neural network to control a robot or other agent interacting with the environment. In this example, the query point could be a point of interest in the last frame of a video sequence, and predictions for preceding frames in the sequence can be provided as input to the policy neural network to provide signals about recent movements of, for example, an agent or other objects in the environment.

[0207] As another example, the predicted position (and optionally the occlusion score) generated by a point-tracking neural network may be provided, along with the corresponding video sequence, as input to a video understanding neural network, such as a motion classification neural network or a topic classification neural network, to provide the video understanding neural network with additional information about motion within the scene.

[0208] As another example, predicted positions (and optionally occlusion scores) generated by point-tracking neural networks can be used to enable imitation learning, i.e., imitation of motion rather than appearance.

[0209] As described above, the first generative neural network and / or the second generative neural network may be a spread neural network. Generally, a spread neural network is a neural network configured to process a spread input that includes (i) the current noisy data item (such as a noisy trajectory, noisy coordinates, noisy occlusion estimate, or noisy video frame) and, optionally, (ii) data that identifies a given time step, in order to produce a denoising output that defines an estimate of the noise component of the current noisy data item given the current time step. The noise component estimate is an estimate of the noise added to the original data item (such as a point trajectory or video frame) to generate the current noisy data item. The denoising output may be, for example, an estimate of the noise component or an estimate of the original data item.

[0210] After training, the system or other inference system uses the trained spread neural network to generate output data items (such as point trajectories or video frames) over multiple time steps by performing a despreader process that gradually denoises the initial data items until the final output data items are reached. Some or all of the values ​​in the initial data items are noisy values, i.e., sampled from a suitable noise distribution.

[0211] In other words, the initialized data item has the same dimensions as the final data item, but it has noisy values. For example, the system can initialize a data item, i.e., generate a first instance of the data item, by sampling each value within the data item from a corresponding noise distribution, e.g., a Gaussian distribution, or a different noise distribution. That is, the output data item will contain multiple values, the initial data item will contain the same number of values, and some or all of the values ​​will be sampled from the corresponding noise distribution.

[0212] As described above, a spread pixel neural network can generally have any suitable neural network architecture that enables the spread pixel neural network to map the input to the output image.

[0213] In some embodiments, the architecture of the spread pixel neural network may be similar to the U-Net neural network architecture described by O. Ronneberger et al. in “U-Net: Convolutional Networks for Biomedical Image Segmentation”, arXiv:1505.04597. In certain examples, the spread pixel neural network may be implemented as a convolutional neural network including a downward analysis pass and an upward synthesis pass, each pass including multiple neural network layers. The analysis pass may include multiple downsampling, e.g., convolutional layers, and the synthesis pass may include multiple upsampling, e.g., up-convolutional layers. In addition to convolutional layers, up and / or downsampling may be partially or completely implemented by interpolation. The neural network may include shortcut skips or residual connections between layers of equal resolution in the analysis pass and the synthesis pass. In some embodiments, at least one of the one or more sets of layers between the analysis pass and the synthesis pass includes a set of fully connected layers.

[0214] In some embodiments, the first or second generative neural network may include one or more self-attention layers. Generally, a self-attention layer may update each element of an embedding (e.g., input data) by applying an attention mechanism to the elements of the embedding, for example, the input embedding is used to determine a set of query vectors and key-value vector pairs (query-key-value attention), and the updated embedding contains a weighted sum of values ​​weighted by a query similarity function for each respective key. There are many different attention mechanisms that can be used. For example, the attention mechanism may be an inner product attention mechanism applied by applying the query vector to each key vector to determine the respective weight for each value vector, and then using the respective weights to combine the multiple value vectors to determine the attention layer output for each element of the input sequence.

[0215] The systems and methods described herein can be used to predict how a physical system or environment at a particular time will evolve over one or more subsequent time steps. That is, the input image may depict a scene including a physical environment, and the corresponding video generated from the input image may include video frames predicting the physical environment at one or more time steps after a particular time. For example, the input image may include one or more objects, and each video frame may be a prediction of the position and / or configuration of each object in the physical environment at the corresponding time step.

[0216] Video can be used as input to a control task. For example, video can be used for model predictive control of an agent such as a robot or machine agent. In such a case, the noisy trajectory may be conditioned on one or more actions that can be performed by the robot or machine agent, and the video is a prediction of the physical environment that would be obtained if the robot or machine agent performed one or more actions. Thus, video can be used by a control system to control a machine agent such as a robot to perform a particular task by processing the predicted video using the control system to generate control signals that control the machine agent to perform the task according to the video. The input image may be acquired by one or more sensors in the physical environment, for example, one or more sensors on the robot or machine agent. The input image may depict a part of the robot or machine agent, such as a gripper hand or other device for manipulating objects. As another example, video frames, or embeddings of video frames, may be provided as additional input to a policy neural network used to select actions to be performed by a robot or other agent interacting with the physical environment in order to perform a particular task, such as navigation in the physical environment, or identification or manipulation of one or more objects in the physical environment.

[0217] In some embodiments, the systems and methods described herein may be used for human pose estimation or prediction. For example, an input image may depict a person posing at a specific time, and each video frame may then be a prediction of the pose the person will take at a corresponding time after that specific time.

[0218] This specification uses the term “configured” in relation to systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action. Referring to a particular neural network, the neural network may be configured to perform a particular action by being trained to perform that action.

[0219] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, for example, one or more modules of computer program instructions, encoded in a tangible, non-temporary storage medium that is executed by or controls the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions may be encoded into artificially generated propagated signals, for example, mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0220] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0221] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required to, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or it can be located in one place or distributed across multiple locations and interconnected by a data communication network to run on multiple computers.

[0222] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0223] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0224] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0225] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0226] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0227] To provide user interaction, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device (for example, a smartphone running a messaging application) and then receiving a response message from the user.

[0228] Data processing devices for implementing machine learning models may also include dedicated hardware accelerator units for handling common computationally intensive parts of machine learning training or production, such as inference and workloads.

[0229] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0230] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application that enables a user to interact with embodiments of the subject matter described herein, or in a computing system that includes one or more such backend, middleware, or frontend components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0231] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0232] While this specification includes details of many specific embodiments, these should not be interpreted as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0233] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0234] Specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require to be performed in a specific or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0235] The nature of this disclosure may be as described in the following clauses.

[0236] Clause 1. A method performed by one or more computers, (i) a video sequence comprising multiple video frames, and (ii) data specifying a point-tracking neural network configured to receive a point-tracking input comprising a query point in one of the multiple video frames, and to generate a point-tracking output for each other video frame of the multiple video frames, comprising the location of the query point in the other video frame and an occlusion estimate of the query point in the other video frame. Training a student point-tracking neural network through unsupervised learning, wherein the student point-tracking neural network is configured to receive a student point-tracking input including (i) the video sequence including a plurality of video frames and (ii) the query point in one of the plurality of video frames, and for each other video frame of the plurality of video frames, to generate a student point-tracking output including the position of the query point in the other video frame and an occlusion estimate of the query point in the other video frame, and the training is performed in each of a plurality of training steps, Receiving one or more sets of training video sequences from a training dataset containing multiple training video sequences, For each training video sequence, To generate a converted video sequence, one or more transformations are applied to the training video sequence, Selecting one or more training query points within the aforementioned training video sequence, For each teacher query point, By processing the video sequence using the aforementioned training point tracking neural network, a training point tracking output of the training query points is generated. Identifying the corresponding student query points within the converted video sequence, The process includes generating a student point-tracking output for the student query points by processing the converted video sequence using the student point-tracking neural network, Training and A method comprising training the student point tracking neural network for each training video sequence and for each teacher query point, using a loss function that measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point.

[0237] Clause 2. The method according to Clause 1, wherein the training point tracking neural network is pre-trained before the training.

[0238] Clause 3. The method according to Clause 2, wherein the point-tracking neural network is pre-trained through supervised learning.

[0239] Clause 4. The method according to Clause 3, wherein the point-tracking neural network is pre-trained through supervised learning on a dataset of synthetic video sequences.

[0240] Clause 5. The teacher and student point-tracking neural networks have the same architecture as described in any one of Clauses 1 to 4.

[0241] Clause 6. The method according to clause 5, further comprising initializing the parameters of the student point tracking neural network using the teacher point tracking neural network before training the student point tracking neural network.

[0242] Clause 7. The training further comprises, in each of at least a subset of the training steps, updating the teacher point tracking neural network using the student point tracking neural network, the method according to any one of clauses 5 or 6.

[0243] Clause 8. Updating the teacher point tracking neural network using the student point tracking neural network comprises updating the parameters of the teacher point tracking neural network to be the exponential moving average (EMA) of the parameters of the student point tracking neural network, the method according to clause 7.

[0244] Clause 9. The method according to any one of clauses 1 to 8, wherein the one or more transformations include a spatial transformation.

[0245] Clause 10. The method according to any one of clauses 1 to 9, wherein the one or more transformations include image degradation.

[0246] Clause 11. Identifying corresponding student query points within the transformed video sequence comprises identifying initial student query points using the teacher point tracking output and transforming the initial student query points to match the one or more transformations to generate the student query points, the method according to any one of clauses 1 to 10.

[0247] Clause 12. The method according to any one of clauses 1 to 11, wherein the teacher point tracking output and the student point tracking output further comprise respective uncertainty scores for each predicted position.

[0248] Clause 13. For each training video sequence and for each teacher query point, training the student point tracking neural network using a loss function that measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point is: The method according to any one of the clauses 1 to 12, comprising generating a pseudo-label from the teacher point tracking output for each training video sequence and for each teacher query point, wherein the loss function measures the error between the pseudo-label and the student point tracking output.

[0249] Clause 14. For each training video sequence and for each teacher query point, training the student point tracking neural network using a loss function that measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point is: The method according to any one of the clauses 1 to 13, comprising determining whether to mask out the teacher query points from the loss function by applying cycle consistency between the student point tracking output and the teacher point tracking output.

[0250] Clause 15. For each training video sequence and for each teacher query point, training the student point tracking neural network using a loss function that measures the difference between the student point tracking output of the corresponding student query point and the teacher point tracking output of the teacher query point is: The method according to any one of the clauses 1 to 14, comprising determining whether to mask out the prediction of the teacher query points in any of the training videos from the loss function by applying a proximity mask.

[0251] Clause 16. The training shall, in each of the multiple training steps, The method according to any one of Clauses 1 to 15, further comprising training the student neural network through supervised learning on a set of labeled video sequences.

[0252] Clause 17. One or more computers and A system comprising: one or more storage devices communicably coupled to one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of each of the methods described in any one of the clauses 1 to 16.

[0253] Clause 18. One or more non-temporary computer storage media that, when executed by one or more computers, stores instructions causing the one or more computers to perform the operations of any one of the methods described in Clauses 1 to 16.

[0254] Clause 19. A method for generating a video, which is performed by one or more computers and animates input images over multiple time steps, Receiving the aforementioned input image, To process a first input derived from the input image using a first generative neural network in order to generate a point trajectory for each of one or more points in the input image, wherein each point trajectory includes, for each of the plurality of time steps in the video, the predicted spatial position of the corresponding point in the video frame at the time step in the video. A method comprising generating each of the video frames within the video frame that animates the input image using a second generative neural network, based on the input image and one or more point trajectories.

[0255] Clause 20. The method according to clause 19, wherein for each of the time steps, each point trajectory includes (i) the predicted spatial position of the corresponding point in the video frame at the time step, and (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded within the video frame at the time step.

[0256] Clause 21. The method according to clause 19 or clause 20, further comprising processing the input image using an image encoder neural network to generate an encoded representation of the input image, wherein the first input includes the encoded representation of the input image.

[0257] Clause 22. The method according to any one of clauses 19 to 21, wherein the first generation neural network is a diffusion neural network that generates each point trajectory from a corresponding noisy trajectory conditioned on the first input.

[0258] Clause 23. The method according to clause 22, wherein the first generation neural network is trained with a training dataset that includes (i) a plurality of video sequences, and (ii) for each of the video sequences, respective point trajectories for each of one or more points in a first frame within the video sequence.

[0259] Clause 24. The method according to clause 22 or clause 23, wherein the first generation neural network includes a two-dimensional convolutional neural network.

[0260] Clause 25. The method according to clause 24, wherein the two-dimensional convolutional neural network is a U-Net.

[0261] Clause 26. The method according to clause 24 or clause 25, wherein the two-dimensional convolutional neural network includes one or more self-attention layers.

[0262] Clause 27. Each corresponding noisy trajectory is defined in the manner described in any one of Clauses 22 to 26, including the concatenation of noisy coordinates and noisy occlusion estimates.

[0263] Clause 28. The method according to Clause 27, wherein the noisy coordinates are extended by position coding.

[0264] Clause 29. The method according to Clause 23, wherein for at least a subset of the video sequence, one or more of the point trajectories are generated by using a point-tracking neural network to process an input containing the corresponding point in the first frame.

[0265] Clause 30. The point-tracking neural network shall, for each video sequence in the subset and for the corresponding point in the first frame in the video, To generate query features for the corresponding points, Using the aforementioned query features, generate a cost volume that includes the respective cost maps for each of the multiple frames in the video sequence, The method according to Clause 29, configured to perform, for each of the plurality of frames, generate the initial position of the corresponding point in the frame and an initial occlusion estimate of the corresponding point in the frame using the cost map for the frame.

[0266] Article 31. The point-tracking neural network is: The method according to clause 30, further configured to generate the point locus for the corresponding point by refining the initial position, refining the initial occlusion estimate for the plurality of frames, or both, using an initial point locus including the initial position and the initial occlusion estimate for the plurality of frames.

[0267] Clause 32. Refining the initial position and initial occlusion estimates of the plurality of frames using an initial point trajectory including the initial position and initial occlusion estimates of the plurality of frames, in each of one or more refinement iterations, From the current point trajectory at the time of the refinement iteration, for each frame, a set of local score maps is generated that captures the similarity between the features near the predicted position in the current point trajectory within the frame and the query features, The method according to clause 31, comprising processing an input including the set of local score maps using an elaborated neural network to generate an update to the current point trajectory.

[0268] Clause 33. The method according to Clause 32, wherein the refined neural network is a depth-directed mixed neural network that propagates information between frames using depth-directed convolutional layers.

[0269] Clause 34. The method according to Clause 32 or Clause 33, wherein the input includes the set of local score maps, the query features, and the current point trajectory.

[0270] Clause 35. The method according to any one of Clauses 19 to 34, wherein the second generative neural network is a spread neural network that generates each frame from corresponding noisy frames conditioned on a conditioned input derived from the input image and the one or more point trajectories.

[0271] Clause 36. The method of Clause 35, wherein for each image, the conditional input includes a warped version of the input image warped according to one or more point trajectories to represent the image.

[0272] Clause 37. The method of Clause 36, wherein for each image, the conditional input includes a warped version of features extracted from the input image, which has been warped according to one or more point trajectories to represent the image.

[0273] Clause 38. The method according to Clause 36 or 37, wherein the spreading neural network generates each frame over a plurality of despread iterations, and for at least a subset of the frames, the input in a given despread iteration includes the current version of one or more preceding frames in the video at the time of the despread iteration.

[0274] Clause 39. One or more computers and A system comprising: one or more storage devices communicably coupled to one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of any one of the methods described in any one of the clauses 19 to 38.

[0275] Clause 40. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions causing the one or more computers to perform the operations of any one of the methods described in Clauses 19 to 38.

Claims

1. A method for generating a video that is executed by one or more computers and animates input images over multiple time steps, Receiving the aforementioned input image, To process a first input derived from the input image using a first generative neural network in order to generate a point trajectory for each of one or more points in the input image, wherein each point trajectory includes, for each of the plurality of time steps in the video, the predicted spatial position of the corresponding point in the video frame at the time step in the video. This includes generating each of the video frames in the video that animates the input image using a second generative neural network, based on the input image and one or more point trajectories, A method for each point trajectory, comprising: (i) the predicted spatial position of the corresponding point in the video frame at the time step; and (ii) an occlusion score that estimates the likelihood that the corresponding point will be occluded in the video frame at the time step.

2. The method according to claim 1, further comprising processing the input image using an image encoder neural network to generate an encoded representation of the input image, wherein the first input includes the encoded representation of the input image.

3. The method according to claim 1, wherein the first generative neural network is a diffuse neural network that generates point trajectories from corresponding noisy trajectories conditioned on the first input.

4. The method according to claim 3, wherein the first generative neural network is trained on a training dataset comprising (i) a plurality of video sequences and (ii) for each of the video sequences, the respective point trajectories for each of one or more points in a first frame within the video sequence.

5. The method according to claim 3, wherein the first generative neural network includes a two-dimensional convolutional neural network.

6. The method according to claim 5, wherein the two-dimensional convolutional neural network is U-Net.

7. The method according to claim 5, wherein the two-dimensional convolutional neural network includes one or more self-attention layers.

8. The method according to claim 3, wherein each corresponding noisy trajectory includes the concatenation of noisy coordinates and noisy occlusion estimates.

9. The method according to claim 8, wherein the noisy coordinates are extended by position coding.

10. The method according to claim 4, wherein, for at least a subset of the video sequence, one or more of the point trajectories are generated by using a point-tracking neural network to process an input containing the corresponding point in the first frame.

11. The point-tracking neural network performs the following for each video sequence within the subset and for the corresponding point within the first frame in the video: To generate query features for the corresponding points, Using the aforementioned query features, generate a cost volume that includes the respective cost map for each of the multiple frames in the video sequence, The method according to claim 10, configured to perform, for each of the plurality of frames, generate the initial position of the corresponding point in the frame and an initial occlusion estimate of the corresponding point in the frame using the cost map for the frame.

12. The aforementioned point-tracking neural network is The method according to claim 11, further configured to generate the point trajectory for the corresponding point by refining the initial position, refining the initial occlusion estimate for the plurality of frames, or both, using an initial point trajectory including the initial position and the initial occlusion estimate for the plurality of frames.

13. Using the initial point trajectory which includes the initial position and initial occlusion estimate of the plurality of frames, refining the initial position and initial occlusion estimate of the plurality of frames is done in each of one or more refinement iterations. From the current point trajectory at the time of the refinement iteration, for each frame, a set of local score maps is generated that captures the similarity between the features of the neighborhood of the predicted spatial position within the current point trajectory in the frame and the query features, The method according to claim 12, comprising processing an input including the set of local score maps using an elevated neural network to generate an update to the current point trajectory.

14. The method according to claim 13, wherein the refined neural network is a depth-direction mixed neural network that propagates information between frames using depth-direction convolutional layers.

15. The method according to claim 13, wherein the input includes the set of local score maps, the query features, and the current point trajectory.

16. The method according to claim 1, wherein the second generative neural network is a spread neural network that generates each frame from corresponding noisy frames conditioned on a conditioned input derived from the input image and one or more point trajectories.

17. The method according to claim 16, wherein for each image, the conditional input includes a warped version of the input image warped according to one or more point trajectories to represent the image.

18. The method according to claim 17, wherein for each image, the conditional input includes a warped version of features extracted from the input image, which has been warped according to one or more point trajectories to represent the image.

19. The method according to claim 17, wherein the spreading neural network generates each frame over a plurality of despread iterations, and for at least a subset of the frames, the input in a given despread iteration includes the current version of one or more preceding frames in the video at the time of the despread iteration.

20. The method according to claim 1, wherein the input image depicts a real-world environment at a specific time, and each of the video frames is a prediction of the real-world environment at a corresponding time after the specific time.

21. The method of claim 20, further comprising using the control system to control the robot or machine agent to perform a task by processing the video using the control system to generate one or more control signals that control the robot or machine agent to perform a specific task according to the video.

22. The method according to claim 1, wherein the input image depicts a person posing at a specific time, and each video frame is a prediction of the pose the person will take at a corresponding time after the specific time.

23. One or more computers, A system comprising: one or more storage devices communicably coupled to one or more computers, the one or more storage devices storing, when executed by the one or more computers, instructions causing the one or more computers to perform the method according to any one of claims 1 to 22.

24. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions causing the one or more computers to perform the method according to any one of claims 1 to 22.

Images