Generative image dynamics

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating, from an input image of a scene at a current time point, a motion output. The motion output incudes, for each of a plurality of pixels of the input image, motion data that characterizes predicted motion of the pixel over a plurality of future time points that are after the current time point. Thus, the system generates the dynamics of a video from a single, still image using a motion prediction neural network.

Description

[0001] GENERATIVE IMAGE DYNAMICS

[0002] BACKGROUND

[0003] This specification relates to generating images using neural networks.

[0004] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to or more other layers in the network, i.e., one or more other hidden layers, the output layer, or both. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0005] SUMMARY

[0006] This specification describes a system implemented as computer programs on one or more computers that performs a generative image dynamics task on an input image.

[0007] That is. the system generates, from the input image and using a generative neural network (also referred to as a ‘'motion prediction” neural network), predicted future dynamics of the pixels in the input image, i.e., generates an output that characterizes the predicted future motion of the pixels in the input image.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0009] Given a single image, the described system can use a neural network, e g., a diffusion neural network, to generate a prediction of per-pixel long-term motion of the pixels in the image. For example, the system can make the prediction in the frequency domain, e.g., in the Fourier domain. This representation can be converted into dense motion trajectories that span an entire video and, along with an image-based rendering engine, can be used for a number of downstream applications, such as turning still images into seamlessly looping dynamic videos, or allowing users to realistically interact with objects in real pictures.

[0010] In other words, as one example, by generating motion outputs as described in this specification, the described techniques can effectively generate a realistic looping video that captures realistic motion of one or more objects in a scene from a single, still image of the scene.

[0011] As another example, as one example, by generating motion outputs as described in this specification, the described techniques can effectively generate a realistic video that captures realistic motion of one or more objects in a scene in response to a force being applied at a specified point in the scene from a single, still image of the scene and an input that identifies the applied force.

[0012] More generally, the described techniques require only a single image to generate coherent long-term motion that realistically models the motion of real objects. For example, when the prediction is made in the frequency domain, the predictions capture the essence of pixel movements more efficiently in a lower-dimensional space, which leads to more coherent long-term generation and more fine-grained control over animations relative to other approaches.

[0013] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 is a diagram of an example image dynamics system.

[0016] FIG. 2 is an example of the operation of the image dynamics system.

[0017] FIG. 3 is a flow diagram of an example process for processing an input image.

[0018] FIG. 4 is an example of generating a future image.

[0019] FIG. 5 is an example of generating a motion output when the motion generation neural network is a diffusion neural network.

[0020] FIG. 6 shows an example of the performance of the described techniques.

[0021] FIG. 7 shows another example of the performance of the described techniques.

[0022] Like reference numbers and designations in the various drawings indicate like elements.

[0023] DETAILED DESCRIPTION

[0024] FIG. 1 is a diagram of an example image dynamics system 100. The image dynamics system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0025] The system 100 is a system that performs a generative image dynamics task on an input image 102.

[0026] That is. the system 100 generates, from the input image 102 and using a generative neural network 110 (also referred to as a “motion prediction” neural network), predicted future dynamics of the pixels in the input image 102, i.e., generates an output that characterizes the predicted future motion of the pixels in the input image 102.

[0027] Thus, the task is referred to as a ‘'generative” image dynamics task because the system 100 predicts the dynamics of the pixels in the image using a generative neural network, i.e., rather than determining the dynamics from actual changes between multiple images taken at different time points.

[0028] Generally, the system 100 performs the task using only the input image 102, and not any images that precede or follow the input image 102 in time. That is, the system 100 predicts the future motion of the pixels of the input image 102 using only the input image 102, so that the predictions represent plausible future motion of the scene depicted in the image 102. of the camera that captured the image 102, or both.

[0029] In other words, the system 100 receives an input image 102 of a scene at a current time point.

[0030] The system 100 then processes the input image 102 using the motion prediction neural network 110 to generate a motion output 120 that includes, for each of the plurality of pixels of the input image 102, motion data that characterizes predicted motion of the pixel over a plurality of future time points that are after the current time point.

[0031] In some implementations, for each pixel, the motion data for the pixel represents the predicted motion of the pixel over the plurality of future time points in the frequency domain, i.e., rather than in the time domain.

[0032] When needed, the system 100 can then apply a transformation to the motion data to transform the motion data into the time domain.

[0033] For example, the motion data can characterize, for each pixel, a motion spectrum over a plurality of output frequencies for the predicted motion of the pixel over the plurality of future time points.

[0034] As a particular example, the motion data for a given pixel can include, for each of the plurality of output frequencies, a set of coefficients of a motion basis for the output frequency. For example, the set of coefficients can be a set of complex Fourier coefficients for the x and v dimensions.

[0035] The motion prediction neural network 110 can generally be any appropriate generative neural network having an architecture that allows the neural network 110 to process the image 102, i.e., to process the intensity values of the pixels of the image 102, to the motion output 120. As a particular example, in some implementations, the motion prediction neural network 110 is a diffusion neural network.

[0036] The diffusion neural network can be any appropriate diffusion neural network that is configured to receive an input that includes a representation of the input image 102.

[0037] In particular, the diffusion neural network is configured to receive a denoising input that includes a representation of the motion output 120 and a representation of the input image 102 and to process the input to generate a denoising output for the representation of the motion output 120.

[0038] In some implementations, the diffusion neural network performs a diffusion process in pixel space, so that the motion outputs (“representations"’) operated on and generated by the diffusion neural network have values that directly correspond to time or frequency domain values. In these examples, the denoising output can be an estimate of the noise component of the motion output, e.g., an estimate of the noise that has been added to a final motion output to arrive at the motion output.

[0039] In some other implementations, the diffusion neural network performs a diffusion process in latent space, e.g., in a latent space that is lower-dimensional than the output space. That is, the motion outputs (“representations”) operated on and generated by the diffusion neural network are latent representations of a motion output and the values are learned, latent values rather than output values within the motion outputs. In these examples, the denoising output is an estimate of the noise that has been added to a latent representation of the final motion output in the latent space to arrive at the input latent representation in the latent space.

[0040] In these implementations, during training, the trained diffusion neural network can be associated with (i) an encoder neural network to encode ground truth motion outputs into the latent space and, during training and after training when generating new motion outputs, (ii) a decoder neural network that receives an input that includes a latent representation of a motion output and decodes the latent representation to reconstruct the motion output.

[0041] In some implementations, the system 100 can use the motion output 120 to generate a respective future image 130 at each of one or more of the future time points.

[0042] As a particular example, the system 100 can generate a video that includes a respective video frame 130 (image) at each of some or all of the future time points.

[0043] In some implementations, the system 100 can use the motion output 120 to generate a respective future image 130 at each of one or more additional time points that are each between a respective pair of the future time points by interpolating the motion output 120 to include the one or more additional time points and then using the interpolated motion output to generate the images at the one or more additional time points.

[0044] Thus, the system 100 can leverage the motion output 120 to turn the input image 102 into a video, e.g., a seamlessly looping dynamic video.

[0045] As another example, the system 100 can receive an input interacting with the image, e.g., a user input dragging and releasing a point within the input image 102 while the input image 102 is presented to a user, and then generate a video that reflects the user interaction with the input image 102 from a force specified by the interaction input and the motion output 120, thereby allowing users to realistically interact with objects in real pictures.

[0046] FIG. 2 is an example 200 of the operation of the image dynamics system 100.

[0047] As shown in the example 200, the system 100 receives an input image 202. i.e., an image of a flower.

[0048] The system 100 processes the input image 202 using the motion prediction neural network 110 to generate a motion output 220 for the input image 202 that predicts the motion of the pixels of the input image 202 in the future.

[0049] As shown in the example 200. the motion output 220 is a spectral volume that includes, for each of a plurality of output frequencies, a set of coefficients of a motion basis for the output frequency. More specifically, in the example 200, the set of coefficients of the motion basis are a set of complex Fourier coefficients for the x and y dimensions.

[0050] Once the system has generated the motion output 220. the system 100 can use the motion output 220 for any of a variety of purposes.

[0051] For example, the system 100 can generate a looping video 230 that reflects the motion characterized in the motion output 220.

[0052] As another example, the system can receive a user input 240 interacting with the flower, i.e., pulling the flower down and to the left as shown in the example 200, and then generate an interactive dynamics video 250 that reflects the motion of the object after the interaction.

[0053] In the example of FIG. 2, the videos 230 and 250 are visualized as a space-time X-t slice along an input scanline 260 within the input image 202.

[0054] FIG. 3 is a flow diagram of an example process 300 for processing an input image. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image dynamics system, e.g., the image dynamics system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. The system receives an input image of a scene at a current time point (step 302).

[0055] The system processes the input image, i.e., processes the intensity values of the pixels of the input image, using a motion prediction neural network to generate a motion output (step 304). Generally, the motion prediction neural network processes only the input image at the current time point and not any other images of the scene at any other time points.

[0056] The motion output includes, for each of the plurality of pixels, motion data that characterizes predicted motion of the pixel over a plurality of future time points that are after the current time point.

[0057] Generally, for each pixel, the motion data characterizes, for each of the plurality of future time points, a predicted displacement between coordinates of the pixel in a future image at the future time point relative to coordinates of the pixel in the input image.

[0058] For example, in some implementations, the motion data can directly specify the predicted displacement at each future time point, i.e., can be a time-domain motion texture that is a representation in the time domain.

[0059] More specifically, the time domain motion of the pixels in the input image over a time period that includes future time points 1 through T can be represented as a sequence of timevarying 2D displacement maps T = {Ft11 = 1, ... , T], where the 2D displacement vector Ft(p) at each pixel coordinate p from the input image IQdefines the position of that pixel at a future time t. In other words, the motion trajectory of a pixel across the T future time points is F(p) = [fy(p) |t = 1 T}.

[0060] However, in these implementations, the time-domain motion texture needs to scale with the number of future time points for which the prediction is being made: i.e., generating T output frames would require predicting T displacement fields. For videos with a large number of frames T, this makes generating the motion output excessively computationally expensive.

[0061] To avoid predicting such a large output representation for long videos, some animation techniques either generate video frames autoregressively or predict each future output frame independently. However, neither strategy ensures the long-term temporal consistency of generated videos.

[0062] Instead, the system can generate, for each pixel, motion data that represents the predicted motion of the pixel over the plurality of future time points in a frequency domain. Because many natural motions can be described as a superposition of a small number of harmonic oscillators represented with different frequencies, amplitude and phases, the resulting motions are quasi-periodic, which allows the system to accurately model them in the frequency domain.

[0063] More specifically, the system can generate, for each pixel, motion data that characterizes a motion spectrum over a plurality of output frequencies for the predicted motion of the pixel over the plurality of future time points. In this example, the motion data can include, for each of the plurality of output frequencies, a set of coefficients of a motion basis for the output frequency. As a particular example, the set of coefficients can be the set of complex Fourier coefficients for the x and v dimensions.

[0064] This results in a motion output that is a frequency space representation of motion in a video referred to as a spectral volume, where a spectral volume is the temporal Fourier transform of per-pixel trajectories extracted from a video.

[0065] By representing the future motion in the frequency domain rather than in the time domain, the system can efficiently represent predicted motion even for very large values of T. In particular, the size of the representation is defined by K. the number of frequencies modeled, which is independent from and generally much smaller than the number of future time points T.

[0066] In particular, the resulting motion output is a 4K-channel 2D motion spectrum map, e.g., with a respective channel for each coefficient for each of the K frequencies, rather than a 2 / '-channel 2D time domain displacement map (or equivalently, a sequence of T 2D displacement maps that include a respective 2D displacement vector at each pixel coordinate).

[0067] In particular, the spectral volume is defined by, for each pixel coordinate p, a spectral T volume representation S(p) = {Syk(p) \k = 0,1, — 1}.

[0068] The spectral volume representation of the pixel location and the motion trajectory of the pixel location are related by the Fast Fourier transform (FFT):

[0069] S(p) = FFT(lF(p)).

[0070] To make the number frequencies represented in the spectral volume independent of T, the system can determine the K frequencies that are modeled in any of a variety of ways. For example, many natural oscillation motions are composed primarily of low -frequency components. Given this, the system can select the first K Fourier coefficients as the K modeled frequencies. As a particular example, the system can use the first K = 16 Fourier coefficients to realistically reproduce the original natural motion in a range of real videos and scenes. An example of generating the motion output when the motion prediction neural network is a diffusion neural network is described below with reference to FIG. 5.

[0071] In some implementations, e.g., when the motion prediction neural network is a diffusion neural network, the output of the motion prediction neural network can be, for each pixel and for each of the plurality of output frequencies, a set of adaptively normalized coefficients of the motion basis for the output frequency’. The system can then generate the final motion output by de-normalizing the coefficients of the motion basis that are in the output of the motion prediction neural network.

[0072] In particular, motion textures can have particular distribution characteristics across frequencies. For example, the amplitude of the spectral volumes spans a range of 0 to 100 and decays approximately exponentially with increasing frequency. As diffusion models require that the absolute values of the output are between -1 and 1 for stable training and denoising, the system normalizes the coefficients of spectral volumes extracted from real videos before using them for training.

[0073] However, if the system scales the magnitudes of these coefficients to [0,1] based on the image dimensions, nearly all the coefficients at higher frequencies will end up close to zero. Models trained on such data can produce inaccurate motions, since during inference, even small prediction errors will cause large relative errors after denormalization.

[0074] To address this issue, the system can perform frequency adaptive normalization. As a particular example, the system can independently normalize the Fourier coefficients at each frequency based on statistics computed from the training set used to train the diffusion neural network. For example, for each individual frequency, the system can compute a specified percentile, e.g., the 90th, 95th, or 99thpercentile of Fourier coefficient magnitudes over all input samples in the training set and use that value as a per-frequency scaling factor. The system can then apply a power transformation, e.g., a square root transformation, to each scaled Fourier coefficient to pull it away from extreme values.

[0075] As a specific example, the final coefficient values of a spectral volume can satisfy:

[0076] S'^(p) = sign (S7.) J | ^— |, where sfis the scaling factor for frequency / .

[0077] Optionally, once the system has generated the motion output, the system can generate, from the motion output, a respective future image of the scene at each of one more of the future time points (step 306). In particular, the system can generate a respective motion trajectory for each of the plurality of pixels from the motion output that specifies, for each of the one or more future time points, coordinates of the pixel in the respective future image at the time point (step 308).

[0078] For example, when the motion output is in the frequency domain, the system can apply a transform from the frequency domain to a time domain to the motion data for the pixel. More specifically, the system can apply an inverse temporal Fast Fourier Transform (FFT) to the motion data for each of the pixels to generate the motion trajectory for the pixel.

[0079] The system can then generate the respective future image(s) using the motion trajectories (step 310). One example of generating a future image at time t using the motion trajectories for the pixels is described below with reference to FIG. 4.

[0080] FIG. 4 shows an example 400 of generating a future image It450 at a future time point t from an input image Io402.

[0081] At a high level, the system can generate features 420 of the input image 402.

[0082] In the example 400, the system generates the features by processing the input image using a feature extraction neural network 410. As shown in the example 400, the features 420 are multi-scale features, with features of each scale being generated from the outputs of a different one of the layers of the feature extraction neural network 410. The feature extraction neural network 410 can generally have any appropriate architecture, e.g., can be a multi-scale convolutional neural network.

[0083] For any given time point / , the system then generates the future image 450 using the features and the displacement map Ft430 for time t.

[0084] In particular, the system splats the features 420 of the input image 402 to the future time point to generate splatted features using the displacement map Ft430, i.e., the predicted motion field for the future time point that specifies the coordinates of the pixels in the respective future image at the future time point.

[0085] As one example, the system can perform softmax splatting 440 on the features of the input image using the predicted motion field. In some cases, the system incorporates motion weights 442 into the softmax splatting 440.

[0086] The motion weights 442 can be, e.g., determined from the predicted motion textures. As a particular example, the weight for each pixel location can be the average magnitude of the predicted motion texture for the pixel location across the time steps. In other words, the system can assume large motions correspond to moving foreground objects, and small or zero motions correspond to background. The system can use motion-derived weights e.g., instead of learnable ones, because in the single view case, learnable weights may not be effective for addressing disocclusion ambiguities.

[0087] In particular, the system can determine motion weights 442 for the pixels from the motion trajectories for the pixels and then perform softmax splatting 440 on the features using the motion weights 442 and the predicted motion field 430. That, at each scale, the system can warp the feature map at the scale using the motion weights 442 and the predicted motion field 430. Generally, for each individual feature map at scale j, the system can resize and scale the predicted motion field according to the resolution. When performing the warping, the system can use the motion weights as a proxy for depth to determine the contributing weight of each source pixel mapped to its destination location.

[0088] Performing softmax splatting is described in more detail in Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 5437-5446, 2020.

[0089] The system then generates the future image 450 at the future time point from the splatted features for the future time point.

[0090] In the example 400, the system generates the future image for the future time point using an image synthesis neural network 460 that is conditioned on the splatted features for the future time point. The image synthesis neural network 460 can generally have any appropriate architecture, e.g., can be a multi-scale convolutional neural network.

[0091] As described above, in some implementations, the system also receives a force input specifying a force applied to an object depicted in the input image. For example, the force input can be generated based on a user interaction interacting with an object in the input image or otherwise specified by a user.

[0092] The system can train the neural networks 410 and 450 in any appropriate way. For example, the system can jointly train the feature extractor and synthesis networks with start and target frames (Io, It) randomly sampled from real videos, using the estimated flow field from Io to warp encoded features from Io, and supervising predicted frames generated by the synthesis neural network against It with a perceptual loss, e.g., a VGG perceptual loss or other appropriate perceptual loss.

[0093] In these implementations, to generate the respective motion trajectory' for each of the plurality of pixels, the system generates the respective motion trajectory' from the force input and the motion data for the pixel. For example, the system can generate the motion displacement field for a given pixel location at a future time point as a weighted sum of motion spectrum coefficients from the motion output modulated by the state of complex modal coordinates at the future time point. As a particular example, the system can simulate the state of the modal coordinates via an explicit Euler method applied to the equations of motion for a decoupled mass-spring-damper system represented in modal space.

[0094] Thus, the motion displacement field can be generated as: where q^ t) are modal coordinates for frequency fj at future time step, e g., determined by simulating the state of the modal coordinates via an explicit Euler method applied to the equations of motion for a decoupled mass-spring-damper system represented in modal space.

[0095] The system can then generate the future image(s) from the motion trajectory as described above.

[0096] FIG. 5 shows an example 500 of generating the motion output 520 when the motion generation neural network is a diffusion neural network 510 from an input image 506.

[0097] In general, when the motion generation neural network is a diffusion neural network, the system first initializes a representation z 512 of the motion output. For example, the system can sample 502 at least some of the values in the representation from a noise distribution, e.g.. a Gaussian noise distribution. In some cases, the system samples all of the values from the noise distribution.

[0098] The system then performs iterative denoising 504 by updating the representation 512 of the motion output at each of a plurality7of update iterations using the diffusion neural network 510. where the representation at iteration n is denoted as z". That is, the system gradually “denoises” the representation 512 across the update iterations using the diffusion neural network to generate a final representation of the motion output 520.

[0099] At each iteration, the system generates a denoising output. For example, the denoising output can be an estimate of a noise component of the representation of the motion output. That is, the denoising output can be an estimate of the noise that has been added to a final (unknown) representation of the motion output to generate the current representation. As another example, the denoising output can be an estimate of the final representation of the motion output. As part of this, the system processes a diffusion input that includes (i) the representation 512 of the motion output and (ii) a representation of the input image 506 using the diffusion neural network 510 to generate a first denoising output.

[0100] The first denoising output is the same type of estimate as the denoising output, i.e., estimates the same quantity as the denoising output.

[0101] In some implementations, the system uses the first denoising output as the denoising output.

[0102] In some other implementations, the system processes one or more additional diffusion inputs that each include the representation 512 of the motion output 520 to generate one or more additional denoising outputs and then combines the additional denoising output(s) with the first denoising output.

[0103] For example, the additional denoising output(s) can include a classifier-free guidance denoising output that is generated by processing a diffusion input that includes the representation 512 of the motion output 520 but does not include the representation of the input image 506. Thus, in this example, the final denoising output at iteration n can satisfy: en= (1 + w)eeznn c) — weeznn; 0) where <ty(z"; n; c) is the first denoising output and e0(zn; n; 0) is the classifier-free guidance denoising output and c is the representation of the input image, and w is a guidance w eight that is received as input by the system.

[0104] As described above, in some implementations, the system uses the motion output 520 to generate a looping video from the input image 506.

[0105] Generating looping videos may require a video that loops seamlessly, where there is no discontinuity between the start and end of the video. However, it is hard to find a large collection of seamlessly looping videos for training the diffusion neural network 510.

[0106] Instead, the system can use the diffusion neural network 510, trained on regular nonlooping video clips, to produce seamless looping video by including a motion self-guidance denoising output as one of the additional denoising outputs. Including the motion selfguidance incorporates an explicit looping constraint into the denoising process. In particular, the motion self-guidance output enforces each pixel’s position and velocity at the start and end frames of the video to be as similar as possible.

[0107] For example, the additional denoising output(s) can include both the motion selfguidance output and the classifier free-guidance output. In these cases, the final denoising output at iteration n can satisfy: where Lg = 11 F? — F" 111+ 11 VFp — VF” 111and / z is a weight value that is received as input by the system and F is the predicted displacement field at time t and update iteration n.

[0108] Thus, as can be seen in these examples, the diffusion neural network is also conditioned on an identifier for the update iteration.

[0109] The system then updates the representation 512 using the denoising output. For example, the system can determine an initial estimate of the final representation, i.e., the unnoised final representation of the motion output, from the diffusion output and then apply an appropriate diffusion sampler, e.g., the DDPM (Denoising Diffusion Probabilistic Model) sampler, the DDIM (Denoising Diffusion Implicit Model) sampler or another appropriate sampler, to the initial estimate to generate an updated representation.

[0110] Optionally, at the last update iteration, the system can refrain from using the diffusion sampler and can instead use the initial estimate of the final representation as the updated representation.

[0111] After the last update iteration, the system generates the motion output 520 using the final representation 512.

[0112] For example, in the example 500, the diffusion neural network 510 performs the denoising process in the latent space. Therefore, the representation 512 of the motion output 520 is a latent representation of the motion output. To generate the motion output 520, the system processes the final representation using a decoder neural network D 540 to generate the motion output F 520. When frequency adaptive normalization is used, the system denormalizes the output generated by the decoder neural network I) 540 to generate the motion output S 520.

[0113] Prior to using the diffusion neural network 510 to generate new motion outputs for new' input images, the system or another training system trains the diffusion neural network 510 on a set of training videos.

[0114] To train the diffusion neural network 510 on a given training video, the system determines a ground truth representation of the ground truth motion output for the given training video. When frequency adaptive normalization is used, the system first performs frequency adaptive normalization on the initial motion output extracted from the given training video to generate the final ground truth motion output.

[0115] For example, when the diffusion neural network operates in the latent space as shown in the example 500, the system can process the ground truth spectral volume for the given training video using an encoder neural network 550 to generate a representation of the spectral volume.

[0116] The system can then sample an updating iteration n and noise enand then combine the representation of the ground truth spectral volume with the sampled noise in accordance with a noise level for the sampled updating iteration to generate a noisy representation zn. For example, the system can determine a weighted sum of the representation of the ground truth spectral volume and the sampled noise, with the weight for each being defined by the sampled updating iteration, so that different updating iterations are associated with different weights. The weights for the updating iterations can be received as input by the system.

[0117] The system can then process the noisy representation znand a representation c of the first image in the training video using the diffusion neural network to generate a diffusion output eeznn; c).

[0118] The system can then train the diffusion neural network 510 on a loss that measures an error between the sampled noise and the diffusion output, e g., a mean squared error (MSE) loss:

[0119] | |en- e9zn; n c) 112.

[0120] The diffusion neural network 510 can generally have any appropriate architecture that allows the neural network to map a representation of the motion output and a representation of the image to the denoising output.

[0121] For example, the diffusion neural network 510 can be a convolutional neural network, e.g., a U-Net, that has multiple convolutional layer blocks. In some of these cases, the diffusion neural network 510 can include one or more cross-attention layer blocks interspersed among the convolutional layer blocks. As will be described below, some or all of the cross-attention blocks can be conditioned on a representation of the input image.

[0122] As another example, the diffusion neural network 510 can be a Transformer neural network that processes the diffusion input through a set of self-attention layers to generate the diffusion output. In these examples, the neural network 510 can also include one or more attention blocks that are conditioned on a representation of the input image.

[0123] In the specific example 500 of FIG. 5, the diffusion neural network 510 includes one or more frequency attention layers 534. More specifically, the diffusion neural network includes multiple blocks 530, with each block including a 2D spatial layer 532 that operates on each channel independently followed by a frequency attention layer 534 that attends across the frequency axis. More specifically, in some cases, directly training the diffusion neural network 510 to predict a spectral volume S' with K frequency bands by outputting a tensor of 4K channels can yield over-smoothed, inaccurate outputs, e.g., because directly learning to output a tensor with a relatively large number of channels is difficult.

[0124] To account for this, the system can implement a frequency-coordinated denoising strategy during training.

[0125] In particular, given an input image, the system first trains the diffusion neural network including only the spatial layers 532 and not the frequency attention layers 534 to predict a single 4-channel frequency slice of spectral volume, e.g., by injecting an extra frequency embedding into the diffusion neural network to identify the frequency to which the slice corresponds.

[0126] The system can then freeze the parameters of the diffusion neural network and introduce the frequency attention layers 534 interleaved with the 2D spatial layers 532 across the K frequency bands, and fine-tune the neural network to generate the full spectral volume.

[0127] Specifically, for a batch size B, the 2D spatial layers 532 treat the corresponding B K noisy latent features of channel size C as independent samples with shape R(B K)xCxHxW.

[0128] The attention layer 534 then interprets these as consecutive features spanning the frequency axis, and the neural network reshapes the latent features from previous 2D spatial layers to RBxKxCxHxW before feeding them to the attention layers. In other words, when this strategy is employed, the frequency attention layers 534 are fine-tuned to coordinate all frequency slices so as to produce coherent spectral volumes.

[0129] To generate the representation of the input image 506, the diffusion neural network 510 can include an image encoder neural network 560, e.g., that is pre-trained or that has been trained jointly with the remainder of the diffusion neural network 510, that processes the input image 506 to generate the representation. For example, the image encoder neural network 560 can be a convolutional neural network or a vision Transformer neural network.

[0130] FIG. 6 shows an example 600 of the performance of the described techniques. In particular, the example 600 shows first chart that shows the performance of various techniques in terms of Frechet Inception Distance (FID). In the first chart, the performance of the described techniques is marked with a line 602. The second chart shows the performance of various techniques in terms of Dynamic Texture Frechet Video Distance (DT-FVD), which measures the FVD with a I3D model trained on the Dynamic Database, a dataset consisting primarily of natural motion textures. In the second chart, the performance of the described techniques is marked with a line 604. As can be seen from the first and second charts, videos generated using the described techniques continue to exhibit lower, i.e., better FID and DT-FVD scores across the entirety of the video, i.e.. across all frame indices. This shows that videos generated by techniques described in the specification are more temporally consistent compared to the other techniques, and do not suffer from drift or degradation over time. For example, this can be attributed at least in part to the fact that the system generates an intermediate motion texture, e.g.. in the frequency domain, prior to using the motion texture to generate the video instead of directly attempting to generate the video from the single image.

[0131] FIG. 7 shows another example 700 of the performance of the described techniques. In particular, the example 700 shows the performance of the described techniques (‘‘ours'’) relative to other techniques in terms of both image synthesis quality and video synthesis quality. Image synthesis quality is measured in FID, sliding window FID (FID sw) with a window size of 30 frames to measure degradation across time within a video, and Kernel Inception Distance (KID). Video sy nthesis quality is measured in terms of FVD and DT- FVD.

[0132] As can be seen from the example 700. the described techniques reliably achieve the best image and video synthesis quality across the evaluated techniques.

[0133] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0134] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory' device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0135] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0136] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0137] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0138] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0139] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0140] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory7can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0141] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory7, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0142] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0143] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.

[0144] Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow framework or a Jax framework.

[0145] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g.. a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0146] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.

[0147] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0148] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0149] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving an input image of a scene at a current time point, wherein the input image comprises a plurality of pixels; and processing the input image using a motion prediction neural network to generate a motion output that comprises, for each of the plurality of pixels, motion data that characterizes predicted motion of the pixel over a plurality of future time points that are after the current time point.

2. The method of claim 1, wherein the motion prediction neural network processes only the input image at the current time point and not any other images of the scene at any other time points.

3. The method of any preceding claim, wherein, for each pixel, the motion data characterizes, for each of the plurality of future time points, a predicted displacement between coordinates of the pixel in a future image at the future time point relative to coordinates of the pixel in the input image.

4. The method of any preceding claim, wherein, for each pixel, the motion data represents the predicted motion of the pixel over the plurality of future time points in a frequency domain.

5. The method of claim 4, wherein, for each pixel, the motion data characterizes a motion spectrum over a plurality of output frequencies for the predicted motion of the pixel over the plurality of future time points.

6. The method of claim 5, wherein, for each pixel, the motion data comprises, for each of the plurality of output frequencies, a set of coefficients of a motion basis for the output frequency.

7. The method of claim 6, wherein the set of coefficients are a set of complex Fourier coefficients for the x and y dimensions.

8. The method of any preceding claim, wherein the motion prediction neural network comprises a diffusion neural network, and wherein processing the input image using a motion prediction neural network to generate a motion output comprises: initializing a representation of the motion output; and updating the representation of the motion output at each of a plurality of update iterations, the updating comprising: generating a denoising output, the generating comprising processing a diffusion input comprising (i) the representation of the motion output and (ii) a representation of the input image using the diffusion neural network to generate a first denoising output; and updating the representation using the denoising output.

9. The method of claim 8, wherein initializing a representation of the motion output comprises sampling at least some of the values in the representation from a noise distribution.

10. The method of claim 8 or 9, wherein the denoising output is an estimate of a noise component of the representation of the motion output.

11. The method of any one of claims 7-10, wherein the motion prediction neural network comprises a decoder neural network, and wherein the representation of the motion output is in a latent space and wherein generating the motion output further comprises: processing the representation of the motion output after the last update iteration using the decoder neural network to generate the motion output.

12. The method of any one of claims 7-11 , wherein the diffusion neural network comprises one or more frequency attention layers.

13. The method of any preceding claim, when dependent on claim 6, wherein the output of the motion prediction neural network comprises, for each pixel and for each of the plurality of output frequencies, a set of adaptively normalized coefficients of the motion basis for the output frequency.

14. The method of any preceding claim, further comprising: generating, from the motion output, a respective future image of the scene at each of one more of the future time points.

15. The method of claim 14, wherein generating, from the motion output, a respective future image of the scene at each of one more of the future time points comprises: generating a respective motion trajectory for each of the plurality of pixels from the motion output that specifies, for each of the one or more future time points, coordinates of the pixel in the respective future image at the time point.

16. The method of claim 15 when dependent on claim 4, wherein generating a respective motion trajectory for each of the plurality of pixels comprises applying a transform from the frequency domain to a time domain to the motion data for the pixel.

17. The method of claim 16. wherein the transform is an inverse temporal Fast Fourier Transform.

18. The method of any one of claims 15-17, further comprising: receiving a force input specifying a force applied to an object depicted in the input image, wherein generating a respective motion trajectory for each of the plurality of pixels comprises generating the respective motion trajectory7from the force input and the motion data for the pixel.

19. The method of any one of claims 15-18, wherein generating the future image at each of the one or more future time points comprises: generating features of the input image; for each future time point: splatting the features of the input image to the future time point to generate splatted features using a predicted motion field for the future time point that specifies the coordinates of the plurality of pixels in the respective future image at the future time point; and generating the future image at the future time point from the splatted features for the future time point.

20. The method of claim 19, wherein generating the future image at the future time point from the splatted features for the future time point comprises generating the future image using an image synthesis neural network that is conditioned on the splatted features for the future time point.

21. The method of claim 19 or claim 20. wherein generating features of the input image comprises processing the input image using a feature extraction neural network.

22. The method of any one of claims 19-21, wherein the features are multi-scale features.

23. The method of any one of claims 19-22, wherein splatting the features of the input image to the future time point to generate splatted features comprises: determining motion weights for the pixels from the motion trajectories for the pixels; and performing softmax splatting on the features using the motion weights and the predicted motion field.

24. The method of any preceding claim, when dependent on claim 8, wherein generating the denoising output comprises generating the denoising output from at least the first denoising output and a motion self-guidance denoising output.

25. The method of any preceding claim, when dependent on claim 8, wherein generating the denoising output comprises generating the denoising output from at least the first denoising output and a classifier-free guidance denoising output.

26. The method of any preceding claim, when dependent on claim 8, wherein generating the denoising output comprises generating the denoising output from at least the first denoising output, a motion self-guidance denoising output, and a classifier-free guidance denoising output.

27. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1 -26.

28. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-26.

Images