Generative image dynamics

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a motion output from an input image of a scene at a current point in time. For each of a plurality of pixels of the input image, the motion output includes motion data characterizing predicted motion of the pixel during a plurality of future time points after the current time point. Thus, the system generates dynamics of the video from a single still image using a motion prediction neural network.

Description

Background Technology

[0001] This specification relates to the use of neural networks to generate images.

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to one or more other layers in the network (i.e., one or more other hidden layers, the output layer, or both). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers that performs generative image dynamics tasks on an input image.

[0004] In other words, the system uses a generative neural network (also known as a "motion prediction" neural network) to generate the predicted future dynamics of the pixels in the input image based on the input image; that is, it generates an output that represents the predicted future motion of the pixels in the input image.

[0005] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0006] Given a single image, the described system can use a neural network, such as a diffuse neural network, to generate predictions of pixel-by-pixel long-term motion in the image. For example, the system can make predictions in the frequency domain, or, for instance, in the Fourier domain. This representation can be transformed into dense motion trajectories across the entire video and, together with an image-based rendering engine, can be used in various downstream applications, such as transforming still images into seamless looping dynamic videos or allowing users to realistically interact with objects in real-world images.

[0007] In other words, as an example, by generating motion output as described in this specification, the described technique can effectively generate realistic looping videos that capture realistic motion of one or more objects in a scene from a single still image of the scene.

[0008] As another example, by generating motion output as described in this specification, the described technique can efficiently generate realistic video from a single still image of a scene and an input of an identified applied force, which captures realistic motion of one or more objects in the scene in response to the application of a force at a specific point in the scene.

[0009] More generally, the described techniques require only a single image to generate coherent long-term motion that realistically models the motion of real objects. For example, when predictions are made in the frequency domain, these predictions capture the essence of pixel movement more efficiently in a low-dimensional space, resulting in more coherent long-term generation and finer-grained control over animation compared to other methods.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the following figures and description. Other features, aspects, and advantages of this subject matter will become apparent from the description, figures, and claims. Attached Figure Description

[0011] Figure 1 This is a diagram of an example image dynamic system.

[0012] Figure 2 This is an example of the operation of a dynamic image system.

[0013] Figure 3 This is a flowchart of an example process for processing an input image.

[0014] Figure 4 This is an example of generating future images.

[0015] Figure 5 This is an example of generating motion output when the motion generation neural network is a diffusion neural network.

[0016] Figure 6 An example of the performance of the described technology is shown.

[0017] Figure 7 Another example of the performance of the described technology is shown.

[0018] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0019] Figure 1 This is a diagram of an example image dynamics system 100. Image dynamics system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0020] System 100 is a system for performing generative image dynamics tasks on input image 102.

[0021] In other words, system 100 uses a generative neural network 110 (also known as a "motion prediction" neural network) to generate the predicted future dynamics of the pixels in the input image 102 based on the input image 102, that is, to generate an output representing the predicted future motion of the pixels in the input image 102.

[0022] Therefore, this task is called a “generative” image dynamics task because system 100 uses a generative neural network to predict the dynamics of pixels in an image, that is, to determine the dynamics not from the actual changes between multiple images taken at different points in time.

[0023] Generally, system 100 uses only the input image 102 to perform the task, without using any images that are temporally before or after the input image 102. That is, system 100 uses only the input image 102 to predict the future motion of the pixels in the input image 102, such that the prediction represents a plausible future motion of the scene depicted in the image 102, a plausible future motion of the camera that captured the image 102, or both.

[0024] In other words, system 100 receives input image 102 of the scene at the current point in time.

[0025] Then, system 100 uses motion prediction neural network 110 to process input image 102 to generate motion output 120, which includes motion data for each of a plurality of pixels in input image 102, the motion data representing the predicted motion of the pixel during a plurality of future time points after the current time point.

[0026] In some implementations, for each pixel, the pixel's motion data represents the predicted motion of that pixel over multiple future time points in the frequency domain—that is, not the time domain.

[0027] When needed, system 100 can then apply a transformation to the motion data to convert the motion data to the time domain.

[0028] For example, for each pixel, motion data can characterize the motion spectrum at multiple output frequencies for the predicted motion of that pixel over multiple future time points.

[0029] As a specific example, motion data for a given pixel may include a set of coefficients for a motion basis for each of a plurality of output frequencies. For example, the set of coefficients could be... x and y A set of complex Fourier coefficients of dimension.

[0030] The motion prediction neural network 110 can typically be any suitable generative neural network with an architecture that allows the neural network 110 to process image 102—that is, to process the intensity values ​​of the pixels in image 102—to generate output motion 120.

[0031] As a specific example, in some implementations, the motion prediction neural network 110 is a diffuse neural network.

[0032] The diffusion neural network can be any suitable diffusion neural network configured to receive input including a representation of the input image 102.

[0033] Specifically, the diffusion neural network is configured to receive a denoised input including a representation of motion output 120 and a representation of input image 102, and process the input to generate a denoised output representing motion output 120.

[0034] In some implementations, the diffusion neural network performs a diffusion process in pixel space such that the motion output (“representation”) operated and generated by the diffusion neural network has a value that directly corresponds to the time-domain or frequency-domain value. In these examples, the denoised output may be an estimate of the noise component of the motion output, for example, that has been added to the final motion output to obtain an estimate of the noise in the motion output.

[0035] In some other implementations, the diffusion neural network performs the diffusion process in the latent space—for example, in a latent space with a lower dimension than the output space. That is, the motion output (“representation”) operated and generated by the diffusion neural network is a latent representation of the motion output, and these values ​​are the learned latent values, not the output values ​​in the motion output. In these examples, the denoised output is an estimate of the noise that has been added to the latent representation of the final motion output in the latent space to obtain the input latent representation in the latent space.

[0036] In these implementations, during training, the trained diffusion neural network can be associated with (i) an encoder neural network that encodes ground truth motion outputs into a latent space, and during and after training when new motion outputs are generated, it can be associated with (ii) a decoder neural network that receives inputs including a latent representation of the motion output and decodes the latent representation to reconstruct the motion output.

[0037] In some implementations, system 100 may use motion output 120 to generate a corresponding future image 130 for each of one or more future time points.

[0038] As a specific example, system 100 can generate a video that includes corresponding video frames 130 (images) at some or all of the future time points.

[0039] In some implementations, system 100 may use motion output 120 to generate a corresponding future image 130 at each of one or more additional time points between a corresponding pair of future time points by interpolating motion output 120 to include one or more additional time points, and then using the interpolated motion output to generate an image at one or more additional time points.

[0040] Therefore, system 100 can use motion output 120 to convert input image 102 into video, such as seamless looping dynamic video.

[0041] As another example, system 100 may receive input that interacts with an image, such as user input that drags and releases a point within input image 102 when it is presented to the user, and then generate a video reflecting the user's interaction with input image 102 from force and motion output 120 specified by the interactive input, thereby allowing the user to realistically interact with objects in a real picture.

[0042] Figure 2 This is an example 200 of the operation of the image dynamics system 100.

[0043] As shown in Example 200, system 100 receives input image 202, that is, an image of a flower.

[0044] System 100 uses motion prediction neural network 110 to process input image 202 to generate motion output 220 for input image 202, which predicts the future motion of pixels in input image 202.

[0045] As shown in Example 200, the motion output 220 is a spectral volume, which, for each of a plurality of output frequencies, may include a set of coefficients of the motion basis for the output frequencies. More specifically, in Example 200, this set of coefficients of the motion basis is a set of complex Fourier coefficients in the x and y dimensions.

[0046] Once the system generates motion output 220, the system 100 can use the motion output 220 for any of its various purposes.

[0047] For example, system 100 can generate loop video 230, which reflects the motion represented in motion output 220.

[0048] As another example, the system can receive user input 240 that interacts with the flower, i.e., as shown in Example 200, by pulling the flower down and to the left, and then generate an interactive dynamic video 250 that reflects the movement of the object after the interaction.

[0049] exist Figure 2In the example, videos 230 and 250 are visualized as spatiotemporal sequences along input scan line 260 within input image 202. X - t slice.

[0050] Figure 3 This is a flowchart of an example process 300 for processing an input image. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, an image dynamics system appropriately programmed according to this specification—e.g. Figure 1 The image dynamic system 100 depicted in the figure is an executable process 300.

[0051] The system receives the input image of the scene at the current time point (step 302).

[0052] The system uses a motion prediction neural network to process the input image—that is, to process the intensity values ​​of the pixels in the input image—to generate a motion output (step 304). Generally, the motion prediction neural network only processes the input image at the current time point, and does not process any other images of the scene at any other time point.

[0053] For each of the multiple pixels, the motion output includes motion data that characterizes the predicted motion of the pixel over a plurality of future time points following the current time point.

[0054] Generally, for each pixel, for each of a plurality of future time points, motion data represents the predicted displacement of the pixel’s coordinates in the future image at the future time point relative to the pixel’s coordinates in the input image.

[0055] For example, in some implementations, motion data can directly specify the predicted displacement at each future time point, i.e., it can be a temporal motion texture as a representation in the time domain.

[0056] More specifically, the pixels in the input image include future time points 1 to... T The time-domain motion during a given time period can be represented as a time-varying 2D displacement map sequence. , where from the input image 2D displacement vector at each pixel coordinate p Define the pixel in the future time t The location of the pixel. In other words, the pixel spans... T The trajectory of movement at a future point in time is .

[0057] However, in these implementations, the temporal motion texture needs to be scaled with the number of future time points the needle predicts: that is, the generation TEach output frame will need to be predicted T Individual displacement fields. For a large number of frames. T The video makes generating motion output very computationally expensive.

[0058] To avoid predicting such a large output representation for long videos, some animation techniques generate video frames autoregressively or predict each future output frame independently. However, neither of these strategies can guarantee the long-term temporal consistency of the generated video.

[0059] Alternatively, the system can generate motion data for each pixel, representing the predicted motion of that pixel over multiple future time points in the frequency domain. Because many natural motions can be described as a superposition of a small number of harmonic oscillators with different frequencies, amplitudes, and phases, the resulting motions are quasi-periodic, allowing the system to accurately model them in the frequency domain.

[0060] More specifically, for each pixel, the system can generate motion data representing the predicted motion spectrum of that pixel at multiple output frequencies over multiple future time points. In this example, the motion data for each of the multiple output frequencies may include a set of coefficients for a motion basis at the output frequencies. As a specific example, this set of coefficients could be... x and y The set of complex Fourier coefficients of the dimension.

[0061] This produces a motion output that is a frequency-space representation of the motion in the video, called the spectral volume, where the spectral volume is the time Fourier transform of the pixel-by-pixel trajectory extracted from the video.

[0062] By representing future motion in the frequency domain rather than the time domain, even for very large... T The system can also efficiently represent the predicted motion. Specifically, the size of this representation is determined by... K By definition, K is the number of frequencies being modeled, and it is related to the number of future time points. T It is irrelevant and is usually much smaller than T.

[0063] Specifically, the resulting motion output is a 4K channel 2D motion spectrum, for example, K Each coefficient of each of the frequencies has a corresponding channel, not 2. T Channel 2D time-domain displacement map (or equivalently, T A sequence of 2D displacement maps, each consisting of a corresponding 2D displacement vector at each pixel coordinate.

[0064] Specifically, for each pixel coordinate p, the spectral volume is represented by the spectral volume. To define.

[0065] The spectral volume representation of a pixel location and the motion trajectory of that pixel location are correlated via a Fast Fourier Transform (FFT): .

[0066] To ensure that the number of frequencies represented in the spectral volume is independent of T The system can determine which model is being used in any of the various ways. K A frequency. For example, many natural oscillating motions consist primarily of low-frequency components. Therefore, the system can select the preceding frequency. K The Fourier coefficients are used as K The system models the frequencies. As a specific example, the system can use the first K=16 Fourier coefficients to realistically reproduce the original natural motion in a series of real videos and scenes.

[0067] The following text is for reference only. Figure 5 An example of generating motion output is described when the motion prediction neural network is a diffusion neural network.

[0068] In some implementations, for example, when the motion prediction neural network is a diffuse neural network, the output of the motion prediction neural network, for each pixel and for each of a plurality of output frequencies, can be a set of adaptively normalized coefficients of the motion basis for that output frequency. The system can then generate the final motion output by denormalizing the coefficients of the motion basis in the output of the motion prediction neural network.

[0069] Specifically, motion textures can have specific distribution characteristics across frequencies. For example, the amplitude of the spectral volume spans from 0 to 100 and decays approximately exponentially with increasing frequency. Since the diffusion model requires the absolute value of the output to be between -1 and 1 for stable training and denoising, the system normalizes the coefficients of the spectral volume extracted from real video and then uses them for training.

[0070] However, if the system scales the magnitude of these coefficients to [0,1] based on the image size, almost all coefficients at higher frequencies will eventually approach zero. Models trained on such data can produce inaccurate motion because even small prediction errors during inference can lead to large relative errors after denormalization.

[0071] To address this issue, the system can perform frequency-adaptive normalization. As a specific example, the system can independently normalize the Fourier coefficients at each frequency based on statistics computed from the training set used to train the diffuse neural network. For instance, for each individual frequency, the system can compute a specified percentile, such as the 90th, 95th, or 99th percentile of the Fourier coefficient amplitude across all input samples in the training set, and use this value as a scaling factor for each frequency. The system can then apply a power transform, such as a square root transform, to each scaled Fourier coefficient to move it away from extrema.

[0072] As a specific example, the final coefficient values ​​of the spectral volume can satisfy: , in It is for frequency j The scaling factor.

[0073] Optionally, once the system has generated motion output, the system can generate a corresponding future image of the scene at each of one or more future time points in the future based on the motion output (step 306).

[0074] Specifically, the system can generate a corresponding motion trajectory for each of the multiple pixels based on the motion output, and the motion trajectory specifies the coordinates of the pixel in the corresponding future image at one or more future time points (step 308).

[0075] For example, when the motion output is in the frequency domain, the system can apply a frequency-to-time domain transformation to the pixel's motion data. More specifically, the system can apply an inverse-time fast Fourier transform (FFT) to the motion data of each pixel to generate the pixel's motion trajectory.

[0076] The system can then use this motion trajectory to generate the corresponding future image (step 310). See below for reference. Figure 4 Describes the generation of motion trajectories using pixels in time. t An example of a future image.

[0077] Figure 4 This shows the input image 402 Generate Future Time Points t Future images Example 400 is 450.

[0078] At a high level, the system can generate features 420 of the input image 402.

[0079] In Example 400, the system generates features by processing the input image using a feature extraction neural network 410. As shown in Example 400, feature 420 is a multi-scale feature, where each scale of feature is generated from the output of a different layer in the feature extraction neural network 410. The feature extraction neural network 410 can generally have any suitable architecture; for example, it can be a multi-scale convolutional neural network.

[0080] For any given point in time t The system then uses features and time-specific features. t Displacement diagram 430 is used to generate a future image 450.

[0081] Specifically, the system splatters features 420 of the input image 402 onto future time points to use a displacement map. 430—that is, the predicted motion field at a future time point—is used to generate the splashed features, which specifies the coordinates of the pixels in the corresponding future image at the future time point.

[0082] As an example, the system can perform a softmax splash 440 on the features of the input image using the predicted motion field. In some cases, the system incorporates motion weights 442 into the softmax splash 440.

[0083] For example, motion weights 442 can be determined from the predicted motion texture. As a specific example, the weight at each pixel location could be the average magnitude of the predicted motion texture across time steps at that pixel location. In other words, the system could assume that large motion corresponds to a moving foreground object, and small motion or zero motion corresponds to the background. The system could use motion-derived weights, for example, instead of learnable weights, because in a single-view scenario, learnable weights might be ineffective in resolving occlusion disambiguation.

[0084] Specifically, the system determines the motion weights 442 of pixels based on their motion trajectories, and then performs a softmax splash 440 on the features using the motion weights 442 and the predicted motion field 430. That is, at each scale, the system can warp the feature map at that scale using the motion weights 442 and the predicted motion field 430. Generally, for each individual feature map at scale j, the system can adjust the size and scale the predicted motion field according to the resolution. During the warping, the system can use the motion weights as a depth indicator to determine the contribution weight of each source pixel mapped to the target location of each source pixel.

[0085] The implementation of softmax splatting is described in more detail in the following literature: Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 5437–5446, 2020.

[0086] Then, the system generates a future image 450 at the future time point based on the splashing characteristics at the future time point.

[0087] In Example 400, the system uses an image synthesis neural network 460 conditioned on the projected features at future time points to generate future images of those future time points. The image synthesis neural network 460 can generally have any suitable architecture; for example, it could be a multi-scale convolutional neural network.

[0088] As described above, in some implementations, the system also receives force input, which specifies the force applied to an object depicted in the input image. For example, this force input may be generated based on user interaction with the object in the input image, or may be otherwise specified by the user.

[0089] This system can train neural networks 410 and 450 in any suitable manner. For example, using a start frame and a target frame (I0, I...) randomly sampled from a real video. t The estimated flow field from I0 is used to distort the encoded features from I0 and a perceptual loss—such as VGG perceptual loss or other appropriate perceptual loss—is applied to I0. t The system supervises the predicted frames generated by the synthetic neural network, and can jointly train the feature extractor and the synthetic network.

[0090] In these implementations, in order to generate the corresponding motion trajectory for each of the multiple pixels, the system generates the corresponding motion trajectory based on the force input and the pixel's motion data.

[0091] For example, the system can generate a motion displacement field at a given pixel location at a future time point as a weighted sum of motion spectrum coefficients from the motion output modulated by the complex modal coordinate state at that future time point. As a specific example, the system can simulate the state of the modal coordinates via an explicit Euler method applied to the equations of motion of a decoupled mass-spring-damper system represented in modal space.

[0092] Therefore, the motion displacement field can be generated as follows:

[0093] in It is a frequency-specific step in the future time. The modal coordinates are determined, for example, by simulating the state of the modal coordinates via an explicit Euler method applied to the equations of motion of the decoupled mass-spring-damper system represented in modal space.

[0094] The system can then generate future images based on the motion trajectory, as described above.

[0095] Figure 5 An example 500 is shown where a motion generation neural network 510 generates a motion output 520 from an input image 506.

[0096] Generally, when the motion generative neural network is a diffuse neural network, the system first represents the motion output. z 512 Initialization. For example, the system may sample at least some of the values ​​in the representation from a noise distribution, such as a Gaussian noise distribution. 502. In some cases, the system samples all the values ​​from the noise distribution.

[0097] Then, the system performs iterative denoising 504 by updating the representation 512 of the motion output at each update iteration in multiple update iterations using a diffuse neural network 510, where the iterations n The representation of the location is shown as z n In other words, the system uses a diffuse neural network to progressively "denoise" representation 512 across update iterations to generate the final representation of motion output 520.

[0098] At each iteration, the system generates a denoised output. For example, this denoised output could be an estimate of the noise components in a representation of the motion output. That is, the denoised output could be an estimate of the noise that has been added to the final (unknown) representation of the motion output to generate the current representation. As another example, the denoised output could be an estimate of the final representation of the motion output.

[0099] As part of this, the system uses a diffusion neural network 510 to process the diffusion input to generate a first denoised output, the diffusion input including (i) a representation 512 of the motion output and (ii) a representation of the input image 506.

[0100] The first denoised output is an estimate of the same type as the denoised output, that is, the estimate is the same amount as the denoised output.

[0101] In some implementations, the system uses the first denoised output as the denoised output.

[0102] In some other implementations, the system processes one or more additional diffuse inputs, each including a representation 512 of motion output 520, to generate one or more additional denoised outputs, and then combines the additional denoised outputs with the first denoised output.

[0103] For example, the additional denoising output may include a classifier-free guided denoising output, which is generated by processing a diffusion input that includes a representation 512 of the motion output 520 but does not include a representation of the input image 506. Therefore, in this example, iteration... n The final denoising output at the point can satisfy:

[0104] in It is the first noise-reduced output, and It is a classifier-free guided denoising output, and c It is a representation of the input image, and w It is the guiding weight that the system receives as input.

[0105] As described above, in some implementations, the system uses motion output 520 to generate looping video based on input image 506.

[0106] Generating looping videos requires videos that loop seamlessly, with no discontinuity between the beginning and end of the video. However, it is difficult to find a large collection of seamlessly looping videos to train a diffusion neural network.510.

[0107] Alternatively, the system can use a diffuse neural network 510 trained on regular non-looping video clips to produce seamless looping video by including the motion-guided denoising output as one of the additional denoising outputs. Including motion guidance incorporates explicit loop constraints into the denoising process. Specifically, the motion-guided output forces the location and velocity of each pixel at the start and end frames of the video to be as similar as possible.

[0108] For example, the additional denoising output may include both motion-guided output and classifier-unguided output. In these cases, iteration... n The final denoising output at the point can satisfy:

[0109] in ,and The system receives the weight values ​​as input, and It is time t and updates and iterations n The predicted displacement field at that location.

[0110] Therefore, it can be seen from these examples that the diffusion neural network is also conditioned on the identifier of the update iteration.

[0111] The system then uses the denoised output to update the representation 512. For example, the system can determine an initial estimate of the final representation—that is, the noise-free final representation of the motion output—from the diffusion output, and then apply an appropriate diffusion sampler—such as a DDPM (Denoising Diffusion Probability Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another appropriate sampler—to the initial estimate to generate the updated representation.

[0112] Alternatively, in the final update iteration, the system may avoid using the diffusion sampler and instead use the initial estimate of the final representation as the updated representation.

[0113] After the last update iteration, the system uses the final representation 512 to generate motion output 520.

[0114] For example, in Example 500, the diffuse neural network 510 performs a denoising process in the latent space. Therefore, the representation 512 of the motion output 520 is a latent representation of the motion output. To generate the motion output 520, the system uses a decoder neural network. D 540 is used to process the final representation to generate motion output. S 520. When using frequency adaptive normalization, the system adapts to the decoder neural network. D The output generated by 540 is denormalized to generate motion output. S 520.

[0115] Before using the diffusion neural network 510 to generate new motion outputs from new input images, the system or another training system trains the diffusion neural network 510 on a set of training videos.

[0116] To train a diffuse neural network 510 on a given training video, the system determines a ground truth representation of the ground truth motion output for that given training video. When using frequency adaptive normalization, the system first performs frequency adaptive normalization on the initial motion output extracted from the given training video to generate the final ground truth motion output.

[0117] For example, when the diffusion neural network operates in the latent space, as shown in Example 500, the system can use the encoder neural network 550 to process the ground truth spectral volume of a given training video to generate a representation of the spectral volume.

[0118] Then, the system can update and iterate. n and noise Sampling is performed, and then the representation of the true spectral volume is combined with the sampled noise based on the sampled, iteratively updated noise level to generate a noisy representation. For example, the system can determine a weighted sum of the true spectral volume and the sampled noise, where the weights of each are defined by the sampled update iterations, such that different update iterations are associated with different weights. The system can receive the weights used for the update iterations as input.

[0119] The system can then use a diffuse neural network to process noisy representations. and the representation of the first image in the training video c To generate diffusion output .

[0120] The system can then train a diffusion neural network 510 on a loss that measures the error between the sampled noise and the diffusion output, such as a mean squared error (MSE) loss.

[0121] The diffusion neural network 510 can typically have any suitable architecture that allows the neural network to map the representation of the motion output and the representation of the image to the denoised output.

[0122] For example, the diffuse neural network 510 can be a convolutional neural network, such as U-Net with multiple convolutional layer blocks. In some of these cases, the diffuse neural network 510 may include one or more cross-attention blocks scattered between the convolutional layer blocks. As will be described below, some or all of the cross-attention blocks may be conditioned on a representation of the input image.

[0123] As another example, the diffusion neural network 510 can be a Transformer neural network that processes the diffusion input through a set of self-attention layers to generate a diffusion output. In these examples, the neural network 510 may also include one or more attention blocks conditioned on a representation of the input image.

[0124] exist Figure 5 In a specific example 500, the diffuse neural network 510 includes one or more frequency attention layers 534. More specifically, the diffuse neural network includes multiple blocks 530, each block including a 2D spatial layer 532 that operates independently on each channel, followed by a frequency attention layer 534 that focuses across the frequency axis.

[0125] More specifically, in some cases, a diffusion neural network 510 is directly trained by outputting a tensor with 4K channels to predict [the desired outcome]. K Spectral volume of each frequency band SThis may result in overly smoothed, inaccurate outputs, for example, because it is difficult to directly learn tensors with a relatively large number of channels.

[0126] To address this issue, the system can implement a frequency-coordinated denoising strategy during training.

[0127] Specifically, given an input image, the system first trains a diffuse neural network that includes only a spatial layer 532 and no frequency attention layer 534 to predict the slice, for example, by injecting additional frequency embeddings into the diffuse neural network to identify the frequency corresponding to a single 4-channel frequency slice of the spectral volume.

[0128] The system can then freeze the parameters of the diffusion neural network, introduce frequency attention layers 534 interleaved with the 2D spatial layer 532 across K frequency bands, and fine-tune the neural network to generate a complete spectral volume.

[0129] Specifically, for batch size B, the 2D spatial layer 532 treats the noisy latent features corresponding to B·K with channel size C as independent samples of shape R(B·K)×C×H×W.

[0130] The attention layer 534 then interprets these as continuous features across the frequency axis, and the neural network reshapes the latent features from the previous 2D spatial layers into RB×K×C×H×W, which is then fed into the attention layer. In other words, when this strategy is employed, the frequency attention layer 534 is fine-tuned to reconcile all frequency slices in order to produce a coherent spectral volume.

[0131] To generate a representation of the input image 506, the diffusion neural network 510 may include an image encoder neural network 560, which, for example, is pre-trained or jointly trained with the rest of the diffusion neural network 510, to process the input image 506 to generate a representation. For example, the image encoder neural network 560 may be a convolutional neural network or a visual Transformer neural network.

[0132] Figure 6Example 600 illustrates the performance of the described techniques. Specifically, Example 600 shows a first graph illustrating the performance of various techniques in terms of Frechet Inception Distance (FID). In the first graph, the performance of the described techniques is marked by line 602. A second graph illustrates the performance of various techniques in terms of Dynamic Texture Frechet Video Distance (DT-FVD), which utilizes an I3D model trained on a dynamic database—a dataset primarily composed of natural motion textures—to measure FVD. In the second graph, the performance of the described techniques is marked by line 604. As can be seen from the first and second graphs, videos generated using the described techniques continue to exhibit lower—i.e., better—FID and DT-FVD scores across the entire video—that is, across all frame indices. This indicates that, compared to other techniques, videos generated by the techniques described in this specification are more temporally consistent and do not experience drift or degradation over time. For example, this can be attributed at least in part to the system generating intermediate motion textures (e.g., in the frequency domain) before using motion textures to generate video, rather than directly attempting to generate video from a single image.

[0133] Figure 7 Another example 700 illustrating the performance of the described technique is shown. Specifically, example 700 illustrates the performance of the described technique (“our”) relative to other techniques in both image synthesis quality and video synthesis quality. Image synthesis quality is measured by: FID; a sliding window FID (FID_sw) with a window size of 30 frames, used to measure temporal degradation within the video; and Kernel Inception Distance (KID). Video synthesis quality is measured in terms of FVD and DT-FVD.

[0134] As can be seen from Example 700, the described technique reliably achieves optimal image and video synthesis quality across all evaluated techniques.

[0135] This specification uses the term "configured" in conjunction with system and computer program components. For configuring one or more computer systems to perform a specific operation or action, it means that software, firmware, hardware, or a combination thereof are installed on the system that causes the system to perform that operation or action during operation. For one or more computer programs configured to perform a specific operation or action, it means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0136] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagated signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0137] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0138] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0139] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0140] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0141] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0142] A computer suitable for executing computer programs may be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to one or more mass storage devices or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0143] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROMs and DVD-ROMs.

[0144] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving responsive messages from the user in response.

[0145] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0146] Machine learning models can be implemented and deployed using machine learning frameworks (such as TensorFlow or Jax).

[0147] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0148] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0149] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0150] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to be performed to achieve the desired result. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequential sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Receive an input image of the scene at the current time point, wherein the input image includes multiple pixels; as well as The input image is processed using a motion prediction neural network to generate a motion output, which includes motion data for each of the plurality of pixels, the motion data representing the predicted motion of that pixel over a plurality of future time points after the current time point.

2. The method of claim 1, wherein the motion prediction neural network processes only the input image at the current time point and does not process any other images of the scene at any other time point.

3. The method of any of the preceding claims, wherein for each pixel, for each of the plurality of future time points, the motion data characterizes the predicted displacement of the pixel's coordinates in the future image at that future time point relative to the pixel's coordinates in the input image.

4. The method of any of the preceding claims, wherein for each pixel, the motion data represents, in the frequency domain, the predicted motion of that pixel during the plurality of future time points.

5. The method of claim 4, wherein for each pixel, the motion data characterizes a motion spectrum at a plurality of output frequencies for the predicted motion of that pixel during the plurality of future time points.

6. The method of claim 5, wherein for each pixel, the motion data includes a set of coefficients for a motion basis for each of the plurality of output frequencies.

7. The method of claim 6, wherein the set of coefficients is a set of complex Fourier coefficients in the x and y dimensions.

8. The method of any preceding claim, wherein the motion prediction neural network comprises a diffuse neural network, and wherein using the motion prediction neural network to process the input image to generate a motion output comprises: Initialize the representation of the motion output; as well as At each of the multiple update iterations, the representation of the motion output is updated, the update comprising: Generate a denoised output, the generation comprising using the diffusion neural network to process a diffusion input to generate a first denoised output, the diffusion input comprising (i) the representation of the motion output and (ii) the representation of the input image; as well as The denoised output is used to update the representation.

9. The method of claim 8, wherein initializing the representation of the motion output includes sampling at least some values ​​of the representation from a noise distribution.

10. The method of claim 8 or 9, wherein the denoised output is an estimate of the noise component represented by the motion output.

11. The method of any one of claims 7 to 10, wherein the motion prediction neural network comprises a decoder neural network, and wherein the representation of the motion output is in a latent space, and wherein generating the motion output further comprises: The representation of the motion output is processed using the decoder neural network after the last update iteration to generate the motion output.

12. The method of any one of claims 7 to 11, wherein the diffuse neural network comprises one or more frequency attention layers.

13. The method of any preceding claim when dependent on claim 6, wherein the output of the motion prediction neural network, for each pixel and for each of the plurality of output frequencies, includes a set of adaptive normalization coefficients for the motion basis at that output frequency.

14. The method of any of the preceding claims, further comprising: The motion output generates a corresponding future image of the scene at each of one or more future time points in the future.

15. The method of claim 14, wherein generating a corresponding future image of the scene at each of one or more of the future time points from the motion output comprises: Based on the motion output, a corresponding motion trajectory is generated for each of the plurality of pixels, wherein the corresponding motion trajectory specifies the coordinates of the pixel in the corresponding future image at each of the one or more future time points.

16. The method of claim 15, which is dependent on claim 4, wherein generating a corresponding motion trajectory for each of the plurality of pixels includes applying a transformation from the frequency domain to the time domain to the motion data of that pixel.

17. The method of claim 16, wherein the transform is an inverse-time fast Fourier transform.

18. The method of any one of claims 15 to 17, further comprising: The system receives a force input, which specifies a force applied to an object depicted in the input image, wherein generating a corresponding motion trajectory for each of the plurality of pixels includes generating the corresponding motion trajectory from the force input and motion data of that pixel.

19. The method of any one of claims 15 to 18, wherein generating the future image at each of the one or more future time points comprises: Generate features of the input image; For each future point in time: The features of the input image are splashed onto the future time point to generate the splashed features using the predicted motion field of the future time point, the predicted motion field specifying the coordinates of the plurality of pixels in the corresponding future image at the future time point; as well as Generate a future image of that future time point based on the characteristics of the splash at that future time point.

20. The method of claim 19, wherein generating a future image at the future time point based on the splashed features at the future time point comprises generating the future image using an image synthesis neural network conditioned on the splashed features at the future time point.

21. The method of claim 19 or claim 20, wherein generating the features of the input image includes processing the input image using a feature extraction neural network.

22. The method of any one of claims 19 to 21, wherein the feature is a multi-scale feature.

23. The method of any one of claims 19 to 22, wherein splashing the features of the input image onto the future time point to generate the splashed features comprises: The motion weight of the pixel is determined based on the motion trajectory of the pixel; as well as The features are then subjected to softmax splashing using the motion weights and the predicted motion field.

24. The method of any preceding claim when dependent on claim 8, wherein generating the denoised output comprises generating the denoised output from at least the first denoised output and the motion-guided denoised output.

25. The method of any preceding claim when dependent on claim 8, wherein generating the denoised output comprises generating the denoised output from at least the first denoised output and the classifier-free guided denoised output.

26. The method of any preceding claim when dependent on claim 8, wherein generating the denoised output comprises generating the denoised output from at least the first denoised output, the motion-guided denoised output, and the classifier-free guided denoised output.

27. A system comprising: One or more computers; as well as One or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method as described in any one of claims 1 to 26.

28. A computer-readable storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method as claimed in any one of claims 1 to 26.

Images