Generating time series using diffusion TRANSFORMER neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a time series of input-conditioned data elements. One of the methods comprises: obtaining the input, where the input comprises a noisy input comprising a plurality of potential representations of an output time series; updating each potential representation using a potential denoising neural network; and generating the output time series of data elements by processing the updated potential representation using a decoder neural network.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 657,463, filed June 7, 2024. The disclosure of the prior application is considered part of and is hereby incorporated by reference in its entirety into the disclosure of this application. BACKGROUND

[0003] This specification relates to processing input using a neural network to generate a time series.

[0004] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Besides the output layer, some neural networks include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY

[0005] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates a time series conditioned on input. The time series includes a respective data element at each of one or more time points. As an example, the time series can be a video. Each data element of the video can be a video frame (or image frame), such as a single still image, that represents a moving visual image or content when played in rapid succession with other frames.

[0006] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0007] The systems described in this specification generate high-resolution time series of various data elements, such as videos, audio, or climate data. Generating time series with high spatial and temporal resolution can present a greater amount of information coherently compared to generating time series with lower spatial or temporal resolution. The temporal resolution of a video is referred to as the frame rate, or the number of frames per second.

[0008] Generating high-resolution time series while maintaining quality, temporal coherence, and consistency with the input is a challenge. For example, some conventional systems for generating videos are unable to generate videos with a spatial resolution or a temporal resolution that is higher than the spatial or temporal resolution of the videos at the time of training.

[0009] The systems described in this specification can generate highly detailed and temporally consistent videos with high spatial resolution, high temporal resolution, or both. As an example, the systems described in this specification can generate videos with a resolution of 768 x 1280 and 24 frames per second.

[0010] Some conventional systems for generating high-resolution and high-frame-rate videos require cascading diffusion models, which generally include a series of diffusion models linked together to generate images, typically with increasing resolution after starting from a base diffusion model running at low resolution. However, due to the low-resolution generation of the first stage, these conventional systems have limited ability to generate highly detailed and rich videos at the final resolution. Additionally, there is generally a mismatch between the training and testing distributions of the cascading diffusion models, resulting in low-quality video generation at inference.

[0011] The systems described in this specification can generate high-quality videos with high resolution, high frame rate, or both using a single latent diffusion model. By representing video frames as latent representations and then further downsampling to a compressed latent space, the systems can leverage the high degree of correlation between video frames, particularly in videos with high resolution and high frame rate, resulting in more efficient video generation compared to systems with lower compression.

[0012] For example, the systems described in this specification encode a sequence of tokens representing patches of each latent representation into an encoded representation that includes a sequence of input tokens, and process the encoded representation to generate a merged sequence of output tokens. The merged sequence of output tokens includes fewer tokens than the number of encoded representations. The systems perform most of the processing on the sequence of output tokens, for example, through a sequence of neural network blocks, to generate an updated sequence of output tokens. The systems then generate an expanded representation from the updated sequence of output tokens, where the expanded representation has the same number of tokens as the encoded representation. By processing the shorter sequence of output tokens through the sequence of neural network blocks, the systems perform most of the computation on the shorter sequence, which requires less computational time and resources compared to processing the longer sequence of tokens, such as the sequences of tokens included in the encoded representation and the expanded representation. Thus, by leveraging downsampling and performing a large amount of computation on a shorter sequence, the systems can generate high-quality videos without requiring cascading diffusion models.

[0013] The merged sequence of output tokens is a compressed representation of the latent representations. That is, the merged sequence of output tokens is a compressed representation of the segments of one or more video frames of the video. Thus, by processing the shorter sequence of output tokens through the sequence of neural network blocks, the systems can generate outputs, such as videos, using fewer computational resources (such as memory and computational power) compared to processing the longer sequence of tokens, such as tokens that directly represent the latent representations or that directly represent the video frames.

[0014] In some examples, the systems described in this specification can process shorter sequences of output tokens using fewer accelerators than the number of accelerators required to process longer sequences of tokens. Accelerators perform matrix operations using specialized circuitry (e.g., ASICs, FPGAs, graphics processing units (GPUs), or tensor processing units (TPUs)), and more specifically, perform matrix operations on distributed machine learning systems that include multiple TPUs and / or GPUs. Some devices that run accelerators have limited memory. By reducing the memory required to generate videos as described above, the systems can be deployed on fewer accelerators than were required to deploy existing video generation models, such as a single accelerator. Thus, the systems can be deployed on devices with limited memory (e.g., user devices). By being deployed on fewer accelerators than existing video generation models, the systems can also reduce the amount of power consumed to generate videos.

[0015] The systems can perform various video generation tasks, such as unconditional video generation and conditional video generation, such as text-to-video generation, or video prediction. For example, the systems can obtain an input that includes a conditioning signal that includes an embedding of text, and the output video is described by the text. As another example, the conditioning signal can include an embedding of one or more video frames. The output video includes a coherent continuation of the one or more video frames. The latent denoising neural network can be updated to condition the latent representation on the input.

[0016] Training of the systems can be performed more efficiently, e.g., using fewer computational resources, than training the systems directly on long videos or videos at the target resolution, or both. For example, the systems can be pre-trained on smaller and shorter videos, and fine-tuned on progressively larger spatial resolutions, larger temporal resolutions, or longer durations. For example, it can be faster to initially train the systems at a base resolution and fine-tune the systems at progressively higher resolutions than to train the systems directly at the higher resolutions.

[0017] In some implementations, the systems can be trained to perform autoregressive generation using the conditioned latent representations. For example, the systems can be trained to generate videos conditioned on a latent representation, thereby performing image-to-video generation. The systems can also be trained to generate videos conditioned on multiple latent representations that provide enough context to the model to understand the direction of motion and to produce consistent motion autoregressively.

[0018] In some implementations, the systems can be trained to perform other sequence processing tasks. For example, the systems can be trained to perform video and / or image classification and understanding tasks. As another example, the systems can be trained to perform non-autoregressive generation. For example, the systems can update multiple latent representations of an output time series in parallel.

[0019] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1A is a block diagram of an example time series generation system.

[0021] FIG. 1B is a block diagram of another example time series generation system.

[0022] FIG. 2 is a flow diagram of an example process for generating a time series of data elements.

[0023] FIG. 3 is a flow diagram of an example process for generating a denoised output.

[0024] FIG. 4 shows an example latent denoising neural network.

[0025] FIG. 5 is a flow diagram of an example process for training an example latent denoising neural network.

[0026] FIG. 6 shows performance of an example time series generation system.

[0027] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0028] FIG. 1A An example time series generation system 100 is shown. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components, and techniques described below are implemented.

[0029] System 100 generates an output time series 152 conditioned on input 102.

[0030] Output time series 152 includes a respective data element at each of one or more time points. In FIG. 1A In an example, output time series 152 can be a video. Each data element of the video can be a video frame, also referred to as an image frame. The video includes video frames, each of which includes a plurality of pixels. Each pixel has one or more intensity values. The system can represent one or more video frames as latent representations.

[0031] Although this specification describes generating videos by way of example, the system can generate time series of other types of data elements, such as climate data, audio data, fluid dynamics data, and the like. The system can be trained to generate time series of a particular type using appropriate training data. The system can also generate other sequences of data elements, such as three-dimensional images, high-resolution images, audio signals, and the like.

[0032] To generate a time series of data elements, the system obtains an input 102. The input 102 includes a noise input 104 that includes a plurality of latent representations 105a-n. Each latent representation can represent one or more data elements of a time series 152. Each latent representation is a representation of one or more data elements in a latent space. The latent space can have a lower dimensionality than the data elements. Each latent representation can include one or more latent variables.

[0033] In some examples, the system can generate the noise input 104 by sampling the noise input 104 from a noise distribution. For example, the system can initialize each latent representation by sampling an initial value for each latent variable included in the latent representation from a corresponding noise distribution (e.g., a Gaussian distribution or another predetermined distribution). Thus, the latent representation includes a plurality of latent variables, where an initial value for each latent variable is sampled from a corresponding noise distribution.

[0034] In some examples, the input 102 also includes a conditioning signal 106, also referred to as a conditioning input. For example, the conditioning signal can include an embedding of text that describes content that the output video should depict. For example, the text can describe spatial resolution and visual characteristics, such as level of detail, subject matter, background, timing, angle, lighting, contrast, type of shot, and the like. In some examples, the system can receive the text from a user. FIG. 1A In an example, the conditioning signal 106 includes an embedding of the text “A slow-motion sequence of a lotus flower emerging from pond water.” In some examples, the system can receive the text from a user.

[0035] In some examples, the system can generate an embedding of text from a natural language sequence of text, e.g., using a text encoder neural network. The text encoder neural network can have any appropriate neural network architecture that allows the neural network to map a natural language sequence of text to an embedding of text, e.g., a feedforward architecture (e.g., an encoder-only Transformer neural network) or a recurrent architecture. An embedding refers to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values. As one particular example, the system can include a T5 text encoder, which is described in further detail in Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” arXiv preprint arXiv:1910.10683 (2019). As another example, the text encoder neural network can include a BERT encoder, which is described in further detail in Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv:1810.04805 (2018).

[0036] While this specification describes the conditioned signal in the form of text (or an embedding of text), in other implementations, the conditioned signal can be a different type of data, e.g., a time series of pre-existing data elements, an embedding of a time series of pre-existing data elements, a pre-existing video, an embedding of a pre-existing video, an image, an embedding of an image, a digital representation of a required object class of a video, an audio signal characterizing a scene that the video should depict, an audio signal including speech that describes the video, an embedding of an audio signal, combinations thereof, and the like. The methods and systems disclosed herein are applicable to any conditioned time series generation.

[0037] The system updates the latent representations 105a-n using a latent denoising neural network 120. The latent denoising neural network performs an inverse diffusion process to update each latent representation 105a-n at each of a plurality of iterations. Specifically, for each latent representation 105a-n, the latent denoising neural network is configured to encode a sequence of tokens representing a patch of the latent representation into an encoded representation comprising a sequence of input tokens. For each latent representation 105a-n, the system processes the encoded representation to generate a merged sequence of output tokens. The merged sequence of output tokens is shorter than the sequence of input tokens. The latent denoising neural network can perform most of the processing to update the latent representation 105a-n using the merged sequence of output tokens, which requires less computational time and resources than processing the longer sequence of tokens. Reference is made below to FIG. 1B to FIG. 4 The latent denoising neural network 120 is described in further detail.

[0038] The system processes the latent representations 105a-n using a decoder neural network 130 to generate the output time sequence 152. Reference is made below to FIG. 1B The decoder neural network 130, also referred to as a time sequence decoder neural network, is described in further detail.

[0039] In examples in which the input 102 comprises a text prompt, the output time sequence 152 comprises a video depicting text of the conditioned signal 106. For example, FIG. 1A An example frame of the video is shown, depicting “A slow-motion sequence of a lotus flower emerging from pond water”. The video is consistent with the text prompt, and has temporal consistency and high resolution. FIG. 1A In some examples, the system 100 can provide the output time sequence 152 for presentation. The system can provide the output video for display, for example, to a user. The user can interact with the system, for example, by providing input to the system via an interface (e.g., a graphical user interface or an application programming interface (API)). Specifically, the user can provide input comprising a conditioned signal. The system can provide the output video to the user, for example, for display on a user device of the user, or for storage in a data store. In some cases, the system can transmit the generated video to a user device of the user, for example, over a data communications network (e.g., the internet).

[0040]

[0041] An example time sequence generation system 100 is described above with reference to FIG. 1B FIG. 1A An example time sequence generation system 100 is described above with reference to

[0042] ​The system processes the input 102 using the latent denoising neural network 120 to generate updated latent representations 105a-n. The latent denoising neural network is configured to update each latent representation, e.g., to denoise each latent representation. For any given latent representation, the system performs an inverse diffusion process to update the latent representation.

[0043] For example, the latent denoising neural network 120 can update the first latent representation 105a by performing an inverse diffusion process. The latent denoising neural network 120 can update the second latent representation 105b by performing an inverse diffusion process conditioned on at least the first latent representation 105a.

[0044] In particular, the latent denoising neural network 120 performs an inverse diffusion process to update each latent representation 105a-n at each iteration of a plurality of iterations.

[0045] At each iteration, the system processes an intermediate input 110 that includes at least the latent representation of the iteration to generate a denoised output 122.

[0046] In examples in which the input 102 includes a conditioning signal 106, the intermediate input 110 at each iteration also includes the conditioning signal 106. The latent denoising neural network 120 updates the latent representation at each iteration conditioned on at least the conditioning signal 106.

[0047] In some examples, the denoised output 122 includes a noise estimate for the latent representation. For example, the noise estimate defines how the actual latent representation (if known) would need to be modified to generate the latent representation given the noise level corresponding to the current iteration.

[0048] In some examples, the denoised output 122 includes an estimate of the actual latent representation given the current intermediate input 110, i.e., an estimate of the latent representation that would result from removing the noise component of the current intermediate input 110.

[0049] In some examples, the denoised output 122 includes an estimate of a v prediction that can be used to estimate the actual latent representation. Examples of v prediction are described below with reference to FIG. 5

[0050] At each iteration, the system uses the denoised output 122 of the iteration to update the latent representation. For example, the system uses the denoised output 122 to modify the latent representation.

[0051] ​The latent denoising neural network 120 can have any suitable architecture for updating each latent representation. As an example, the latent denoising neural network can include a diffusion Transformer model. An example suitable diffusion Transformer model is described in Gupta et al., “Photorealistic Video Generation with Diffusion Models,” arXiv preprint arXiv:2312.06662 (2023), which is incorporated by reference herein in its entirety.

[0052] The latent denoising neural network can include multiple types of layers, including layers for performing attention, such as layer normalization layers, cross-attention layers, feedforward layers, multi-head attention layers, MLP layers, and the like. Reference is made below to FIG. 4 One example of the latent denoising neural network 120 is described in more detail.

[0053] The system generates an output time series of data elements by processing the updated latent representations 105a-n using the time series decoder neural network 130. For example, the time series decoder neural network 130 can be configured to decode the latent representations into one or more video frames.

[0054] For example, the system can generate the output time series 152 of data elements by processing each latent representation 105a-n using the time series decoder neural network 130 to generate one or more respective video frames of the latent representation. The system can combine the respective video frames of each latent representation to generate the output time series 152 of data elements. Thus, the system can process multiple latent representations 105a-n in parallel, thereby reducing the computational time required to decode the latent representations 105a-n compared to processing the latent representations serially.

[0055] In some examples, the time series decoder neural network 130 can be a time series decoder neural network of an autoencoder for which the time series encoder neural network is configured to generate latent representations of one or more data elements in a time series of data elements. For example, the time series decoder neural network 130 can be a decoder of a video autoencoder.

[0056] In some examples, the autoencoder can be a causal autoencoder. For example, the autoencoder can have a causal 3D convolutional neural network (CNN) encoder-decoder architecture. Example autoencoders are described in Yu et al., “Language Model Beats Diffusion – Tokenizer is Key to Visual Generation,” arXiv preprint arXiv:2310.05737 (2024) and Gupta et al., “Photorealistic Video Generation with Diffusion Models,” arXiv preprint arXiv:2312.06662 (2023).

[0057] FIG. 2 is a flowchart of an example process 200 for generating a time series of data elements conditioned on an input. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a time series generation system, e.g., system 100 depicted in FIG. 1, appropriately programmed, in accordance with this specification, can perform process 200. FIG. 1A to FIG. 1B

[0058] The system obtains an input (step 202). The input includes a noise input that includes a plurality of latent representations of an output time series.

[0059] In some implementations, the system can sample the noise input from a noise distribution. For example, for each latent representation, the system can sample random noise from a noise distribution as In some examples, the noise distribution can be a Gaussian distribution.

[0060] In some examples, the input can further include a conditioning signal. The conditioning signal can further include information for guiding the generation of the output time series of data elements. For example, the conditioning signal can include data representing text describing the output time series of data elements, data representing one or more data elements, etc.

[0061] ​In some examples, the data representing the one or more data elements can include one or more data elements, or an embedding representing the one or more data elements. In some examples, the system can generate the embedding representing the one or more data elements from the one or more data elements. For example, the system can process the one or more data elements using a time series encoder neural network. As one example, for one or more video frames, the system can process the one or more video frames using a video encoder, e.g., an autoencoder.

[0062] The system updates each latent representation (step 204). For example, the system can update each latent representation using a latent denoising neural network. The latent denoising neural network is configured to update latent representations. Reference is made above to FIG. 2 An example latent denoising neural network is described.

[0063] In some examples, the latent denoising neural network can be configured to update the latent representations autoregressively. For example, each latent representation can have a position corresponding to one or more data elements of the output time series. The latent denoising neural network can update a particular latent representation conditioned on one or more previously updated latent representations, e.g., latent representations that have been updated for any previous position of the latent representations prior to the particular position of the particular latent representation.

[0064] To update each latent representation, the system updates the latent representations at each iteration of a plurality of iterations (step 206). At each iteration, the system performs steps 208-210.

[0065] The system processes the intermediate input of the iteration to generate a denoised output (step 208). The intermediate input of the iteration includes the latent representations. The denoised output can include, e.g., a noise estimate for the latent representations , an estimate of the actual latent representations given the intermediate input, or an estimate of the v predictions that can be used to estimate the actual latent representations.

[0066] In some examples, for each latent representation after the first latent representation, the intermediate input can include one or more previously updated latent representations.

[0067] In examples in which the input includes a conditioning signal, the intermediate input of the iteration includes the conditioning signal.

[0068] Reference is made below to FIG. 3 An example process for generating a denoised output is described in further detail.

[0069] The system updates the latent representation using the iterated denoised output (step 210). For example, the system can update the latent representation, e.g., using any appropriate diffusion model state transition rule, such as DDIM (further details of which can be found in J. Song et al., “Denoising Diffusion Implicit Models,” ICLR 2021, which is incorporated by reference herein in its entirety), DDPM (further details of which can be found in J. Ho et al., “Denoising Diffusion Probabilistic Models,” NeurIPS 2020, which is incorporated by reference herein in its entirety), or other appropriate state transition rule.

[0070] In some implementations, the system can use guidance when performing the reverse diffusion process. That is, the reverse diffusion process is sometimes a guided reverse diffusion process. In some examples, the guidance can be classifier-free guidance.

[0071] Using classifier-free guidance can involve generating multiple denoised outputs at any given iteration by the latent denoising neural network. In examples in which the input includes a conditioning signal c, the system can generate a first denoised output for the iteration by processing an intermediate input that includes the conditioning signal. The system can generate a second denoised output for the iteration by processing an intermediate input that does not include the conditioning signal, e.g., that contains a null value. The system can combine the first and second denoised outputs, e.g., according to a weighting and to generate a combined denoised output. The system can use the combined denoised output to update the latent representation.

[0072] The system generates an output time series of data elements (step 212). For example, the system can process the updated latent representation using a time series decoder neural network.

[0073] In some examples, the output time series of data elements is a video. Each data element is a video frame. In these examples, the time series decoder neural network can be configured to generate one or more video frames given latent representations of one or more video frames.

[0074] In some of these examples, the input can include a conditioning signal that represents text that describes the output video. For example, the conditioning signal can include text that describes the output video, or an embedding of text that describes the output video.

[0075] In some examples, the system can generate an embedding of the text describing the output video given the text describing the output video. For example, the system can process the text describing the output video using a text encoder, such as a text encoder neural network. The text encoder neural network can have any appropriate neural network architecture that allows the neural network to map a natural language sequence of text to an embedding of the text, such as a feed-forward architecture (e.g., an encoder-only Transformer neural network) or a recurrent architecture. As an example, the text encoder can include a T5 text encoder.

[0076] FIG. 3 is a flowchart of an example process 300 for generating a denoised output. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a time series generation system (e.g., the system 100 depicted in FIG. 1) appropriately programmed in accordance with this specification can perform the process 300. For example, the system can perform the process 300 as part of step 208 described above with reference to FIG. 2 to process an iterated intermediate input to generate a denoised output. FIG. 1A to FIG. 1B FIG. 2

[0077] The system generates a sequence of tokens (step 302). Each token represents a respective patch of the latent representation.

[0078] In some examples, each respective patch can include a spatiotemporal region, such as a spatial region over one or more data elements represented by the latent representation. In some examples, each respective patch can include a spatial region of a particular data element represented by the latent representation. For example, the system can generate a sequence of non-overlapping patches where , and p is a patch size. As a particular example, each respective patch can have a patch size of p = 1, i.e., one patch per pixel per frame.

[0079] In some examples, to generate the tokens, the system can generate a respective patch embedding for each respective patch of the latent representation. For example, the system can partition the latent representation into a sequence of patches. The system can process each patch using one or more embedding layers to generate a respective patch embedding.

[0080] The system can generate one or more corresponding positional embeddings for each respective patch embedding. The corresponding positional embeddings can be learnable or fixed. In some examples, the one or more corresponding positional embeddings can be derived from a spatial positional embedding and a temporal positional embedding. For example, the one or more positional embeddings can include a positional embedding that is a sum of a spatial positional embedding and a temporal positional embedding. As a particular example, the one or more positional embeddings can include a positional embedding that is a sum of normalized absolute positional embeddings for a height axis, a width axis, and a time axis.​​​

[0081] In examples in which the sequence of data elements includes a video frame, the system can add a temporal position embedding corresponding to the latent representation of the video frame.

[0082] The system can generate a sequence of tokens by combining the respective patch embedding and one or more corresponding position embeddings for each respective patch. For example, the system can add the respective patch embedding and one or more corresponding position embeddings. Representing a latent representation as a sequence of tokens is described in further detail in Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929 (2020).

[0083] The system generates an encoded representation of the sequence of tokens (step 304). The encoded representation includes a sequence of input tokens. For example, the sequence of input tokens can have the same number of tokens as the sequence of tokens. Each input token can correspond to a respective token and can have an updated value from the corresponding token.

[0084] For example, the system can use an encoder subnetwork to generate the encoded representation. The encoder subnetwork can include one or more neural network blocks. A neural network block operates on the entire sequence length of the sequence of tokens that each represent a respective patch. Reference is made below to FIG. 4 An example neural network block is described in further detail.

[0085] Accordingly, the system can process the full number of tokens using the encoder subnetwork. Processing the full number of tokens, where each token represents a smaller patch, can improve performance compared to processing smaller sequences of tokens where each token represents a larger patch. Because the computational cost of processing the full number of tokens using a neural network block is high, the system can perform processing on a smaller number of tokens, as described below.

[0086] The system processes the encoded representation to generate a merged sequence of output tokens (step 306). The output tokens can have the same dimensionality as the input tokens. The number of tokens in the merged sequence is less than the number of input tokens in the sequence of input tokens. Accordingly, the system can perform processing for other steps using a smaller number of tokens than the full sequence length of the sequence of tokens that each represent a respective patch, thereby reducing the computational resources that would otherwise be required to process a larger number of tokens. In particular, for larger models and longer sequences, neural network layers such as feedforward layers can consume a large amount of memory.

[0087] For example, to generate the merged sequence, the system can generate a plurality of groups of input tokens from the sequence of input tokens. Each group can include a plurality of input tokens. As a particular example, the group can represent a local 3D patch having dimensions .

[0088] For each group, the system can concatenate the input tokens of the group. For each group, the system can apply a linear transformation to the concatenated input tokens to generate an initial output token.

[0089] In some examples, for each group, the system can add an average patch value to the initial output token to generate an output token for the group. For example, the average patch value can be an average of the input tokens of the group. The system can include the output token of each group in the merged sequence of output tokens.

[0090] The system processes the merged sequence using a sequence of neural network blocks to generate an updated merged sequence (step 308). The updated merged sequence includes a respective updated output token for each output token in the merged sequence.

[0091] Each neural network block can be configured to process an output token in the merged sequence to update the output token. Each neural network block can be configured to apply spatial self-attention, spatio-temporal self-attention, and / or cross-attention. Example neural network blocks are described in further detail below with reference to FIG. 4 .

[0092] By processing the merged sequence, which includes a smaller number of tokens than the sequence of tokens, the system can reduce the computational resources that would otherwise be required to process the larger number of tokens.

[0093] The system processes the updated merged sequence to generate an expanded representation (step 310). The expanded representation includes a number of tokens that is the same as the number of input tokens in the sequence of input tokens.

[0094] To restore the number of tokens in the sequence to the number of input tokens in the sequence of input tokens, for each output token in the updated merged sequence, the system can apply a second linear transformation to the output token to generate a respective representation of the output token. The respective representation of the output token can have the same dimensions as the concatenated input tokens described above.

[0095] For each respective representation, the system can divide the respective representation of the output token into a group of a plurality of initial patch tokens. Each initial patch token of the group can have the same dimensions as the input tokens described above. The group corresponds to the output token.

[0096] In some examples, for each initial patch token in each group, the system can add the corresponding output token of the group to the initial patch token to generate an updated patch token. The system can include the updated patch token of each group in the expanded representation.

[0097] Accordingly, the system can recover the updated merged representation to have the same number of tokens as the input tokens to further process the full number of tokens.

[0098] The system processes the decoder input to generate a denoised output (step 312). The decoder input can be derived from the expanded representation. For example, the decoder input can include a sequence of decoder input tokens. The decoder input can have the same number of tokens as the expanded representation and the encoded representation.

[0099] In some examples, the decoder input includes a combination of the expanded representation and the encoded representation. For example, the system can use skip connections to combine the encoded representation and the expanded representation. As an example, the system can combine (e.g., add) each token of the expanded representation with a corresponding token of the encoded representation to generate a decoder input token in a sequence of decoder input tokens.

[0100] For example, the system can process the decoder input using a decoder subnetwork to generate the denoised output. The encoder subnetwork can include one or more neural network blocks. A neural network block operates on the entire sequence length of a sequence of tokens each representing a respective patch. See FIG. 4 An example neural network block is described below.

[0101] Processing the full number of tokens each representing a smaller patch can improve performance compared to processing a smaller sequence of tokens each representing a larger patch of tokens. Computing the full number of tokens using a sequence of neural network blocks is computationally expensive. Accordingly, processing the full number of tokens using a smaller number of neural network blocks in, for example, the encoder subnetwork and the decoder subnetwork, and processing the smaller number of tokens using a sequence of neural network blocks can achieve high performance while also reducing the computational resources that would otherwise be required to process the full number of tokens using a sequence of neural network blocks.

[0102] FIG. 4 An example latent denoising neural network 120 described above with reference to FIG. 1A to FIG. 1B is shown. The latent denoising neural network 120 includes a plurality of neural network layers, such as an encoder subnetwork 410, a patch merging layer 420, one or more neural network blocks 430 (also referred to as WALT blocks), a patch expansion layer 440, and a decoder subnetwork 450.

[0103] The system obtains a sequence of tokens 408 representing a video, as described above with reference to FIG. 3 .

[0104] The latent denoising neural network 120 processes the sequence of tokens 408 using an encoder subnetwork 410. The encoder subnetwork 410 is configured to generate an encoded representation 412 of the sequence of tokens 408. The encoder subnetwork 410 can include one or more neural network blocks, such as the neural network block 430 described below. In FIG. 4 In examples, the encoder subnetwork 410 includes two neural network blocks. For each of the B latent representations in the batch, the encoder subnetwork can operate on a sequence of tokens of the latent representation representing a dimension (t, h, w).

[0105] The latent denoising neural network 120 processes the encoded representation 412 using a patch merging layer 420. The patch merging layer 420 is configured to generate a merged sequence of output tokens 422 from the encoded representation 412. For example, the patch merging layer 420 can perform the steps 306 described above with reference to FIG. 3

[0106] The merged sequence of output tokens 422 includes fewer tokens than the encoded representation 412. For example, the patch merging layer 420 can generate an output token for each group of input tokens of the encoded representation 412. In FIG. 4 In examples, each group can represent a local 3D patch of dimension .

[0107] The latent denoising neural network 120 processes the merged sequence 422 using a sequence of neural network blocks 430 to generate an updated merged sequence 432. For each of the B latent representations in the batch, each neural network block can operate on a sequence of output tokens representing ) local patches. For example, the sequence of neural network blocks 430 can perform the steps 308 described above with reference to FIG. 3

[0108] The updated merged sequence 432 includes a respective updated output token for each output token in the merged sequence 422.

[0109] In examples, the sequence of neural network blocks 430 can include L neural network blocks. The sequence of neural network blocks 430 can include any suitable number of neural network blocks. In some examples, the sequence of neural network blocks 430 can include more neural network blocks than the number of neural network blocks included in the encoder subnetwork 410, the decoder subnetwork 450, or both. For example, the sequence of neural network blocks 430 can include 8, 10, 12, etc. neural network blocks. FIG. 4

[0110] ​​​In some examples, each neural network block is configured to apply spatial self-attention on a particular latent representation. For example, the latent representation can have a time dimension and two spatial dimensions. The system can partition the output tokens of the merged sequence into multiple spatial groups of output tokens, where each spatial group includes output tokens for the same time index of the time dimension. The system can apply attention on the output tokens of each spatial group.

[0111] For example, the latent denoising neural network 120 can use a sequence of neural network blocks to process each spatial group of output tokens to update each output token. Each neural network block can be configured to apply attention on the output tokens of the spatial group using keys, values, and queries derived from the output tokens.

[0112] In some examples, each neural network block is configured to apply spatio-temporal self-attention on a particular latent representation. The system can partition the output tokens of the merged sequence into spatio-temporal groups of output tokens, where each spatio-temporal group includes output tokens for the same spatial region over multiple time indices of the time dimension. The system can apply attention on the output tokens of each spatio-temporal group.

[0113] For example, the latent denoising neural network 120 can use a sequence of neural network blocks to process each spatio-temporal group of output tokens to update each output token. Each neural network block can be configured to apply attention on the output tokens of the spatio-temporal group using keys, values, and queries derived from the output tokens.

[0114] In these examples, each neural network block can apply a windowed attention mechanism to improve computational efficiency. For example, the system can use a windowed attention layer to avoid full attention layers between large patches.

[0115] In examples where the input includes a conditioning input, the sequence of neural network blocks is configured to apply attention on the conditioning signal and the output tokens of the merged sequence. For example, the latent denoising neural network can include multiple layers, each layer configured to apply attention using keys and values derived from the conditioning signal and queries derived from the output tokens. For example, each neural network block can be configured to apply attention on the conditioning signal and the output tokens of the merged sequence to update the output tokens of the merged sequence using keys and values derived from the conditioning signal and queries derived from the output tokens of the merged sequence.

[0116] In some examples, each neural network block can perform spatial self-attention, cross- attention, and spatio-temporal self-attention in any order. For example, each neural network block can perform spatial self-attention, cross-attention, spatio-temporal self-attention, and cross- attention. In some examples, some neural network blocks can perform different combinations of spatial self- attention, cross-attention, and spatio-temporal self-attention. For example, some neural network blocks can perform spatial self-attention and spatio-temporal self-attention. In some examples, some neural network blocks can perform only spatial self-attention or only spatio-temporal self-attention.

[0117] Each neural network block can include multiple types of neural network layers for performing attention, such as layer normalization layers, cross-attention layers, feedforward layers, multi-head attention layers, MLP layers, and the like.

[0118] In FIG. 4 In the example of FIG. 4, the first neural network block 434 includes a spatial self- attention layer followed by a spatial cross-attention layer. The second neural network block 436 includes a spatio-temporal self-attention layer followed by a spatial cross-attention layer.

[0119] For example, a cross-attention layer can perform cross-attention on the conditioned signal and the latent representation using keys and values derived from the conditioned signal and a query derived from an output token of the merged sequence. As a particular example, a cross- attention layer can be a spatial cross-attention layer that performs cross-attention on the conditioned signal and the latent representation using keys and values derived from the conditioned signal and a query derived from an output token of the merged sequence that belongs to the same spatial group.

[0120] In some examples, the latent denoising neural network 120 can use one or more of: absolute positional embeddings, relative positional embeddings, or rotary positional embeddings (RoPE). As a particular example, the latent denoising neural network 120 can use absolute positional embeddings and RoPE.

[0121] As an example, each neural network block can use rotary positional embeddings (RoPE). The system can use rotary positional embeddings to infer new sequence lengths, such as sequence lengths that are longer than the sequence lengths seen during training. Moreover, in some cases, using RoPE can reduce training loss and adapt more quickly to different spatial resolutions and temporal durations, as described below with reference to FIG. 5 Further details are described in further detail.

[0122] For example, each neural network block can receive a block input for the neural network block that includes a merged sequence of output tokens. For each output token, the neural network block can generate one or more rotation position embeddings. Each rotation position embedding can be generated by a rotation matrix. In some implementations in which the neural network module generates more than one rotation position embedding, the rotation position embeddings can be generated using different rotation matrices or the same rotation matrix. In some examples, the system can determine the rotation matrix based on a position of the output token in the merged sequence of output tokens and a set of frequencies.

[0123] The neural network block can process the output tokens and the rotation position embeddings by applying one or more attention mechanisms to generate a block output for the neural network block that includes a respective updated output token for each output token in the merged sequence. For example, the system can transform a query vector and a key vector for each output token using one or more rotation position embeddings when applying attention.

[0124] In some examples, the system can transform one or more sub-vectors of the key vector and the query vector using a corresponding rotation position embedding. As an example, the system can divide each of the query vector and the key vector into N sub-vectors. The system can divide the query vector and the key vector into N sub-vectors, each sub-vector including a subset of dimensions.

[0125] As a particular example, the query vector and the key vector can be 1024-dimensional vectors. The system can divide the query vector and the key vector into four sub-vectors, each sub-vector having 256 dimensions.

[0126] One or more of the N sub-vectors can have a corresponding axis. In some examples, N can be greater than a number of axes of the time series.

[0127] For example, for a time series that is a video having three axes (e.g., time, spatial width, and spatial height), N can have a value of four. The query vector and the key vector can each represent a patch in (t, x, y) coordinates. The system can transform a first sub-vector of the query vector and the key vector along a time axis, respectively. The system can transform a second sub-vector of the query vector and the key vector along a vertical y-axis, respectively. The system can transform a third sub-vector of the query vector and the key vector along a horizontal x-axis, respectively. The system can leave a fourth sub-vector of the query vector and the key vector untransformed, respectively. For example, the system can transform the sub-vectors by applying independent one-dimensional rotation position embeddings (each corresponding to an axis) to each sub-vector.

[0128] The system processes the updated merge sequence 422 using a patch expansion layer 440. The patch expansion layer 440 is configured to generate an expanded representation 442 from the updated merge sequence 422, which includes the same number of tokens as the number of input tokens in the encoded representation 412. For example, the patch expansion layer 440 can perform the steps 310 described above with respect to the sequence of tokens 410. FIG. 3 The steps 310 described above.

[0129] The system processes the decoder input 448 using a decoder subnetwork 450. For example, the decoder input 448 can include a combination of the encoded representation 412 and the expanded representation 442.

[0130] The decoder subnetwork 450 is configured to generate a denoised output 452. The decoder subnetwork 450 can include one or more neural network blocks, such as the neural network block 430. In some examples, the decoder subnetwork 450 can include one or more neural network layers that map the decoder input 448 to a latent space of latent representations.

[0131] The encoder subnetwork and the decoder subnetwork can include one or more neural network blocks, similar to the sequence neural network blocks 430 described above. For example, the encoder subnetwork can perform one or more attention mechanisms on a sequence of tokens to generate an encoded representation. The decoder subnetwork can perform one or more attention mechanisms on a decoder input that includes a sequence of tokens derived from the expanded representation to generate a denoised output. In some examples, the decoder subnetwork can include an output subnetwork that processes an output of a last neural network block of the decoder subnetwork to generate the denoised output. As an example, the output subnetwork can be configured to project the output of the last neural network block of the decoder subnetwork to the denoised output.

[0132] For example, each neural network block can be configured to apply spatial self-attention, spatio-temporal self-attention, and / or cross-attention.

[0133] For example, one or more neural network blocks can be configured to apply spatial self-attention to a particular latent representation. For example, the latent representation can have a temporal dimension and two spatial dimensions. For the encoder subnetwork, the system can partition the sequence of tokens into token spatial groups, where each spatial group includes tokens of the same temporal index in the temporal dimension. The system can apply attention on the tokens of each spatial group. For the decoder subnetwork, the system can partition the decoder input into token spatial groups. The system can apply attention on the tokens of each spatial group. For example, the spatial window attention can be limited to all tokens within the latent representation of size , where the first dimension is temporal.

[0134] In some examples, one or more of the neural network blocks can be configured to apply spatio-temporal self-attention on a particular latent representation. For an encoder subnetwork, the system can partition the sequence of word pieces into spatio-temporal groups of word pieces, where each spatio-temporal group includes word pieces for the same spatial region over multiple time indices of the temporal dimension. The system can apply attention on the word pieces of each spatio-temporal group. For a decoder subnetwork, the system can partition the decoder input into spatio-temporal groups of word pieces. The system can apply attention on the word pieces of each spatio-temporal group. For example, the spatio-temporal window attention can be restricted to word pieces within a 3D window of size . .

[0135] In some examples where the input includes a conditioning input, each neural network block of the encoder subnetwork is configured to apply attention on the conditioning signal and the sequence of word pieces. In some examples, each neural network block of the decoder subnetwork is configured to apply attention on the conditioning signal and the decoder input.

[0136] In some examples, each neural network block of the encoder subnetwork, the decoder subnetwork, or both can use a rotated position embedding (RoPE). For example, each neural network block of the encoder subnetwork can receive a block input for the neural network block that includes a sequence of word pieces 408. For each word piece, the neural network block can generate a rotated position embedding using a rotation matrix. The neural network block can process the word piece and the rotated position embedding by applying one or more attention mechanisms to generate a block output for the neural network block that includes a respective updated word piece for each word piece in the sequence. For example, the system can transform the query vector and the key vector for each word piece using one or more rotated position embeddings when applying attention. Rotated position embeddings are described in further detail above.

[0137] As another example, each neural network block of the decoder subnetwork can receive a block input for the neural network block that includes the decoder input 448. For each decoder input word piece of the decoder input 448, the neural network block can generate a rotated position embedding using a rotation matrix. The neural network block can process the decoder input word piece and the rotated position embedding by applying one or more attention mechanisms to generate a block output for the neural network block that includes a respective updated word piece for each decoder input in the decoder input 448. For example, the system can transform the query vector and the key vector for each decoder input word piece using one or more rotated position embeddings when applying attention. Rotated position embeddings are described in further detail above.

[0138] FIG. 5is a flowchart of an example process 500 for training an example latent denoising neural network. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a time series generation system (e.g., the system 100 depicted in FIG. 1) or a separate training system, appropriately programmed in accordance with this specification, can perform the process 500. FIG. 1A to FIG. 1B The system 100 depicted in FIG. 1) or a separate training system can perform the process 500.

[0139] The system can repeatedly perform the process 500 to train a latent denoising neural network. For example, the system can perform the process 500 on a plurality of training examples.

[0140] The system obtains a training example (step 502). The training example includes a training time series of data elements. In some examples, the training time series of data elements includes one data element.

[0141] In some examples, the training example includes a training conditioning input. The training conditioning input can include one or more conditioning latent representations.

[0142] The training example includes: (i) a respective training time series of data elements that includes one or more training data elements; and (ii) a training conditioning signal. In some examples, the training conditioning signal can include text, an embedding of text, one or more data elements, an embedding of one or more data elements, or a null value.

[0143] The system generates one or more benchmark true latent representations (step 504). Each benchmark true latent representation can represent one or more data elements in the training time series of data elements.

[0144] For example, the system can generate each benchmark true latent representation for the training time series of data elements using a time series encoder neural network. As an example, the time series encoder neural network can be a time series encoder neural network of an autoencoder, where a time series decoder neural network is configured to reconstruct the training time series of data elements.

[0145] For example, the benchmark true latent representation z can be determined as where E(x) is the time series encoder neural network, x is the training time series of data elements, and is a time and space compression factor.

[0146] In some examples, the system can perform the process 500 after training the time series encoder neural network, the time series decoder neural network, or both. The training of the time series encoder neural network and the time series decoder neural network are described in further detail below.

[0147] The system adds noise to each ground-truth latent representation to generate a training input (step 506). For example, the system can sample a diffusion timestep corresponding to a noise level . The system can sample the noise from a noise distribution, e.g., from a Gaussian distribution . The system can add the noise to the ground-truth latent representation according to the diffusion timestep to create the training input. In some examples, the system can scale the noise based on a noise schedule for the sampled diffusion timestep t. For example, the training input can be described as where is a noise schedule, e.g., a monotonically decreasing function from 1 to 0, and is the ground-truth latent representation.

[0148] For each training input, the system generates a training denoised output (step 508). For example, the system can generate the training denoised output from the training input by processing a training intermediate input that includes the training input using the latent denoising neural network according to current values of the parameters of the latent denoising neural network.

[0149] In some examples, the training intermediate input includes a training conditioning signal.

[0150] The system determines an update to the parameters of the latent denoising neural network (step 510). For example, the system can determine an update to the parameters for optimizing a training objective.

[0151] For example, the training objective can measure an error between a target w and a denoised output generated by processing a training intermediate input in step 508. As an example, the system can train the latent denoising neural network to optimize a denoising objective where c is a conditioning signal, is the training input, t is the diffusion timestep, and is the latent denoising neural network. The target w may, for example, be the noise , the ground-truth latent representation , or a v predicted value. In some examples, the v predicted value can be described as .

[0152] In some examples, the training objective measures an error between the ground-truth latent representation and a denoised representation generated using the training denoised output.

[0153] In some examples, the system can further train (e.g., fine-tune) the latent denoising neural network. For example, the latent denoising neural network can have been trained on smaller, shorter videos at a base resolution. The system can fine-tune the latent denoising neural network on a set of tuning examples that include one or more of: a larger spatial resolution, a larger temporal resolution, or a longer duration, compared to the time series on which the latent denoising neural network was trained. In some examples, the system can fine-tune the latent denoising neural network at the target resolution and duration for inference.

[0154] In some examples, the system can fine-tune the latent denoising neural network with progressively larger resolutions and longer durations. For each increase in resolution, duration, or both, the system can interpolate the absolute positional embeddings and scale the window size. For example, the system can sequentially fine-tune the latent denoising neural network on a set of tuning examples with one or more of: different spatial resolutions, different temporal resolutions, or different durations. The set of tuning examples can be ordered by increasing the spatial resolution, temporal resolution, or duration in a sequence. For example, the sequence can start with a set of tuning examples with lower spatial resolution, temporal resolution, or duration.

[0155] Accordingly, the system consumes fewer computational resources to train the latent denoising neural network to generate videos with larger resolutions, longer durations, or both, compared to training the latent denoising neural network from scratch directly on larger resolutions or durations. For example, the progressively trained latent denoising neural network can train faster at a base resolution compared to training the latent denoising neural network directly at the target resolution. Moreover, training faster at a base resolution can scale the training of the latent denoising neural network, or fine-tune for longer at the target resolution, under a fixed computational budget. Furthermore, in examples where the neural network blocks use rotational positional embeddings, the latent denoising neural network can adapt faster, e.g., using fewer training iterations or less computational time, to achieve a threshold performance on videos with larger resolutions, longer durations, or both.

[0156] As another example, the system can fine-tune the latent denoising neural network on a set of adjustment examples that include high-quality time series. For example, the system can select high-quality videos from a training dataset of videos. The system can determine the quality of a video based on motion exhibited in the video. For example, the system can use a pre-trained optical flow model to determine motion measurements between video frames in a video. The system can select videos for inclusion in the adjustment examples that exhibit motion that satisfies a threshold motion measurement. For example, the system can filter out videos that have too little motion or too much motion, e.g., videos with motion measurements that are outside a threshold range. Thus, the system can enhance the ability of the latent denoising neural network to generate temporally consistent videos.

[0157] In some examples, the system can train the latent denoising neural network jointly on text-image pairs and text-video pairs. In some examples, the text can be represented by text embeddings. For example, one or more of the training examples can include an image, and a training conditioning signal that represents text that describes the image. One or more of the training examples can include a video, and a training conditioning signal that represents text that describes the video. Thus, the system can train the latent denoising neural network to perform text-conditioned image and video generation.

[0158] Alternatively or additionally, the system can train the latent denoising neural network according to a frame prediction task. For example, for one or more of the training examples, the training conditioning signal can include one or more conditioning latent representations. Each conditioning latent representation can represent one or more data elements. In some examples, the system can randomly sample the number of conditioning latent representations, e.g., from a range of 1 to n conditioning latent representations. Conditioning on a single conditioning latent representation that represents a video frame can allow the latent denoising neural network to perform image-to-video generation. Conditioning on more than one conditioning latent representation can provide the latent denoising neural network with sufficient context to understand a direction of motion and to produce consistent motion autoregressively based on the training latent representations. Thus, the system can train the latent denoising neural network to perform long video generation using autoregressive prediction.

[0159] In some examples, the training can train a time series decoder neural network, a time series encoder neural network, or both. In some examples, the time series decoder neural network, the time series encoder neural network, or both can be trained and frozen prior to training of the latent denoising neural network.

[0160] In some examples, the time series decoder neural network and the time series encoder neural network are part of an autoencoder. In examples in which the time series generation system generates videos, the autoencoder can be trained to reconstruct images, videos, or both.

[0161] The system can train the autoencoder on training images, training videos, or both. In some examples, the training images, training videos, or both can be derived from a set of training videos. As an example, the training videos can have a lower resolution than the videos in the video training set. For example, the training videos can include cropped patches from the videos in the video training set. Alternatively or additionally, the training videos can be resized from the videos in the video training set. In examples where the autoencoder is a CNN-based autoencoder, the system can exploit the translational equivariance property of CNNs to enable the autoencoder to efficiently generalize when applied to higher resolutions by training the autoencoder using the cropped patches or resized videos.

[0162] In some examples, the system can train the autoencoder on training videos that are shorter than the videos generated at inference time. The system can fine-tune the autoencoder on videos that are longer than the training videos. Thus, the system can train the autoencoder on short videos, enabling faster training and reducing the consumption of computational resources that would otherwise be used to train on long videos.

[0163] For example, the autoencoder can be trained to optimize one or more of a pixel-level reconstruction loss (e.g., mean squared error), a perceptual loss, or an adversarial loss. In some examples, the autoencoder can be trained using a combination (e.g., sum) of the pixel-level reconstruction loss, the perceptual loss, and the adversarial loss.

[0164] FIG. 6 Performance of example time series generation systems is shown. Specifically, FIG. 6 Performance of various techniques according to the Frechet Video Distance (FVD), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and perceptual metric (LPIPS) metrics for evaluating the quality of generated videos is shown on the video generation task.

[0165] As can be seen from Table 600, for the unconditional video generation task of length 128 frames, the time series generation systems described in this specification (labeled “Time Series Generation System”) outperform other video generation techniques in terms of FVD (lower is better). FVD measures the quality and realism of the generated videos by comparing the distribution represented by the generated videos to the distribution of real video data.

[0166] As can be seen from Table 650, for the conditional video generation task of predicting 80 frames conditioned on 20 frames, the time series generation system described in this specification outperforms other video generation techniques in terms of FVD (lower is better), PSNR (higher is better), SSIM (higher is better), and LPIPS (lower is better). Specifically, the time series generation system described in this specification outperforms the TECO technique despite having less than half the number of parameters. Additionally, FIG. 6 Performance of the time series generation system described in this specification is shown without progressive training or high spatio-temporal compression. Thus, even without progressive training or high spatio-temporal compression, the system can train on sequences of over 10,000 latent representations or over 128 video frames without requiring large amounts of on-accelerator memory.

[0167] The specification uses the term“configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions, means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0168] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied

[0169] The term“data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0170] A computer program (which can also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a data communication network.

[0171] In this specification, the term“database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an index database can include multiple collections of data, each of which can be organized differently and accessed differently.

[0172] Similarly, in this specification, the term“engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0173] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0174] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0175] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0176] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0177] Data processing devices for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing general and compute-intensive portions of machine learning training or production (i.e., inference, workloads).

[0178] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0179] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0180] The computing system can include a client and a server. The client and the server typically are remote from each other and interact through a communication network. The relationship of client and server is one of communication and can or can not be a master-slave relationship. In some embodiments, the server transmits data (e.g., an HTML page) to the client device for purposes of displaying data to and receiving user input from a user interacting with the device, e.g., a user that is a client. Data generated at the client device, e.g., a result of the user interaction, can be received from the client device at the server.

[0181] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0182] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0183] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some instances, multitasking and parallel processing can be advantageous.

[0184] This specification also includes the following clauses:

[0185] Clause 1. A computer-implemented method of generating an output time series of data elements conditioned on an input, the method comprising: obtaining the input, wherein the input comprises a noisy input, the noisy input comprising a plurality of latent representations of the output time series; updating each latent representation using a latent denoising neural network, wherein the latent denoising neural network is configured to update the latent representation, the updating comprising updating the latent representation at each of a plurality of iterations, wherein the updating comprises, at each iteration: processing an intermediate input comprising the iteration of the latent representation to generate a denoised output, the processing comprising: generating a sequence of tokens each representing a respective patch of the latent representation; generating an encoded representation of the sequence of tokens, the encoded representation comprising a sequence of input tokens; processing the encoded representation to generate a merged sequence of output tokens, wherein a number of output tokens of the merged sequence is less than a number of input tokens in the sequence of input tokens; processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence, the updated merged sequence comprising a respective updated output token for each of the output tokens in the merged sequence; processing the updated merged sequence to generate an expanded representation, the expanded representation comprising a same number of tokens as the number of input tokens in the sequence of input tokens; and processing a decoder input derived from the expanded representation to generate the denoised output; updating the latent representation using the denoised output of the iteration; and generating the output time series of data elements by processing the updated latent representation using a decoder neural network.

[0186] Clause 2. The method of clause 1, wherein the latent denoising neural network is configured to update the latent representation autoregressively, and wherein for each latent representation other than a first latent representation, the intermediate input comprises one or more previously updated latent representations.

[0187] Clause 3. The method of any preceding clause, wherein processing the encoded representation to generate a merged sequence of output tokens comprises: generating a plurality of groups of input tokens from the sequence of input tokens, wherein each group comprises a plurality of input tokens; for each group, concatenating the input tokens of the group; and for each group, applying a linear transformation to the concatenated input tokens to generate an initial output token.

[0188] Clause 4. The method of clause 3, further comprising: for each group, adding an average patch value to the initial output token to generate an output token for the group, wherein the average patch value is an average of the input tokens of the group; and including the output token of each group in the merged sequence of output tokens.

[0189] Clause 5. The method of any of clauses 3-4, wherein processing the updated merged sequence to generate an expanded representation comprises, for each output token in the updated merged sequence, applying a second linear transformation to the output token to generate a respective representation of the output token, and for each respective representation, splitting the respective representation into a group of multiple initial patch tokens, wherein the group corresponds to the output token.

[0190] Clause 6. The method of clause 5, further comprising, for each initial patch token in each group, adding the corresponding output token of the group to the initial patch token to generate an updated patch token, and including the updated patch token of each group in the expanded representation.

[0191] Clause 7. The method of any preceding claim, wherein the latent representation has a temporal dimension and two spatial dimensions, and wherein processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence comprises dividing the output tokens of the merged sequence into a plurality of spatial groups of output tokens, wherein each spatial group includes output tokens of a same temporal index for the temporal dimension, and processing each spatial group of output tokens using the sequence of neural network blocks to update each output token, wherein each neural network block is configured to apply attention over the output tokens of the spatial group using a key, a value, and a query derived from the output token.

[0192] Clause 8. The method of clause 7, wherein processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence comprises dividing the output tokens of the merged sequence into a plurality of spatio-temporal groups of output tokens, wherein each spatio-temporal group includes output tokens from a same spatial region across a plurality of temporal indices of the temporal dimension, and processing each spatio-temporal group of output tokens using the sequence of neural network blocks to update each output token, wherein each neural network block is configured to apply attention over the output tokens of the spatio-temporal group using a key, a value, and a query derived from the output token.

[0193] Clause 9. The method of any preceding clause, wherein each neural network block is configured to: receive a block input of the neural network block that includes the merged sequence of output tokens; for each output token, generate a rotated positional embedding using a rotation matrix; and process the output token and the rotated positional embedding by applying one or more attention mechanisms to generate a block output of the neural network block that includes a respective updated output token for each of the output tokens in the merged sequence.

[0194] Clause 10. The method of any preceding clause, wherein the decoder input includes a combination of the expanded representation and the encoded representation.

[0195] Clause 11. The method of any preceding clause, wherein the input further comprises a conditioning signal, and wherein the intermediate input further comprises the conditioning signal.

[0196] Clause 12. The method of clause 11, wherein each neural network block is configured to apply attention on the conditioning signal and the output token of the merged sequence to update the output token of the merged sequence using keys and values derived from the conditioning signal and a query derived from the output token of the merged sequence.

[0197] Clause 13. The method of any preceding clause, wherein obtaining the input comprises sampling the noisy input from a noise distribution.

[0198] Clause 14. The method of any preceding clause, wherein generating the sequence of tokens each representing a respective patch of the latent representation comprises: generating a respective patch embedding for each respective patch of the latent representation; generating one or more corresponding positional embedding for each respective patch embedding; and generating the sequence of tokens by combining the respective patch embedding and the one or more corresponding positional embedding for each respective patch embedding.

[0199] Clause 15. The method of clause 14, wherein the one or more corresponding positional embedding is derived from a spatial positional embedding and a temporal positional embedding.

[0200] Clause 16. The method of any of clauses 14 to 15, wherein each respective patch comprises a spatio-temporal region over one or more data elements represented by the latent representation.

[0201] Clause 17. The method of any of clauses 14 to 15, wherein each respective patch comprises a spatial region of a particular data element represented by the latent representation.

[0202] Clause 18. The method of any preceding clause, wherein the decoder neural network has been trained and frozen prior to training the latent denoising neural network.

[0203] Clause 19. The method of any preceding Clause, wherein the latent denoising neural network has been trained by repeatedly: obtaining a training example comprising a training time series of data elements; generating, using an encoder neural network, a plurality of ground-truth latent representations for the training time series of data elements; adding noise to each ground-truth latent representation to generate a training input; for each training input: generating, from the training input, a training denoised output by processing a training intermediate input comprising the training input using the latent denoising neural network in accordance with current values of parameters of the latent denoising neural network; and determining an update to the parameters of the latent denoising neural network for optimizing a training objective.

[0204] Clause 20. The method of Clause 19, wherein the training objective measures an error between the ground-truth latent representations and denoised representations generated using the training denoised outputs.

[0205] Clause 21. The method of any of Clauses 19-20, further comprising fine-tuning the latent denoising neural network.

[0206] Clause 22. The method of any of Clauses 19-21, wherein the training example further comprises a training conditioning input, and wherein the training conditioning input comprises one or more conditioning latent representations.

[0207] Clause 23. The method of any of Clauses 19-22, wherein respective training time series of data elements comprise one data element.

[0208] Clause 24. The method of any preceding Clause, wherein the output time series of data elements is a video, and wherein each data element is a video frame.

[0209] Clause 25. The method of Clause 24, wherein the input comprises a conditioning input, and wherein the conditioning input comprises an embedding of text describing the video.

[0210] Clause 26. The method of any of Clauses 24-25, wherein the decoder neural network is configured to generate one or more video frames given latent representations of the one or more video frames.

[0211] Clause 27. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any of Clauses 1-26.

[0212] Clause 28. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any of clauses 1 to 26.

Claims

1. A computer-implemented method for generating an output time series of data elements conditioned on an input, the method comprising: obtaining the input, wherein the input comprises a noisy input comprising a plurality of potential representations of the output time series; Each latent representation is updated using a latent denoising neural network, the update comprising: The latent representation is updated at each of a plurality of iterations, including at each iteration: processing an intermediate input of the iteration including the latent representation to generate a denoised output, the processing comprising: generating a sequence of tokens each representing a corresponding patch of the latent representation; generating an encoded representation of the sequence of word-grams, the encoded representation comprising the sequence of input word-grams; processing the encoded representation to generate a merged sequence of output tokens, wherein the number of output tokens in the merged sequence is less than the number of input tokens in the sequence of input tokens; processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence, the updated merged sequence including a respective updated output word-gram for each of the output word-grams in the merged sequence; processing the updated merged sequence to generate an expanded representation comprising the same number of word-grams as the number of input word-grams in the sequence of input word-grams; and processing a decoder input derived from the extended representation to generate the denoised output; updating the latent representation using the denoised output of the iteration; and An output time series of data elements is generated by processing each updated latent representation using a decoder neural network.

2. The method of claim 1 , wherein the latent denoising neural network is configured to autoregressively update the latent representation, and wherein for each latent representation other than the first latent representation, the intermediate input comprises one or more previously updated latent representations.

3. The method of any preceding claim, wherein processing the encoded representation to generate a merged sequence of output tokens comprises: generating a plurality of groups of input word-grams from the sequence of input word-grams, wherein each group includes a plurality of input word-grams; For each group, concatenate the input word-grams of the group; as well as For each group, a linear transformation is applied to the concatenated input tokens to generate an initial output token.

4. The method of claim 3, further comprising: For each group, adding an average patch value to the initial output word-grams to generate output word-grams for the group, wherein the average patch value is an average of the input word-grams for the group; and The output word-grams of each group are included in a merged sequence of the output word-grams.

5. The method of any one of claims 3 to 4, wherein processing the updated merged sequence to generate the extended representation comprises: For each output word-gram in the updated merged sequence, applying a second linear transformation to the output word-gram to generate a corresponding representation of the output word-gram; as well as For each corresponding representation, the corresponding representation is split into groups of a plurality of initial patch tokens, wherein the groups correspond to the output tokens.

6. The method of claim 5, further comprising: For each initial patch word-gram in each group, adding the corresponding output word-gram of the group to the initial patch word-gram to generate an updated patch word-gram; as well as The updated patch word-grams of each group are included in the expanded representation.

7. The method of any preceding claim, wherein the latent representation has a time dimension and two spatial dimensions, and wherein processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence comprises: dividing the output tokens of the merged sequence into a plurality of spatial groups of output tokens, wherein each spatial group includes output tokens of the same time index for the time dimension; as well as Each spatial group of output tokens is processed using a sequence of the neural network blocks to update each output token, wherein each neural network block is configured to apply attention to the output tokens of the spatial group using keys, values, and queries derived from the output tokens.

8. The method of claim 7 , wherein processing the merged sequence using a sequence of neural network blocks to generate an updated merged sequence comprises: dividing the output tokens of the merged sequence into a plurality of spatiotemporal groups of output tokens, wherein each spatiotemporal group comprises output tokens whose time indices are from the same spatial region across the time dimension; as well as Each spatiotemporal group of output tokens is processed using a sequence of the neural network blocks to update each output token, wherein each neural network block is configured to apply attention to the output tokens of the spatiotemporal group using keys, values, and queries derived from the output tokens.

9. The method of any preceding claim, wherein each neural network block is configured to: receiving a block input of the neural network block comprising a merged sequence of the output word-grams; For each output token, generate a rotated position embedding using the rotation matrix; as well as The output tokens and the rotated position embeddings are processed by applying one or more attention mechanisms to generate a block output of the neural network block, the block output comprising a corresponding updated output token for each of the output tokens in the merged sequence.

10. A method as claimed in any preceding claim, wherein the decoder input comprises a combination of the extended representation and the encoded representation.

11. A method as claimed in any preceding claim, wherein the input further comprises a conditioning signal, and wherein the intermediate input further comprises the conditioning signal.

12. The method of claim 11 , wherein each neural network block is configured to apply attention to the conditioning signal and the output tokens of the merged sequence to update the output tokens of the merged sequence using keys and values ​​derived from the conditioning signal and queries derived from the output tokens of the merged sequence.

13. A method as claimed in any preceding claim, wherein obtaining the input comprises sampling the noise input from a noise distribution.

14. The method of any preceding claim, wherein generating the sequence of word-grams each representing a respective patch of the latent representation comprises: generating a corresponding patch embedding for each corresponding patch of the latent representation; generating one or more corresponding position embeddings for each of the corresponding patch embeddings; as well as The sequence of word-grams is generated by combining, for each respective patch embedding, the respective patch embedding and the one or more corresponding position embeddings.

15. The method of claim 14, wherein the one or more corresponding position embeddings are derived from a spatial position embedding and a temporal position embedding.

16. A method as claimed in any one of claims 14 to 15, wherein each respective patch comprises a spatiotemporal region over one or more data elements represented by the latent representation.

17. A method as claimed in any one of claims 14 to 15, wherein each respective patch comprises a spatial region of a particular data element represented by the latent representation.

18. A method as claimed in any preceding claim, wherein the decoder neural network has been trained and frozen prior to training the latent denoising neural network.

19. A method as claimed in any preceding claim, wherein the latent denoising neural network has been trained by repeating the following operations: obtaining a training example comprising a training time series of data elements; generating one or more ground truth latent representations for the training time series of data elements using an encoder neural network; Noise is added to each ground truth latent representation to generate training inputs; For each training input: generating a training denoised output from the training input by processing a training intermediate input including the training input using the latent denoising neural network according to current values ​​of parameters of the latent denoising neural network; and Updates to the parameters of the underlying denoising neural network for optimizing a training objective are determined.

20. The method of claim 19, wherein the training objective measures the error between the baseline true latent representation and a denoised representation generated using the training denoised output.

21. The method of any one of claims 19 to 20, further comprising fine-tuning the latent denoising neural network.

22. The method of any one of claims 19 to 21, wherein the training examples further comprise training conditioning inputs, and wherein the training conditioning inputs comprise one or more conditioning latent representations.

23. A method as claimed in any one of claims 19 to 22, wherein the corresponding training time series of data elements comprises one data element.

24. A method as claimed in any preceding claim, wherein the output time series of data elements is a video, and wherein each data element is a video frame.

25. The method of claim 24, wherein the input comprises a conditioned input, and wherein the conditioned input comprises an embedding of text describing the video.

26. The method of any one of claims 24 to 25, wherein the decoder neural network is configured to generate the one or more video frames given a latent representation of the one or more video frames.

27. A system comprising: One or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the respective method of any one of claims 1 to 26.

28. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 26.

Images