Variable length video generation from text description

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating video. In one aspect, a method includes receiving a first text prompt; using the video generation neural network to generate an initial segment of the video on the condition of the first text prompt; and updating the video for each of the one or more update iterations by obtaining an additional textual cue for each update iteration and generating additional segments of the video using a video generation neural network on the condition of the textual cue for the update iteration.

Description

Background Art

[0001] This specification relates to using machine learning models to process data.

[0002] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model parameters.

[0003] Some machine learning models are deep models that employ multiple layers of a model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output. Summary of the invention

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that generates a video conditioned on one or more text inputs. The video includes a corresponding video frame at each time step in a plurality of time steps.

[0005] In particular, the system receives a first text prompt, for example, from a user of the system. The user may submit the text prompt in any of a variety of ways, for example, by typing text using an input device or by submitting an audio input that is transcribed by the system.

[0006] The system generates an initial segment of a video using a video generation neural network and conditioned on a first text prompt, the initial segment comprising a corresponding video frame at each of a plurality of initial time steps in the video.

[0007] Then, at each iteration in one or more update iterations, the system updates the video by generating an additional video segment for that update iteration, the additional video segment comprising a corresponding video frame at each of a plurality of time steps immediately following a last video frame in the video up to that update iteration.

[0008] To update the video at a given update iteration, the system obtains additional text cues for that update iteration and generates an additional video segment for that time step using a video generation neural network and conditioned on (i) the additional text cues for that update iteration and conditioned on (ii) one or more video frames that have already been generated (i.e., one or more video frames that were already in the video as of that update iteration).

[0009] Depending on how the system receives text prompts, the system can generate variable length videos based on text prompts in different ways. For example, the system can receive a single text prompt, and then use additional update iterations to extend the length of the generated video, while maintaining temporal coherence and relevance to the text prompts. As another example, before generating a video, the system can receive corresponding text prompts for each of a plurality of scenes in the video. Then, the system can associate a first text prompt with a first segment, and associate each update iteration with a corresponding received text prompt in the received text prompt. Then, the system can generate a coherent video comprising a plurality of scenes described by the text prompts. As another example, after generating a given segment of a video, the system can replay the segment (or the entire video so far) to the user. Then, the user can submit a new input, which specifies a new text prompt describing the desired content of the next video segment.

[0010] The system may then output the generated video, for example, by storing the video or providing the video for playback on a user device of the user who submitted the text prompt.

[0011] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0012] The system can generate long multi-scene videos from textual prompts while maintaining the temporal order of the generated videos. Unlike conventional methods for generating videos from textual descriptions that are limited to generating short clips of organized videos, the described system can generate multiple organized sequences of videos from multiple descriptions, which maintain order with each other (i.e., each generated video sequence is meaningful relative to the previously generated video sequence). By utilizing spatiotemporal encoding of video sequences, the described system is able to maintain better temporal order between video frames both within a single generated video sequence and between multiple generated video sequences. In addition, the described system can compress videos into fewer tokens per video than other systems that do not use the described technology. By utilizing spatiotemporal encoding, the described system is also able to learn more efficiently from available training data (including still image data), and thus can obtain better video generation for a given training effort. Therefore, the described system can generate variable-length videos based on the overall story told by the textual description. For example, the system can generate variable-length videos describing textual descriptions while keeping the number of video tokens to a minimum so that video tokens can be modeled within computational constraints (e.g.) by transformer neural networks or other sequence generation neural networks.

[0013] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example variable length video generation system.

[0015] Figure 2 is a flow chart of an example process for variable length video generation.

[0016] Figure 3 An illustration of generating variable length videos based on textual prompts.

[0017] Figure 4 is a block diagram of an example video generation neural network.

[0018] Figure 5 is a block diagram of an example prompt-conditioned neural network.

[0019] Figure 6 is a block diagram of an example word manager system.

[0020] Figure 7 is a diagram of encoding video frames into spatiotemporal tokens.

[0021] Figure 8 is a diagram of decoding spatiotemporal tokens into video frames.

[0022] Fig. 9 is a block diagram of an example video encoder neural network.

[0023] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0024] Figure 1 An example variable length video generation system 100 is shown. Variable length video generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0025] The variable length video generation system 100 is configured to generate a sequence 102 of video frames over a sequence of time steps based on a received sequence of text prompts 104. Typically, the sequence 102 of video frames includes one or more segments of video frames. A segment of video frames is a sequence of video frames shorter than the sequence 102 over a continuous sequence of time steps.

[0026] The system 100 generates a sequence of video frames 102 by generating a segment of video frames for each received text prompt from the sequence of text prompts 104. The system 100 can generate a variable length sequence of video frames 102 by processing additional text prompts after processing the initial sequence of text prompts 104 until the generated sequence of video frames 102 reaches a desired length. Thus, the video generated by the system 100 can be variable in length, i.e., can include a different number of video frames.

[0027] System 100 processes text prompt sequence 104 as context to inform the generation of video frame sequence 102 .

[0028] Typically, a textual hint is natural language text that describes the content of one or more video frames (eg, the content of a scene depicted in the one or more video frames).

[0029] As an example, the text prompt sequence 104 may be multiple repetitions of a single text prompt, and the system 100 may generate a video frame sequence 102 that depicts the scene described by the single text prompt and is a coherent sequence of the multiple generated video frame segments.

[0030] As another example, text prompt sequence 104 may include multiple different text prompts, and system 100 may generate video frame sequence 102 that depicts multiple scenes described by the different text prompts and is a coherent sequence of the multiple generated video frame segments.

[0031] The system 100 may receive a sequence of text prompts 104 and output a sequence of video frames 102 generated by any means suitable for completing the text-to-video generation task. For example, the system 100 may receive the sequence of text prompts 104 from a memory and may output the generated sequence of video frames 102 to the memory. As another example, the system 100 may receive the sequence of text prompts 104 from a user and may output the generated sequence of video frames 102 to the memory, or may send the generated sequence of video frames 102 for playback or storage on a user device.

[0032] In some implementations, the system 100 can operate interactively with a user to generate the video frame sequence 102 .

[0033] As an example, the system 100 may iteratively generate the video frame sequence 102 across multiple update iterations based on feedback from a user.

[0034] In this example, at a first update iteration, the system 100 may request and receive a first text prompt from a user, and generate a first segment of video frames.

[0035] In this example, at each subsequent update iteration, system 100 may display a previously generated video to the user, request additional text prompts from the user, and generate additional segments of video frames consistent with both the text prompts received from the user and the previously generated video within sequence 102.

[0036] In this example, the user can indicate that the system 100 should stop generating video frames, and the system 100 can then output the generated sequence of video frames 102 to a memory, or send the generated sequence of video frames 102 for playback or storage on a user device. In this example, the text prompt sequence 104 would refer to the complete sequence of text prompts received from the user in such an interactive operation.

[0037] System 100 includes a video generation neural network 110 that processes a text prompt 108 and an optional contextual video frame 106 to generate a video frame 112 .

[0038] Variable length video generation system 100 iteratively processes text cues 108 obtained from text cues sequence 104 across multiple update iterations to add corresponding generated video frames 112 to video frame sequence 102 .

[0039] At the first update iteration, the system 100 may generate an initial set of video frames 112 using only the text cues 108 obtained from the text cues sequence 104 for the first update iteration.

[0040] At each update iteration after the first update iteration, the variable length video generation system 100 processes the context video frames 106 obtained by the system 100 from the video frame sequence 102 for that update iteration, together with the text prompts 108 obtained by the system 100 from the text prompt sequence for that update iteration, to generate video frames 112 for that update iteration. At each update iteration after the first update iteration, the system 100 adds the generated video frames 112 to the video frame sequence 102 for use as the context video frames 106 in a later update iteration. In some implementations, at each update iteration after the first update iteration, the variable length video generation system 100 processes a predetermined number of context video frames 106 that have been most recently added to the video frame sequence 102 as of that update iteration.

[0041] System 100 or another training system can train video generation neural network 110 to process text prompts 108 to generate video frames 112 using any suitable methodology for training a conditional generative model using training data that includes pairs of example text prompt sequences 104 and example video frame sequences 102. Such training can be done using any suitable objective function that measures how well neural network 110 processes example text prompt sequences 104 to generate videos that mimic example video frame sequences 102.

[0042] As an example, the neural network 110 may have an architecture suitable for handling text-to-video generation as a sequence-to-sequence conversion task, such as a bidirectional transformer, and an appropriate objective function may be to maximize the likelihood of generating an example video frame sequence 102 given a text prompt sequence 104. An example architecture of the neural network 110 is described in more detail below.

[0043] Figure 2 2 is a flow chart of an example process 200 for variable length video generation. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a variable length video generation system (e.g., Figure 1 The variable length video generation system 100) can perform process 200.

[0044] The system receives a first text prompt (step 202).

[0045] The system generates an initial segment of the video using a video generation neural network and conditioned on the first text prompt (step 204). The initial segment includes a corresponding video frame at each time step in a plurality of initial time steps in the video.

[0046] Then, the system generates corresponding additional segments at each update iteration in one or more update iterations. In particular, at each update iteration, the system updates the video by generating additional video segments for the update iteration, the additional video segments including corresponding video frames at each time step of a plurality of time steps immediately following the last video frame in the video up to the update iteration.

[0047] In particular, at any given update iteration, the system can obtain additional text prompts for that update iteration. For example, the system can receive additional text prompts from a sequence of text prompts in memory. As another example, the system can request text prompts from a user and receive text prompts from the user.

[0048] The system then generates additional video segments for the update iteration using the video generation neural network and conditioned on the additional text prompt for the update iteration and conditioned on one or more video frames in the video up to the update iteration (step 206).

[0049] References below Figures 3 to 9 Describes in more detail the use of a video generation neural network to generate segments of a video.

[0050] Thus, the final video generated by the system includes an initial segment of the video, followed by additional video segments generated at one or more update iterations.

[0051] Figure 3 A diagram of an application of an implementation of the variable length video generation system 100 is shown.

[0052] In this example, the system 100 receives text prompts 302 and generates a segment of five video frames 304 conditioned on the received text prompts 302. As an example of interactive video generation based on feedback from a user, the system 100 may request text prompts from the user, receive text prompts 302 from the user, and generate video frames 304 based on the text prompts 302 provided by the user.

[0053] The system 100 receives the text prompt 306 and generates the next segment of five video frames 308 conditioned on the received text prompt 306 and the previous five video frames 304. Continuing with the example of interactive video generation based on feedback from the user, the system 100 may provide the previously generated video frames 304 for the user to review, request a new text prompt from the user, receive the text prompt 306 from the user, and generate the video frames 308 based on the text prompt 306 provided by the user and the previously generated video frames 304.

[0054] The system 100 receives the text prompt 310 and generates the next segment of five video frames 312 conditioned on the received text prompt 310 and the previous five video frames 308. Continuing with the example of interactive video generation based on feedback from the user, the system 100 may provide previously generated video frames 304 and 308 for the user to review, request a new text prompt from the user, receive the text prompt 310 from the user, and generate video frame 312 based on the text prompt 310 provided by the user and the previously generated video frames 304 and 308.

[0055] The system 100 eventually receives the text prompt 314 and generates the next segment of five video frames 316 conditioned on the received text prompt 314 and the previous five video frames 312. Continuing with the example of interactive video generation based on feedback from the user, the system 100 may provide previously generated video frames 304, 308, and 312 for the user to review, request a new text prompt from the user, receive a text prompt 314 from the user, and generate video frame 316 based on the text prompt 314 provided by the user and the previously generated video frames 304, 308, and 312.

[0056] Continuing with the example of interactive video generation based on feedback from a user, the system 100 provides previously generated video frames 304, 308, 312, and 316 for the user to review, requests a new text prompt from the user, receives an indication from the user that the system 100 should stop generating video frames, and finally outputs the sequence of generated video frames 304, 308, 312, and 316 to a memory or sends the sequence of generated video frames 304, 308, 312, and 316 for playback or storage on a user device.

[0057] For clarity, the set of text prompts 302, 306, 310, and 314 in this example forms a sequence corresponding to text prompt sequence 104, and individual text prompts from the set correspond to text prompts 108. The set of generated video frame segments 304, 308, 312, and 316 in this example forms a sequence corresponding to video frame sequence 102, and individual video frame segments from the set correspond to generated video frames 112. Video frames 308, 312, and 316 correspond to context video frames 106 when used to condition on the generation of the next video segment.

[0058] Figure 4 An example video generation neural network 110 is shown. Video generation neural network 110 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0059] In some implementations, the video generation neural network 110 may include a text embedding neural network 404 and a prompt-conditional neural network 402 .

[0060] The text embedding neural network 404 can process the text prompt 108 and generate an encoded representation 406 (also referred to as the encoded prompt 406) of the text prompt 108. The text embedding neural network 404 can have any architecture suitable for encoding text into a numerical value. For example, the text embedding neural network 404 can be an encoder of a text-to-text transformer (such as BERT or T5) that has been pre-trained to perform a text processing task (such as text prediction or text generation). In some implementations, the pre-trained text embedding neural network 404 remains frozen during the end-to-end training of the entire neural network 110. Keeping the first neural network frozen during the training of another neural network means that the parameters of the first neural network 404 are not modified during the training of the other neural network. In some implementations, the system 100 fine-tunes the pre-trained text embedding neural network 404 by keeping the network 404 frozen during the first part of the end-to-end training of the entire neural network 110, and then continuing to train the network 404 using the end-to-end training objective during the rest of the end-to-end training of the entire network 110.

[0061] The prompt-conditioned neural network 402 may process the encoded prompt 406 and the optional contextual video frame 106 and generate the video frame 112. The prompt-conditioned neural network 402 may include the following references Figure 5 Multiple component neural networks explained.

[0062] Figure 5 An example cue-conditioned neural network 402 is shown.

[0063] In some implementations, the prompt-conditional neural network 402 can include a word-gram prediction neural network 502 and a video decoder neural network 504 .

[0064] The word-gram prediction neural network 502 can process the encoded hint 406 and optional contextual video data to generate predicted video word-grams 508. As used throughout this specification, a video word-gram is a sequence of numerical values ​​that is part of an encoding of a description of one or more video frames. A sequence of video word-grams can describe a segment of a video frame.

[0065] The video decoder neural network 504 may process the sequence of predicted video tokens 508 to generate a video frame described by the predicted video tokens 508 .

[0066] The video decoder neural network 504 may generally have any suitable architecture that allows the video decoder neural network 504 to map a sequence of video tokens to a sequence of video frames. Figure 8 An example of the operations performed by a video decoder neural network is described in more detail.

[0067] In some implementations, the word-gram prediction neural network 502 can process the contextual video word-grams 510 as optional contextual video data and prompt the conditional neural network 402 to include a word-gram manager system 506. The word-gram manager system 506 can receive the contextual video frame 106, store a sequence of contextual video word-grams, receive and add the predicted video word-grams 508 to the sequence of contextual video word-grams, and output contextual video word-grams 510 from the stored sequence of contextual video word-grams.

[0068] The word-gram prediction neural network 502 may iteratively process the contextual video word-grams 510 and the encoded cues 402 to generate a sequence of predicted video word-grams 508 over a plurality of generative time steps.

[0069] At the first generative time step of the network 502, the word-gram manager 506 receives the context video frame 106, initializes the stored sequence of context video word-grams, and outputs a first set of context video word-grams 510 to the word-gram prediction neural network 502. The word-gram prediction neural network processes the first set of context video word-grams 510 and the encoded hints 402 to generate a first set of predicted video word-grams 508. The word-gram manager 506 adds the first set of predicted video word-grams 508 to the stored set of context video word-grams.

[0070] At each subsequent generative time step of the network 502, the word-gram manager 506 outputs the set of contextual video word-grams 510 for that generative time step to the word-gram prediction neural network 502. The word-gram prediction neural network processes the set of contextual video word-grams 510 for each subsequent generative time step and the encoded hints 402 to generate the set of predicted video word-grams 508 for that generative time step. The word-gram manager 506 adds the set of predicted video word-grams 508 for each subsequent generative time step to the stored set of contextual video word-grams.

[0071] At the first generative time step of the network 502, during the first update iteration of the system 100 when generating the initial video frame, the system 100 may not receive the context video frame 106. The word-gram manager 506 and the word-gram prediction neural network 502 may be configured to operate appropriately when the system 100 does not receive the context video frame 106. For example, the word-gram manager 506 may be configured to output a first set of context word-grams 510 with a predefined null value if the word-gram manager 506 does not receive the context video frame 506. As another example, the network 506 may be configured to process a variable number (possibly zero) of context word-grams 510, and during the first update iteration of the system 100, the word-gram manager 506 may omit outputting the first set of context word-grams 510.

[0072] In some implementations, the word-gram prediction neural network can process the received context video frame 106 as optional context video data.

[0073] The word-gram prediction neural network 502 may have any architecture suitable for text-conditioned word-gram prediction.

[0074] For example, the word-gram prediction neural network 502 can be a bidirectional transformer model that processes an input sequence including a sequence of contextual video words 510 and the encoded hints 402 to generate a sequence of predicted video words 508. As another example, the word-gram prediction neural network 502 can be a conditional bidirectional transformer model that processes an input sequence of contextual video words 510 conditioned on the encoded hints 402 to generate a sequence of predicted video words 508.

[0075] Typically, word-gram prediction neural network 502 is trained during the training of overall neural network 110 .

[0076] Figure 6 An example word manager system 506 is shown. The word manager system 506 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0077] In some implementations, the word-gram manager system 506 can store the sequence of masked video words 602 as a sequence of stored contextual video words. A masked video word-gram can be a video word-gram having a numerical value set to a predetermined value indicating that the word-gram is masked, or a masked video word-gram can be a video word-gram having an arbitrary numerical value with an additional numerical flag indicating that the word-gram is masked.

[0078] The masked word-gram sequence 602 has an autoregressive order in which the sequence 602 encodes the corresponding sequence of video frames. In some implementations, the word-gram manager system 506 stores the masked word-gram sequence 602 in the autoregressive order. In some implementations, the word-gram manager system may also store and process an additional value describing the autoregressive order that is included with or within the masked word-gram sequence 602.

[0079] The masked word-gram sequence 602 may include unmasked video words that have the same format as the masked words described, but lack an appropriate indication that the words are considered masked. In an implementation in which the word-gram manager system 506 stores a sequence of masked words, the word-gram manager may provide a subset of words from the sequence 602 for each generative time step of the word-gram prediction neural network 502 to be processed by the network 502 as contextual video words 510. At each generative time step of the word-gram prediction neural network 502, the word-gram manager system may add the predicted video words 508 generated by the network 502 to the stored sequence of contextual video words by storing the predicted video words 508 as unmasked video words in the sequence 602. When the word-gram manager 506 stores masked video words and unmasked contextual video words in the sequence 602, this is referred to as storing a combined sequence of masked words and contextual video words.

[0080] For the first generative time step of the network 502, the word-gram manager system 506 can initialize the word-gram sequence 602 to be a sequence consisting entirely of masked word-grams. If the word-gram manager system 506 receives the context video frame 106 for the first generative time step of the network 502, the word-gram manager can store the unmasked video word-grams that encode the context video frame 106 in the stored sequence 602, and can output a first set of context video word-grams 510 to the word-gram prediction neural network 502, the first set of context video word-grams including some or all of the unmasked video word-grams that encode the context video frame 106. The word-gram manager 506 can add the first set of predicted video word-grams 508 to the sequence 602 by replacing the masked word-grams within the sequence 602 with the first set of unmasked predicted video word-grams 508. For each subsequent generative time step of the network 502, the word-gram manager 506 can output a set of contextual video word-grams 510 for the generative time step to the word-gram prediction neural network 502, the set of contextual video word-grams including some or all of the unmasked video word-grams stored in the sequence 602. The word-gram manager 506 can add the set of predicted video word-grams 508 for each subsequent generative time step to the sequence 602 by replacing the masked word-grams within the sequence 602 with the set of unmasked predicted video word-grams 508 for the generative time step.

[0081] When the neural network 110 includes a word manager that stores masked words within the word sequence 602 and replaces those masked words with the predicted video words 508, the loss function used to train the entire network 110 can be any objective function suitable for measuring how well the word prediction neural network 502 processes the text embeddings of the example text prompts to predict the video word sequence corresponding to the example video frame sequence. For example, the loss function used to train the network 110 can be:

[0082] in is the set of indices of all masked tokens, and is given the set of all previously unmasked tokens and text embeddings of ground-truth example text prompts is assigned to the masked word by the word prediction neural network 502 in the case of The probability of a true value example .

[0083] In some implementations, the word-gram manager system 506 can include a video encoder network 604. The video encoder network 604 can process the context video frame 106 and generate corresponding video word-grams. The word-gram manager system 506 can add the output word-grams from the video encoder neural network as unmasked word-grams to the masked word-gram sequence 602.

[0084] In some implementations, the video encoder neural network 604 may perform spatiotemporal encoding of video frames, and the video decoder neural network 504 may perform spatiotemporal decoding of video tokens.

[0085] The spatiotemporal encoding of a segment of a video frame includes one or more spatial video tokens whose combined values ​​represent the initial frame of the video segment. The segment of a video frame is described by values ​​(e.g., RGB values) assigned to spatial regions (e.g., individual pixels or groups of pixels) of the video frame. The spatiotemporal encoding further includes a number of spatiotemporal tokens for each specific region of the video frame, which characterize how the region changes over time within the duration of the segment of the video frame.

[0086] To perform spatiotemporal decoding, the video decoder neural network 506 processes the sequence of spatiotemporal encoded video tokens and thereby produces a suitably corresponding sequence of video frames.

[0087] Figure 7A diagram showing how spatiotemporal video tokens encode information from video frame data. The diagram generally depicts how information represented by a sequence of video tokens is related to information represented within a sequence of video frames, and may not depict the exact mechanism used to create a sequence of spatiotemporal video tokens from a sequence of video frames.

[0088] A sequence of spatiotemporal video tokens 706A, 706B, 706C, etc. is generated by processing a sequence of video frames 702A, 702B, 702C, etc. Each spatiotemporal token encodes information about a particular region of a particular video frame, which will be referred to as the current frame of the token. For example, token 706A encodes information about region 704A within video frame 702A, and video frame 702A is considered the current frame of token 704A. Within a sequence of video tokens that encodes the entirety of a sequence of video frames, multiple subsequences of video tokens may be required, wherein tokens of the same subsequence encode information about shared regions of frames of the video sequence, and wherein tokens of different subsequences encode information about different regions of frames of the video sequence. Tokens 706A, 706B, 706C, etc. form such subsequences, wherein regions 704A, 704B, 706C, etc. are the same regions of different video frames. The sequence of video frames follows an ordering, typically but not necessarily capturing the relative time of each frame, such that each particular video frame in the sequence can be described as having a history that is the set of all video frames that includes that particular video frame and all video frames that occur earlier within the ordering of the video frame sequence. For example, frame 702A has a history that includes only frame 702A, frame 702B has a history that includes frames 702B and 702A, and frame 702C has a history that includes frames 702C, 702B, and 702A. The process of encoding video frame information into a particular word-element involves encoding video data from the same region of frames within the history of the current frame of the particular word-element. For example, word-element 706A encodes video data for region 704A, word-element 706B encodes video data for regions 704A and 704B, and word-element 706C encodes video data for regions 704A, 704B, and 704C. In this described sense, spatiotemporal video words are autoregressively dependent on the sequence of video frames. During encoding, an ordering (referred to herein as a word ordering) is provided for a spatiotemporal video sequence that depends at least on the ordering of the current frame of video words and optional additional information, which optional additional information may include a spatial ordering of regions within the encoded video frame. The word ordering may be explicitly represented as a numerical value included in or provided together with the sequence of video words, or may be implicitly represented by a sequential ordering of video words in a memory. A sequence of spatiotemporal video words need not exhaustively or losslessly encode a sequence of video frames in order to be considered as an encoding of a sequence of video frames. Thus, from a particular sequence of spatiotemporal video words that encode a particular sequence of video frames, a subset of the sequence of spatiotemporal video words may be considered to form a subsequence of video words that also encode the same particular sequence of video frames.

[0089] Figure 8A diagram showing how spatiotemporal video tokens may be decoded to produce a sequence of video frames. The diagram generally depicts how information represented by the sequence of video tokens is related to information represented within the sequence of video frames, and may not depict the exact mechanism used to create the sequence of video frames from the sequence of spatiotemporal video tokens.

[0090] A sequence of video frames 802A, 802B, 802C, etc. is decoded from a sequence of spatiotemporal video words 806A, 806B, 806C, etc. Information about a particular region within a decoded video frame is encoded within a particular video word. For example, information about region 804C is encoded within video words 806A, 806B, and 806C. For each decoded region within a video frame, there is a particular video word (referred to herein as the current word for that region) within the sequence of spatiotemporal video words, which may be considered the latest word for that region. Each decoded region within a video frame may be described as having a word history, which is a set of all spatiotemporal video words including the current word for that region and all video words that appear earlier in the word ordering of the spatiotemporal video word sequence. For example, region 804A has a word-meta history that includes only word-meta 806A, region 804B has a word-meta history that includes word-meta 806B and 806A, and region 804C has a word-meta history that includes word-meta 806C, 806B, and 806A. The process of decoding a particular region of video frame data involves decoding the video data encoded within the video word-meta within the word-meta history of the particular region. For example, region 804A is determined by processing information encoded within word-meta 806A, region 804B is determined by processing information encoded within word-meta 806B and 806A, and region 804C is determined by processing information encoded within word-meta 806C, 806B, and 806A. In this described sense, a video frame is autoregressively decoded from a sequence of spatiotemporal video word-meta.

[0091] Fig. 9 An example video encoder neural network 604 is shown. Video encoder neural network 604 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0092] In an implementation where the video encoder neural network 604 is configured to perform spatiotemporal encoding of video frames, the video encoder neural network 604 may include a frame tokenizer network 902, a spatial attention network 904, and a causal attention network 906, and may store a token codebook 910.

[0093] The frame tokenizer network 902 can process the context video frame 106 to generate a sequence of initial video tokens. The frame tokenizer can have any architecture suitable for image-to-sequence conversion. For example, the frame tokenizer network 902 can be an encoder of a Vision Transformer network.

[0094] The spatial attention network 904 can process a sequence of video tokens to generate a corresponding sequence of updated spatially attended video tokens. The spatial attention network 904 can have any architecture suitable for sequence-to-sequence conversion. For example, the spatial attention network 904 can be a transformer architecture that implements all-to-all attention. As a further example, the spatial attention network 904 can be a transformer architecture that implements all-to-all attention between tokens corresponding to the same video frame.

[0095] The causal attention network 906 can process a sequence of video tokens to generate a corresponding sequence of updated causally attended video tokens. The causal attention network 906 can have any architecture suitable for sequence-to-sequence conversion. For example, the causal attention network 906 can be a transformer architecture that implements all-to-all attention. As a further example, the causal attention network 906 can be a transformer architecture that implements all-to-all attention between tokens corresponding to the same spatial region. The token quantizer 908 may process a sequence of video tokens to generate a corresponding sequence of quantized video tokens, where each quantized video token is a codeword whose value is stored or represented in a token codebook 910. The token quantizer 908 may have any architecture suitable for performing vector quantization, such as in a vector quantized VAE (VQ-VAE) or in a vector quantized GAN (VQ-GAN).

[0096] The video encoder neural network 604 first processes the input context video frame 106 using the frame tokenizer network 902 to produce a sequence of initial video tokens. The video encoder neural network 604 processes the sequence of initial video tokens using the spatial attention network 904 to produce a sequence of spatially attended tokens. The video encoder neural network 604 processes the sequence of initial video tokens using the causal attention network 906 to produce a sequence of causally attended video tokens. The video encoder neural network 604 finally processes the sequence of causally attended video tokens using the token quantizer 908 to produce an output sequence of quantized video tokens 912. The video encoder neural network 604 can encode a still image by not using the causal attention network 906 to process the sequence of spatially attended tokens but using the token quantizer 908 to process the sequence of spatially attended tokens to produce an output sequence of quantized tokens.

[0097] In implementations where the video decoder neural network 504 is configured to perform spatiotemporal decoding of video frames, the video decoder neural network 504 may include a decoder causal attention network, a decoder spatial attention network, and a token decoder network.

[0098] The decoder causal attention network can process a sequence of spatiotemporally encoded video tokens to generate a corresponding sequence of updated causally attended video tokens. The decoder causal attention network can have any architecture suitable for sequence-to-sequence conversion. For example, the decoder causal attention network can be a transformer architecture that implements all-to-all attention. As a further example, the decoder causal attention network can be a transformer architecture that implements all-to-all attention between tokens corresponding to the same spatial region.

[0099] The decoder spatial attention network can process a sequence of causally attended video tokens to generate a corresponding sequence of updated spatially attended video tokens. The decoder spatial attention network can have any architecture suitable for sequence-to-sequence conversion. For example, the decoder spatial attention network can be a transformer architecture that implements all-to-all attention. As a further example, the decoder spatial attention network can be a transformer architecture that implements all-to-all attention between tokens corresponding to the same video frame.

[0100] The word decoder network can process the sequence of spatially attended video words to produce segments of video frames 112. The word decoder network can have any architecture suitable for sequence-to-video conversion. For example, the word decoder network can be a linear projection network. The video decoder neural network 504 first processes the input spatiotemporally encoded predicted video tokens 508 using a decoder causal attention network to produce a sequence of causally attended video tokens. The video decoder neural network 504 processes the sequence of causally attended video tokens using a decoder spatial attention network to produce a sequence of spatially attended tokens. The video decoder neural network 504 finally processes the sequence of spatially attended video tokens using a token decoder network to produce an output segment of a video frame 112. The video decoder neural network 504 can decode a still image by not processing the input token sequence using a decoder causal attention network but processing the input token sequence using a decoder spatial attention network to produce a sequence of spatially attended tokens.

[0101] The video decoder neural network 504 and the video encoder neural network 604 may be jointly pre-trained using any suitable methodology to perform image or video processing tasks. For example, the networks 504 and 604 may be jointly pre-trained to perform video reconstruction using a training set consisting of sample video sequences and using one or more objective functions suitable for measuring video reconstruction performance. As another example, the networks 504 and 604 may be jointly pre-trained to perform image reconstruction using a training set consisting of sample image sequences and using one or more objective functions suitable for measuring image reconstruction performance. In some implementations, the decoder network 504 and the encoder network 604 may remain frozen during the end-to-end training of the entire network 110. In some implementations, the system 100 may fine-tune the decoder network 504 and the encoder network 604 by keeping the networks 504 and 604 frozen during the first portion of the end-to-end training of the entire neural network 110, and then continuing to train the networks 504 and 604 using the end-to-end training objectives during the remainder of the end-to-end training of the entire network 110.

[0102] Objective functions suitable for image and video reconstruction may include distortion losses that measure the pixel-by-pixel error between the generated video frames and the sample video frames, such as the root mean square distance (or L2 distance). Objective functions suitable for image and video reconstruction may include divergence or perceptual losses that convincingly measure how well the generated video frames match the distribution of the sample video frames, such as Frechet Inception distance, Inception score, image perceptual loss, and video perceptual loss. In some implementations, a neural network called a discriminator can be trained along with the decoder network 504 and the encoder network 604 to classify the reconstruction as coming from the distribution of the reconstruction or from the distribution of the sample data. In implementations where the discriminator is trained along with the networks 504 and 604, a suitable objective function may include an adversarial loss that measures how accurately the discriminator is able to classify the reconstructed data. In implementations where the encoder network 604 employs vector quantization, a suitable objective function may include a vector quantization loss, such as the example loss used by VQ-VAE and VQ-GAN:

[0103] in is the word to be quantized, is the vector quantization of word units, is a predetermined committed loss weight, and is a stop gradient function that returns its input operand as a constant for the purpose of differentiation for backpropagation.

[0104] in represents the vector quantization loss, represents the L2 distance, represents the image perceptual loss, represents the video perceptual loss, Denoting the adversarial loss, an example objective function suitable for joint pre-training of the decoder network 504 and the encoder network 604 is as follows:

[0105] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0106] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), generating an artificially generated propagation signal to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0107] The term "data processing equipment" refers to data processing hardware, and includes all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also optionally include code that creates an execution environment for a computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0108] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages ​​or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0109] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0110] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0111] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for fulfilling or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or incorporated into the special logic circuit system. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from it or transfer data to it or both. However, the computer does not have to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0112] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0113] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application), and receiving a response message from the user in return.

[0114] A data processing device used to implement a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0115] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).

[0116] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, web browser, or application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0117] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from it. Data generated at the user device, for example, the result of a user interaction, can be received from the device at the server.

[0118] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be peculiar to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0119] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0120] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for generating a video, the video comprising a corresponding video frame at each time step in a sequence of time steps, the method comprising: receiving a first text prompt; generating, using a video generation neural network and conditioned on the first text prompt, an initial segment of the video, the initial segment comprising a corresponding video frame at each of a plurality of initial time steps in the video; as well as For each update iteration of one or more update iterations, updating the video by generating an additional video segment for the update iteration, the additional video segment comprising a corresponding video frame at each of a plurality of time steps immediately following a last video frame in the video up to the update iteration, the updating comprising: Obtaining additional text prompts for the update iteration; as well as The additional video segments are generated for the update iteration using the video generating neural network and conditioned on the additional text prompt for the update iteration and conditioned on one or more video frames in the video as of the update iteration.

2. The method of claim 1, wherein generating an initial segment of the video using a video generation neural network and conditioned on the first text prompt, the initial segment comprising a corresponding video frame at each of a plurality of initial time steps in the video, comprising: processing the first text prompt using a text embedding neural network to generate an encoded representation of the first text prompt; as well as The initial segment of the video is generated using the video generation neural network while the video generation neural network is conditioned on the encoded representation.

3. The method of claim 2, wherein the video generation neural network comprises a word-gram prediction neural network and a video decoder neural network, and wherein using the video generation neural network to generate the initial segment of the video while the video generation neural network is conditioned on the encoded representation comprises: generating a sequence of video tokens representing the initial segment using the token prediction neural network conditioned on the encoded representation of the first textual cue; as well as The sequence of video tokens is processed using the video decoder neural network to generate the video frames in the initial segment.

4. The method of claim 3, wherein the word-gram prediction neural network is configured to: receiving an input sequence of video tokens, wherein one or more of the video tokens in the input sequence are masked tokens; and The input sequence of video tokens is processed conditioned on the encoded representation of the textual prompt to generate a corresponding predicted token for each of the one or more masked tokens.

5. The method of claim 4, wherein generating a sequence of video tokens representing the initial segment conditioned on the encoded representation of the first textual cue using the token prediction neural network comprises: The sequence of video tokens is initialized to include only a sequence of masked tokens, and at each of a plurality of generation time steps: processing the sequence of video tokens conditioned on the encoded representation of the textual prompt to generate a corresponding predicted token for each of the masked tokens in the sequence; as well as The sequence is updated by replacing one or more of the masked word-grams with the corresponding predicted word-grams for the masked word-grams.

6. The method of any one of claims 3 to 5, wherein generating the additional video segment for the time step using the video generation neural network and conditioned on the additional text prompt for the update iteration and conditioned on one or more video frames in the video as of the update iteration comprises: processing the additional text prompt using the text embedding neural network to generate an encoded representation of the additional text prompt; as well as The additional segment of the video is generated using the video generation neural network when the video generation neural network is conditioned on the encoded representation of the additional text prompt and on one or more video frames in the video as of the update iteration.

7. The method of claim 6, wherein the video generation neural network further comprises a video encoder neural network, and wherein generating the additional segment of the video using the video generation neural network when the video generation neural network is conditioned on the encoded representation of the additional textual prompt and on one or more video frames in the video as of the update iteration comprises: Processing last K video frames in the video as of the update iteration using the video encoder neural network to generate context sequences representing video tokens of the last K video frames; generating, using the word-gram prediction neural network, conditioned on the encoded representation of the additional textual cues and the contextual sequence of video word-grams, an additional sequence of video word-grams representing the additional segments; and The additional sequence of at least video tokens is processed using the video decoder neural network to generate the video frames in the additional segments.

8. The method of claim 7 when dependent on any of claims 4 or 5, wherein using the word-gram prediction neural network to generate an additional sequence of video word-grams representing the additional segment conditioned on the encoded representation of the additional textual prompt and the contextual sequence of video word-grams comprises: Initializing a combined sequence of video tokens to include the context sequence of video tokens and a corresponding sequence of masked tokens for each video token in the additional sequence, and at each generation time step of a plurality of generation time steps: processing the combined sequence of video tokens conditioned on the encoded representation of the additional textual cue using the token prediction neural network to generate a corresponding predicted token for each of the masked tokens in the combined sequence; as well as The combined sequence is updated by replacing one or more of the masked word-grams with the corresponding predicted word-grams for the masked word-grams.

9. The method of claim 7 or claim 8, wherein the video encoder neural network is configured to receive an input video segment and process the input video segment to generate an output sequence of video tokens representing the video segment and comprising: One or more spatial words, the one or more spatial words representing a first frame in the input video segment independently of other frames in the input video segment; and A plurality of spatiotemporal word-grams each representing a corresponding spatial region in a corresponding set of a plurality of video frames, and the plurality of spatiotemporal word-grams autoregressively depending on a previous frame from the input video segment relative to the corresponding set of a plurality of video frames.

10. The method of claim 9, wherein the encoder neural network is configured to: generating a sequence of initial video tokens from the input video segment; processing the sequence of initial video tokens from the input video clip using one or more Transformer layers that apply all-to-all attention along a spatial dimension to generate a sequence of updated video tokens; processing the sequence of updated video tokens using one or more Transformer layers that apply causal attention along the temporal dimension to generate an initial output sequence of video tokens; as well as Quantization is applied to the initial output sequence of video tokens using the learned codebook to generate the output sequence of video tokens.

11. The method of any one of claims 9 or 10, wherein the word-gram prediction neural network is trained to perform text-conditioned word-gram prediction on a sequence of video word-grams generated by the video encoder neural network after the video encoder neural network has been trained.

12. The method of claim 11, wherein the word-gram prediction neural network is trained based on training examples, the training examples comprising: (i) video training examples, each of the video training examples comprising a video clip and a corresponding text prompt; and (ii) Image training examples, each of which includes only a single image and a corresponding textual prompt.

13. A method as claimed in any preceding claim, wherein the word-gram prediction neural network is a bidirectional Transformer.

14. A method as claimed in any preceding claim, wherein the first text prompt and each additional text prompt are the same text prompt.

15. A method as claimed in any preceding claim, wherein the first text prompt and the one or more additional text prompts comprise at least two text prompts that are different from each other.

16. The method of any preceding claim, further comprising: receiving an input image; and wherein an initial segment of the video is generated using a video generation neural network and conditioned on the first text prompt, the initial segment comprising a corresponding video frame at each of a plurality of initial time steps in the video, including: The initial segment of the video is generated using the video generation neural network while constraining a first image at a first time step in the video to be the input image and conditioned on the first text prompt.

17. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding method as claimed in any preceding claim.

18. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any preceding claim.

Images