Autoregressive language model for video generation
By employing an autoregressive generative language model and a progressive training strategy, the problems of loss imbalance and error accumulation in long video generation are mitigated, enabling high-quality generation of minute-level videos and solving the problem of visual quality degradation in long video generation in existing technologies.
Patent Information
- Application Number
- CN202510953038.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-07-15
- Filing Date
- 2025-07-10
- Publication Date
- 2026-01-16
AI Technical Summary
Existing video generation models suffer from training loss imbalance and error accumulation when generating long videos, especially minute-long videos, leading to a decline in visual quality and difficulty in capturing complex events and maintaining consistency.
An autoregressive generative language model is adopted, combined with a progressive short-to-long training strategy and loss reweighting. Multi-stage training is used to alleviate loss imbalance, and video tokenizers and inference strategies are used to extend video length. Low-resolution training and super-resolution processing are combined to improve video quality.
It generates minute-long videos with a consistent look, large motion dynamics, and natural scene transitions, improving the temporal coherence and visual quality of video generation.
Smart Images

Figure CN121357384A_ABST
Abstract
Description
Cross Reference to Related Applications This disclosure claims priority to U.S. Application No. 18 / 773,329, filed July 15, 2024, the contents of which are incorporated by reference in their entirety. BACKGROUND
[0001] Categories of video generation methods include generative adversarial network (GAN)-based methods, diffusion-based methods, and language model-based methods. Among them, diffusion-based methods have recently attracted much attention. Most diffusion-based methods encode videos into latent space for efficient training and utilize progressive inference strategies to generate videos with high spatio-temporal resolution. Recently, language models have been used for visual generation, with a focus on tokenizing visual data into a form that these models can process. Quantization techniques are commonly used, and transformers are employed to model the resulting tokens.
[0002] For image generation, autoregressive or masked transformers are prevalent. In short video generation, image-level or video-level tokenizers are used, which contain spatio-temporal compression and causal structure. However, short video generation models typically focus on making clips (e.g., videos under 10 seconds in length, often 1-3 seconds in length), which limits their ability to capture complex events and remain consistent over longer periods of time. Previous work has explored using various methods to generate long videos (e.g., videos longer than ten seconds). Recently, video diffusion models have been extended to long video generation. Some methods focus on sampling noise vectors and aggregating overlapping short video segments, respectively. Other methods propose an autoregressive approach that uses memory blocks to maintain consistency and appearance. In the language model domain, approaches have been envisioned that use masked video transformers to generate variable length videos. SUMMARY
[0003] Implementations are provided for autoregressively generating videos using a video generation model. One aspect includes a method comprising: performing a progressive multi-stage training process comprising a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation; and the second stage comprises further training the video generation model using a training dataset comprising labeled video-text pairs, wherein further training the video generation model comprises: for each labeled video-text pair: generating at least one text token using a text tokenizer and a text annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and a video of the labeled video-text pair; autoregressively generating frame tokens using the at least one text token; and training the video generation model using a loss value computed from the frame tokens and the video tokens.
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any portion of this disclosure. BRIEF DESCRIPTION OF DRAWINGS
[0005] Figure 1 A schematic diagram of an example computing system for training an autoregressive video generation model is shown.
[0006] Figure 2 A dataflow graph of an example training process for an autoregressive video generator is shown, which can be implemented using a computer system of Figure 1 .
[0007] Figure 3 A plot depicting training loss for different frame ranges is shown.
[0008] Figure 4 A process flow diagram of an example method for training an autoregressive video generation model is shown, which can be implemented using a computer system of Figure 1 .
[0009] Figure 5 A process flow diagram of an example method for generating a video using an autoregressive video generation model is shown, which can be implemented using a trained model of Figure 1 .
[0010] Figure 6 A schematic view of an example computing environment in which a computer system of Figure 1 can be implemented is shown. DETAILED DESCRIPTION
[0011] As described above, video diffusion models have been extended to long video generation. Some methods focus on sampling noise vectors and aggregating overlapping short video segments, respectively. Other methods propose an autoregressive approach that uses memory blocks to maintain consistency and appearance. In the language model domain, approaches have been envisioned to use masked video transformers to generate variable length videos. However, despite these advances, generating long videos with rich motion dynamics, consistent appearance, and high visual quality in open domains remains a challenge. Video generation models, including diffusion-based methods and language model-based methods, have strong capabilities in generating short videos (e.g., videos with a duration of less than 10 seconds, typically 1-3 seconds in length). To capture more comprehensive content, it is desirable to generate long videos (e.g., videos with a duration of more than 10 seconds, and can include videos of minute level) with consistent appearance, greater motion dynamics, and natural scene transitions. Autoregressive generative language models have been successful in generating long and coherent text sequences, demonstrating their ability to capture long-range dependencies and complex temporal patterns. However, using autoregressive generative language models for video generation is limited to generating short videos of a few seconds.
[0012] In natural language processing, generative language models can be trained on long sequences and extended beyond the training length. However, training autoregressive generative language models on long video sequences or extending short video generators to generate long videos results in suboptimal performance for minute-level video generation. The main obstacle is that there is a large degree of redundancy between video tokens and strong inter-frame dependencies. The video tokens of the current frame largely depend on the tokens of the previous frames, which presents two challenges for long video generation. The first challenge is loss imbalance during training. When training with next-token prediction targets, it is much more difficult to predict early frame tokens from text prompts than to predict later frame tokens based on ground-truth tokens of previous frames. The imbalance in token difficulty at different stages leads to loss imbalance during training. The problem becomes more severe as the video length increases, with the cumulative loss of multiple simple tokens far exceeding the loss of a few difficult tokens and dominating the gradient direction. The second challenge involves error accumulation during inference. During training, the model predicts the next token based on the previous ground-truth token, but during inference it must predict the next token based on the previously predicted token. This difference between training and inference leads to error accumulation during inference. Due to the strong inter-frame dependencies between video tokens and the large number of video tokens, this error accumulation is non-negligible and can lead to a decrease in visual quality for long video inference.
[0013] In view of the above observations, embodiments of autoregressive generative language models for long video generation are provided. As described herein, new embodiments of autoregressive generative language model-based video generators can be employed to generate content-rich, coherent, and dynamic minute-level long videos. In some embodiments, the autoregressive generative language model-based video generator is trained on ten-second videos, and thus can generate ten-second videos. This capability can be extended to generate minute-level long videos from textual prompts.
[0014] The autoregressive generative language model-based video generators described herein generally include two components: a video tokenizer that compresses a video into a sequence of discrete video tokens, and an autoregressive generative language model that models the unified sequence of text tokens followed by video tokens through next-token prediction. The autoregressive generative language model can have a transformer architecture. To mitigate the problem of long video training loss imbalance, a progressive short-to-long training strategy can be implemented, with video length increasing stepwise. In addition, loss reweighting can be performed on early frames to prevent the model from being dominated by the multiple easy-to-difficult tokens in later frames. Inference strategies can be implemented, including video token re-encoding and sampling strategies, to further extend video length by iteratively generating the next frame from previously generated frames. To enable training and inference using longer videos, the autoregressive generative language model-based video generator can be implemented to employ low-resolution videos (e.g., 128x128 pixels). Super-resolution and refinement modules can be utilized to enhance the resolution and fine-grained details of the generated low-resolution videos.
[0015] Turning now to the drawings, autoregressive generative language model-based video generators and related embodiments are described in further detail. Figure 1 An example computing system 100 for training an autoregressive video generation model is shown. The example computing system 100 can utilize various types of computing devices and be implemented on multiple such devices, including mobile devices, smartphones, personal computers, laptop computers, computing servers, and the like. The example computing system 100 includes processing circuitry 102 and a memory 104 storing instructions that, during execution, cause the processing circuitry 102 to perform the various processes described herein.
[0016] The memory 104 stores an untrained video generation model 106. Various types of models can be implemented. In the depicted example, the untrained video generation model 106 is a model based on an autoregressive generative language model that is capable of text-to-video generation. The video generation model 106 includes a text tokenizer that converts text, such as a text prompt, into text tokens. A video tokenizer that is capable of encoding a video into discrete video tokens can also be used. In sum, the text tokens and the video tokens can be modeled as a unified sequence. The video generation model 106 can also include a transformer with only a decoder that enables video generation by autoregressively predicting video tokens from text tokens and, if any, previous video tokens. The text tokenizer and the video tokenizer can be implemented in a variety of ways. In some implementations, the video tokenizer utilizes a causal 3D convolutional neural network (CNN) architecture to provide spatio-temporal joint compression and joint modeling of images and videos.
[0017] The untrained video generation model 106 is passed through a training module 108 that trains the model 106 to produce a trained video generation model 110. To extend the temporal coverage of a video within a limited number of tokens, the video generation model 106 can be configured to train and generate low-resolution videos and super-resolution can be performed during post-processing. In the depicted example, the training module 108 performs a multi-stage training process for long videos. As noted above, long sequence training suffers from a loss imbalance problem. During training, the model learns through next token prediction. In general, it is much easier to predict tokens for later frames from previous ground truth video tokens and text tokens. In contrast, it is more challenging to predict early frame tokens with little visual cues from previous frames. The cumulative loss of multiple easy-to-predict tokens in later frames outweighs the loss of a few hard-to-predict tokens in early frames and dominates the gradient direction, resulting in an undesirable visual quality of the generated video.
[0018] To mitigate the aforementioned challenge of loss imbalance, training module 108 implements a multi-stage progressive short-to-long training strategy. This strategy allows model 106 to first learn the text-conditioned appearance and motion of short videos, then smoothly scale to longer-range dependencies and more complex motion patterns in longer videos. In the depicted example, training module 108 performs training in three stages, with the length of the training videos progressively increasing. In the first stage training process 112, model 106 performs text-to-image generation training, which helps the model lay a solid foundation for modeling the appearance and structure of each frame. In the depicted example, the first stage training process 112 trains model 106 on a training dataset of labeled static image-text data 114. In the second stage training process 116, model 106 is trained on short video clips provided by video-text training dataset 118. In this stage, model 106 learns to capture short-term temporal dependencies and motion patterns while maintaining the visual quality of each frame. The short video clips can have various lengths. In some implementations, the short video clips have seventeen frames. In the third training phase 120, model 106 is trained on long videos from the long video-text training dataset 122. The long videos in the long video-text training dataset 122 can be any video longer than the video clips used during the second training phase 116. In some embodiments, the long video-text training dataset 122 includes long videos with sixty-five frames. In some embodiments, the long video-text training dataset 122 includes long videos with a duration of at least ten seconds.
[0019] Figure 2 A data flow diagram of an example training process 200 for an autoregressive video generator is shown. The example training process 200 can be, for example... Figure 1 The training module 108 is used for implementation. The overall framework includes a text tokenizer 202 for encoding the input text 206 and a video tokenizer 204 for encoding the input video frames 208. The text tokenizer and video tokenizer can be implemented in various ways. In some implementations, the video tokenizer 204 implements a causal 3D CNN architecture, which provides spatiotemporal joint compression and joint modeling of images and videos. The encoded spatiotemporal features can be quantized into discrete tokens. The performance of the video tokenizer 204 can depend on its implementation and training. For example, in some implementations, the video tokenizer 204 can compress a ten-second video (65 frames, 128×128 resolution per frame) into a sequence of 17×16×16 discrete tokens with a vocabulary size of 8192.
[0020] The framework also includes an autoregressive generative language model 210 with a decoder-only transformer that regressively predicts subsequent video tags based on text tags and (if any) previous video tags. By converting video frames 208 into discrete tags via a video tokenizer 204, text and video tags can be modeled as a unified tag sequence for video generation. In some implementations, causal attention is applied to the unified tag sequence (i.e., all tags). Text-to-video generation can be performed by using a decoder-only transformer to autoregressively predict video tags based on text tags. For simplicity, specific individual tags are omitted in the following formulas. Let t = {t1, t2, ..., t...} N Let} represent a sequence of text tokens, where N is the number of text tokens. Let v = {v1, v2, ..., v...} L} represents the video tag sequence, where L is the number of video tags. The autoregressive generative language model 206 models the uniform tag sequence s = [t; v] and is trained using the loss to predict the next tag for each video tag: Among them, v i Let v represent the i-th marker in the video sequence v, and v <i Indicates v i All previous video tags.
[0021] Video generation models trained on short video clips (e.g., videos less than ten seconds long) are typically limited in their ability to capture long-term dependencies and complex dynamics in longer videos (e.g., videos at least ten seconds long). Theoretically, training these models on longer videos would enable them to learn and generate more coherent and context-rich video content. However, training directly on long videos results in poor performance, even with extensive iterative training. During training, the model learns through next-tag prediction, where predicting tags for later frames based on previous ground truth video tags and text tags is much easier. In contrast, predicting tags for earlier frames is more challenging when visual cues from previous frames are scarce.
[0022] Figure 3A plot depicting training loss for different frame ranges is shown. Loss curves for different frame ranges are shown when training on a video with sixty-five frames (with 4,356 labels, covering ten seconds). As shown, labels in early frames (frames 1-17) have greater loss than labels in later frames. The average loss for labels from frames 50-65 is the smallest. This loss imbalance is a problem for long sequence training, as the cumulative loss of multiple easy-to-predict labels in later frames (e.g., frames 18-65) outweighs the loss of a few hard-to-predict labels in early frames (e.g., frames 1-17) and dominates the gradient direction, resulting in an undesirable visual quality of the generated video.
[0023] To mitigate the challenge of the aforementioned video label difficulty imbalance, a progressive short-to-long training strategy with loss reweighting can be implemented. Returning to Figure 2 , the example training process 200 implements a progressive short-to-long training scheme that is divided into three stages with increasing video length. The multi-stage training scheme allows the model 206 to first learn the text-conditioned appearance and motion of short videos, and then smoothly adjust to longer-range dependencies and more complex motion patterns in longer videos. In stage 1, the model 206 is trained on text-to-image generation on a large static image dataset, which helps the model lay a solid foundation for modeling the appearance and structure of each frame. In stage 2, the model 206 continues to be jointly trained on images and short video clips of seventeen frames, where the model learns to capture short-term temporal dependencies and motion patterns while maintaining the visual quality of each frame. In stage 3, the number of video frames is increased to sixty-five frames, covering a time range of ten seconds. Other training schemes can also be implemented. For example, different types and numbers of stages can be implemented. Stage training of different frame lengths can be implemented. In some implementations, the training process is a progressive training scheme with two stages.
[0024] In some implementations, the example training process 200 is implemented with a loss reweighting scheme, which helps train on long videos (e.g., ten-second videos). To further reinforce the supervision of early frames and prevent the model 206 from forgetting the priors from stage 1 and stage 2, a loss reweighting scheme can be applied in stage 3. In some implementations, greater loss weights are applied to labels of early frames. In one example, the overall weighted loss is formulated as: where the first term represents the loss of K labels corresponding to early frames (e.g., frames 1-17), and the second term represents the loss of L-K labels corresponding to later frames (e.g., frames 18-65). λ is a positive value that is used to reinforce the loss weight of early frames.
[0025] With loss weighting and progressive training strategies, model 206 can effectively mitigate the long video training issues discussed above. Since model 206 is trained over a ten-second time horizon, it can generate ten-second long videos with better temporal coherence and consistency, while maintaining strong appearance and motion priors learned from images and short video clips. In some implementations, an inference strategy is applied to extend the length of the generated videos to the minute level. Various generative language models have the capability of length generalization, and it can be expected that a generative language model-based video generator trained on ten-second videos can be extended to generate longer videos autoregressively. However, generalization beyond the training video length is not trivial and can lead to error accumulation and quality degradation. For example, a one-minute video corresponds to a much longer sequence of video tokens than most text sequences typically encountered in language modeling tasks (e.g., approximately 26,112 video tokens under the settings of Figure 2 The considerable length and strong inter-frame dependency between video tokens pose challenges to extending a generative language model-based generator to long video generation.
[0026] One way to extend videos beyond the training length is to leverage the strengths of autoregressive language models to iteratively generate the tokens of the next video clip given a textual prompt and the previously generated tokens of the current video clip. However, this strategy can lead to video quality degradation for video frames outside the training range. This problem stems from token misalignment caused by the causal video tokenizer. The tokens of the last n frames in a video clip are derived based on the context of all previous frames, while the tokens of the first n frames in a new video clip are derived without the context of the previous video clip. Thus, directly generating the tokens of the new clip from the previous tokens causes a shift in the distribution of the input features of the generative language model. To address this issue, the video tokens generated by the generative language model can be decoded into pixel space videos, and the video tokenizer can be used to re-encode the last n frames. The re-encoded video tokens and the textual tokens are used as conditions to generate the tokens of the next video clip.
[0027] Due to the autoregressive nature of the model and the strong inter-frame dependency of video tokens, error accumulation can easily occur when using autoregressive language models to decode video tokens. Errors made when predicting one token can propagate and affect the generation of subsequent tokens, leading to video quality degradation as the length increases. To mitigate this issue, a top-k sampling strategy can be employed. During token sampling, the top-k most likely tokens can be sampled, thus reducing the impact of potential errors on the generation of subsequent tokens, mitigating the error accumulation problem. A too small value of k (e.g., k = 1) results in a video that is almost static with little motion. To balance dynamic motion and error accumulation, a larger value of k (e.g., k = 50) can be chosen.
[0028] The video tokenizer 204 and the model based on a generative language model 206 can be configured to operate on various resolutions. To enable training and inference using longer videos, low resolution can be utilized. This design trades spatial resolution for longer video sequences during training and inference. In some implementations, the video tokenizer 204 and the model based on a generative language model 206 operate on a resolution of 128 x 128. During inference, post-processing techniques can be applied to enhance the spatial resolution of the generated videos (e.g., super-resolution techniques) without affecting the content and motion of the videos.
[0029] Figure 4 A process flow diagram of an example method 400 for training an autoregressive video generation model is shown. Various types of video generation models can be utilized. In some implementations, a model based on an autoregressive generative language model is implemented. The example method 400 implements a progressive multi-stage training process that includes multiple stages. Any number of stages can be implemented. In some implementations, the progressive multi-stage training process includes at least two stages. In further implementations, the progressive multi-stage training process includes at least three stages. Each stage can be configured to train the video generation model on different datasets having images and / or videos of different temporal lengths.
[0030] The example method 400 includes a first stage in step 402 in which the video generation model is trained to perform text-to-image generation using a first training dataset. The first training dataset can be implemented in various ways. In some implementations, the first training dataset includes labeled image-text pairs. For example, the first training dataset can include images paired with annotations that describe their respective images. During training, these annotations provide text prompts to the video generation model and the associated images provide ground truth labels.
[0031] The example method 400 includes a second stage in step 404 in which the video generation model is trained using a second training dataset that includes labeled video-text pairs. Each labeled video-text pair includes a video with an associated text annotation. The labeled video-text pairs can include videos having similar temporal lengths. In some implementations, the labeled video-text pairs include videos having seventeen frames.
[0032] A training loop can be performed for each labeled video-text pair. The training loop using the second training dataset can include generating at least one text token using a text tokenizer and the text annotation of the labeled video-text pair. The text tokenizer can be implemented to parse the input text and split it into text tokens. Any type of text tokenizer can be used. The training loop using the second training dataset can also include generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair. The video tokenizer can be implemented to encode and compress the input video frames into video tokens. In some implementations, the number of tokens output by the video tokenizer is less than the number of input frames. In some implementations, the video tokenizer 204 can compress a ten second video (65 frames, 128x128 resolution per frame) into a sequence of 17x16x16 discrete tokens with a vocabulary size of 8192. Any type of video tokenizer can be used. For example, the video tokenizer can include a trained CNN architecture.
[0033] The training loop using the second training dataset can also include autoregressively generating a plurality of frame tokens from the at least one text token. In some implementations, a decoder-only transformer is used to autoregressively predict the next video token based on a sequence of tokens, which can include text tokens and previous video tokens. For example, the process can include predicting a first frame based on a text token. Successive frames can be autoregressively predicted based on the text token and previously generated frames. The video generation model can then be trained using a loss value computed from the plurality of frame tokens and the video tokens.
[0034] The example method 400 includes a third stage in step 406, which includes training the video generation model using a third training dataset including labeled long video-text pairs. The third stage is similar to the second stage, but uses a different dataset with longer duration videos. In some implementations, the training process using the third training dataset includes a loss reweighting scheme. For example, a greater loss weight can be applied to tokens of early frames compared to later frames. Various types of training datasets can be utilized. In some implementations, the labeled long video-text pairs include long videos, each of which has more frames than the videos used in the second training dataset. In further implementations, each long video has sixty-five frames. Alternatively, the long videos can be described in terms of duration. In some implementations, the labeled long video-text pairs include long videos each of which has a duration of at least ten seconds.
[0035] Figure 5A process flow diagram showing an example method 500 for generating a video using an autoregressive video generation model. Various types of video generation models can be utilized. In some implementations, an autoregressive generative language model-based model trained by the method 400 is implemented. The example method 500 includes receiving a text prompt in step 502. In step 504, the example method 500 includes autoregressively generating a video using the text prompt and the video generation model. Generating a video using an autoregressive video generation model can be performed in a variety of ways. In some implementations, the autoregressive video generation model uses a text tokenizer to generate text tokens from the text prompt. The text tokens can be used to autoregressively generate a video. For example, a first frame can be generated based on the text tokens, and successive frames can be autoregressively generated based on the text tokens and previously generated frames. Figure 4
[0036] In some implementations, the generated video is longer than the video on which the autoregressive video generation model was trained. For example, a video generated using an autoregressive video generation model trained on a sixty-five frame video can exceed one minute. Such a video can be generated in a variety of ways. In some implementations, additional tokens beyond the length of the training video can be autoregressively generated from the text tokens and previously generated video tokens. In some implementations, the generated tokens are decoded into a pixel space video, and then the last predetermined number of frames of the pixel space video are re-encoded using a video tokenizer. New video tokens can then be generated based on the text tokens and the re-encoded video tokens.
[0037] The example method 500 optionally includes performing super-resolution processing on the generated video in step 506. In some implementations, the generated video initially has a resolution of 128 x 128 pixels. Any type of super-resolution processing can be utilized.
[0038] This disclosure describes implementations of autoregressive generative language model-based video generation models that can generate minute-long videos with consistent appearance, large motion dynamics, natural scene transitions. A progressive short-to-long training scheme with loss reweighting can be used to address and mitigate the challenges of long video training. Inference strategies are provided to extend the generated videos beyond the training length. The model can be deployed for various uses, including assisting visual artists and filmmakers in video creation, improving their efficiency.
[0039] In some embodiments, the methods and processes described herein can relate to the computational systems of one or more computing devices. In particular, such methods and processes can be implemented as computer-application or service, an application-programming interface (API), a library, and / or other computer-program product.
[0040] Figure 6 A non-limiting embodiment of a computing system 600 that can implement one or more of the methods and processes described above is illustrated schematically in simplified form. The computing system 600 is shown in simplified form. The computing system 600 can embody the computing system 100 described above and illustrated in FIG. 1. Components of the computing system 600 can be included in one or more personal computers, server computers, tablet computers, home-entertainment computers, network-computing devices, video-game devices, mobile- computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented-reality devices. Figure 1
[0041] The computing system 600 includes a logic processor 602, volatile memory 604, and a non-volatile storage device 606. The computing system 600 can optionally include a display subsystem 608, input subsystem 610, communication subsystem 612, and / or other components not shown in FIG. 6. Figure 6
[0042] The logic processor 602 includes one or more physical devices configured to execute instructions. For example, the logic processor can be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions can be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
[0043] The logic processor can include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logic processor can include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. Processors of the logic processor 602 can be single-core or multi-core, and the instructions executed thereon can be configured for sequential, parallel, and / or distributed processing. Individual components of the logic processor optionally can be distributed among two or more separate devices, which can be remotely located and / or configured for coordinated processing. Various aspects of the logic processor can be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
[0044] The non-volatile storage device 606 includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 606 can be transformed in accordance with the implementation.
[0045] The non-volatile storage device 606 can include physical devices such as optical memory, semiconductor memory, and / or magnetic memory, among others. The non-volatile storage device 606 can include nonvolatile, dynamic, static, read / write, read-only, sequential access, location- addressable, file-addressable, and / or content- addressable devices. It should be appreciated that the non-volatile storage device 606 is configured to retain instructions, even when power is cut to the non-volatile storage device 606.
[0046] The volatile memory 604 can include physical devices comprising random-access memory. The volatile memory 604 is typically used by the logic processor 602 to temporarily store information during processing software instructions. It should be appreciated that the volatile memory 604 does not continue to store instructions when the power is cut to the volatile memory 604.
[0047] Aspects of the logic processor 602, the volatile memory 604, and the non-volatile storage device 606 can be integrated into one or more hardware-logic components. Such hardware-logic components can include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
[0048] The terms "module," "program," and "engine" can be used to describe an aspect of the computing system 600 typically implemented in software by a processor to perform a particular functionality tied to a more specific machine that can be referenced in the implementation of a "module," "program," or "engine." As such, a module, program, or engine can be instantiated via logic processor 602 executing an
[0049] The display subsystem 608, when included, can be used to present a visual representation of data held by the non-volatile storage device 606. The visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein alter data held by the non-volatile storage device and thus transition the state of the non-volatile storage device, the state of the display subsystem 608 can likewise be transitioned to visually represent the alterations of the underlying data. The display subsystem 608 can include one or more display devices utilizing virtually any type of technology. Such display devices can be combined with the logic processor 602, volatile memory 604, and / or non-volatile storage device 606 in a shared enclosure, or such display devices can be peripheral display devices.
[0050] The input subsystem 610, when included, can comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, camera, or microphone.
[0051] The communication subsystem 612, when included, can be configured to communicatively couple various computing devices described herein to each other, and to other devices. The communication subsystem 612 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured for communication via a wired or wireless local- or wide-area network, broadband cellular network, or the like. In some embodiments, the communication subsystem can allow the computing system 600 to send and / or receive messages to and / or from other devices via a network such as the Internet.
[0052] The following paragraphs provide additional description of the subject matter of the present disclosure. One aspect provides a method for training a video generation model, the method comprising: performing a progressive multi-stage training procedure, wherein the progressive multi-stage training procedure comprises a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs, each labeled video-text pair comprising a video and a textual annotation, wherein further training the video generation model using the second training dataset comprises, for each labeled video-text pair: generating at least one text token using a text tokenizer and the textual annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair; generating a plurality of frame tokens from regressively from the at least one text token; and training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens. Training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens. In this aspect, additionally or alternatively, the progressive multi-stage training procedure further comprises a third stage comprising further training the video generation model using a third training dataset comprising labeled long video-text pairs. In this aspect, additionally or alternatively, a loss reweighting scheme is applied during the third stage to apply greater loss weight to tokens of early frames compared to tokens of late frames. In this aspect, additionally or alternatively, the video tokenizer has been trained to perform temporal compression using a convolutional neural network architecture. In this aspect, additionally or alternatively, the labeled long video-text pairs comprise long videos having 65 frames. In this aspect, additionally or alternatively, the long videos have a resolution of 128x128, and wherein the video tokenizer can compress the long videos into a sequence of 17x16x16 discrete tokens, with a vocabulary size of 8192. In this aspect, additionally or alternatively, the videos of the labeled video-text pairs of the second training dataset have 17 frames.
[0053] Another aspect provides a computing system for training a video generation model, the computing system comprising: processing circuitry and a memory storing instructions that, when executed, cause the processing circuitry to: perform a progressive multi-stage training procedure, wherein the progressive multi-stage training procedure comprises a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs, each labeled video-text pair comprising a video and a textual caption, wherein further training the video generation model using the second training dataset comprises, for each labeled video-text pair: generating at least one text token using a text tokenizer and the textual caption of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair; autoregressively generating a plurality of frame tokens using the at least one text token; and training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens. In this aspect, additionally or alternatively, the progressive multi-stage training procedure further comprises a third stage comprising further training the video generation model using a third training dataset comprising labeled long video-text pairs. In this aspect, additionally or alternatively, a loss reweighting scheme is applied during the third stage to apply greater loss weight to tokens of early frames compared to tokens of late frames. In this aspect, additionally or alternatively, the video tokenizer has been trained to perform temporal compression using a convolutional neural network architecture. In this aspect, additionally or alternatively, the labeled long video-text pairs comprise long videos having 65 frames. In this aspect, additionally or alternatively, the long videos have a resolution of 128x128, and wherein the video tokenizer can compress the long videos into sequences of 17x16x16 discrete tokens having a vocabulary size of 8192. In this aspect, additionally or alternatively, the videos of the labeled video-text pairs of the second training dataset have 17 frames.
[0054] Another aspect provides a method of generating a video using a video generation model, the method comprising: receiving a textual prompt; autoregressively generating the video using the textual prompt and the video generation model, wherein the video generation model has been trained using a progressive multi-stage training process comprising a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs. In this aspect, additionally or alternatively, the labeled video-text pairs of the second training dataset comprise videos having 17 frames. In this aspect, additionally or alternatively, the progressive multi-stage training process further comprises a third stage comprising further training the video generation model using a third training dataset comprising labeled long video-text pairs comprising long videos having 65 frames. In this aspect, additionally or alternatively, autoregressively generating the video comprises: generating at least one text token using the textual prompt and a text tokenizer; generating a first frame token using the at least one text token; autoregressively generating successive frame tokens using previous tokens, wherein the previous tokens comprise at least the first frame token and the at least one text token; and decoding the first frame token and the successive frame tokens into the video. In this aspect, additionally or alternatively, autoregressively generating the video comprises: decoding the generated video tokens into a pixel space video; re-encoding a last predetermined number of frames of the pixel space video using a video tokenizer; and autoregressively generating successive frame tokens using at least one text token and the re-encoded last predetermined number of frames. In this aspect, additionally or alternatively, the method further comprises performing super-resolution processing on the video.
[0055] As used herein, “and / or” is defined as inclusive or, i.e., as including any and all combinations of one or more of the associated listed items, as specified by the following true- value table: A B A V B True True True True False True False True True False False False False
[0056] It is to be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. As such, various acts illustrated and / or described can be performed in the sequence illustrated and / or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes can be changed.
[0057] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems, and configurations disclosed herein, as well as any and all equivalents thereof.
Claims
1. A method for training a video generation model, the method comprising: performing a progressive multi-stage training process, wherein the progressive multi-stage training process comprises a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs, each labeled video-text pair comprising a video and a textual annotation, wherein further training the video generation model using the second training dataset comprises: for each labeled video-text pair in the labeled video-text pairs: generating at least one text token using a text tokenizer and the textual annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair; self-regressively generating a plurality of frame tokens using the at least one text token; and training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens.
2. The method of claim 1, wherein the progressive multi-stage training process further comprises a third stage comprising further training the video generation model using a third training dataset comprising labeled long video-text pairs.
3. The method of claim 2, wherein a loss reweighting scheme is applied during the third stage to apply greater loss weight to tokens of early frames compared to tokens of late frames.
4. The method of claim 2, wherein the video tokenizer has been trained to perform temporal compression using a convolutional neural network architecture.
5. The method of claim 2, wherein the labeled long video-text pairs comprise long videos having 65 frames.
6. The method of claim 5, wherein the long videos have a resolution of 128x128, and wherein the video tokenizer is capable of compressing the long videos into sequences of 17x16x16 discrete tokens having a vocabulary size of 8192.
7. The method of claim 1, wherein the videos of the labeled video-text pairs of the second training dataset have 17 frames.
8. A computing system for training a video generation model, the computing system comprising: processing circuitry and memory storing instructions that, when executed, cause the processing circuitry to: perform a progressive multi-stage training process, wherein the progressive multi-stage training process comprises a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs, each labeled video-text pair comprising a video and a textual annotation, wherein further training the video generation model using the second training dataset comprises: for each labeled video-text pair in the labeled video-text pairs: generating at least one text token using a text tokenizer and the textual annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair; self-regressively generating a plurality of frame tokens using the at least one text token; and training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens. the second stage includes further training the video generation model using a second training dataset comprising labeled video-text pairs, each labeled video-text pair comprising a video and a textual annotation, wherein further training the video generation model using the second training dataset comprises: for each labeled video-text pair in the labeled video-text pairs: generating at least one text token using a text tokenizer and the textual annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and the video of the labeled video-text pair; self-regressively generating a plurality of frame tokens using the at least one text token; and training the video generation model using a loss value computed from the plurality of frame tokens and the video tokens.
9. The computing system of claim 1, wherein the progressive multi-stage training procedure further includes a third stage, the third stage comprising further training the video generation model using a third training dataset comprising labeled long video-text pairs.
10. The computing system of claim 9, wherein a loss reweighting scheme is applied during the third stage to apply a greater loss weight to tokens of early frames compared to tokens of late frames.
11. The computing system of claim 9, wherein the video tokenizer has been trained to perform temporal compression using a convolutional neural network architecture.
12. The computing system of claim 9, wherein the labeled long video-text pairs comprise long videos having 65 frames.
13. The computing system of claim 12, wherein the long videos have a resolution of 128x128, and wherein the video tokenizer is capable of compressing the long videos into sequences of 17x16x16 discrete tokens having a vocabulary size of 8192.
14. The computing system of claim 8, wherein the videos of the labeled video-text pairs of the second training dataset have 17 frames.
15. A method of generating a video using a video generation model, the method comprising: receiving a textual prompt; self-regressively generating the video using the textual prompt and the video generation model, wherein the video generation model has been trained using a progressive multi-stage training procedure comprising a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation using a first training dataset comprising labeled image-text pairs; and the second stage comprises further training the video generation model using a second training dataset comprising labeled video-text pairs.
16. The method of claim 15, wherein the labeled video-text pairs of the second training dataset comprise videos having 17 frames.
17. The method of claim 16, wherein, The progressive multi-stage training process further includes a third stage that includes further training the video generation model using a third training dataset that includes labeled long video-text pairs, the labeled long video-text pairs including long videos having 65 frames.
18. The method of claim 15, wherein autoregressively generating the video comprises: generating at least one text token using the text prompt and a text tokenizer; generating a first frame token using the at least one text token; autoregressively generating successive frame tokens using previous tokens, wherein the previous tokens include at least the first frame token and the at least one text token; and decoding the first frame token and the successive frame tokens into the video.
19. The method of claim 15, wherein autoregressively generating the video comprises: decoding the generated video tokens into a pixel space video; re-encoding a last predetermined number of frames of the pixel space video using a video tokenizer; and autoregressively generating successive frame tokens using at least one text token and the re-encoded last predetermined number of frames.
20. The method of claim 15, further comprising: performing super-resolution processing on the video.
19. The method of claim 15, wherein autoregressively generating the video comprises: decoding the generated video tokens into a pixel space video; re-encoding a last predetermined number of frames of the pixel space video using a video tokenizer; and autoregressively generating successive frame tokens using at least one text token and the re-encoded last predetermined number of frames. performing super-resolution processing on the video.
19. The method of claim 15, wherein autoregressively generating the video comprises: decoding the generated video tokens into a pixel space video; re-encoding a last predetermined number of frames of the pixel space video using a video tokenizer; and autoregressively generating successive frame tokens using at least one text token and the re-encoded last predetermined number of frames. performing super-resolution processing on the video.
19. The method of claim 15, wherein autoregressively generating the video comprises: decoding the generated video tokens into a pixel space video; re-encoding a last predetermined number of frames of the pixel space video using a video tokenizer; and autoregressively generating successive frame tokens using at least one text token and the re-encoded last predetermined number of frames. performing super-resolution processing on the video.
Citation Information
Cited By
Natural language driven video generation method based on intention deconstruction
CN121815043A
A natural language driven video generation method based on intention deconstruction
CN121815043B
