Text-to-video generation

Abstract

A method and system for text-to-video generation. The method comprises the following steps: receiving text input; generating a representation frame based on the text input using a model trained on the text-image pair; generating a set of frames based on the representation frame and a first frame rate; interpolating the group of frames to a higher frame rate; generating a first video based on the interpolated set of frames; improving the resolution of the first video based on the first super-resolution model and the second super-resolution model; and generating an output video based on the results of the super-resolution models.

Description

Background Art Technical Field

[0001] The present disclosure generally relates to an adaptive artificial intelligence (AI) text-to-video generation model using text-to-image generation and motion learning. More specifically, the present disclosure includes generating videos from text without paired text-video data using a spatiotemporal pipeline.

[0002] Related technologies

[0003] Recently, large-scale generative networks have been applied to generate images from text, based on image generation frameworks trained on large-scale paired text-image datasets. Similarly, text-video generation frameworks can be trained on paired text-video datasets. However, text-video lags severely due to the lack of large-scale datasets with high-quality text-video pairs and the complexity of modeling higher-dimensional video data. Therefore, standard video generation focuses on simple domains and is trained on smaller (or private) text-video pairs. Therefore, a more accessible, adaptable, and faster video generation model is needed. Summary of the invention

[0004] The subject disclosure provides a system and method for text-to-video generation. In one aspect of the disclosure, the method includes: receiving a text input; generating a representation frame based on a text embedding of the text input; generating a set of frames based on the representation frame and a first frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; improving the resolution of the first video based on spatiotemporal information to generate a second video; and generating an output video by improving the resolution of the second video based on spatial information.

[0005] Another aspect of the present disclosure relates to a system configured for text-to-video generation. The system includes: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to perform a plurality of operations. The operations include: receiving a text input; generating a representation frame based on a text embedding of the text input; generating a set of frames based on the representation frame and a first frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; using a first super-resolution model to increase the resolution of the first video based on spatiotemporal information to generate a second video; and generating an output video by increasing the resolution of the second video based on spatial information using a second super-resolution model.

[0006] Another aspect of the present disclosure relates to a non-transitory computer-readable storage medium having instructions contained thereon, which can be executed by one or more processors to perform one or more methods for text-to-video generation. These instructions cause one or more processors to: receive text input; generate a representation frame based on a text embedding of the text input; decode a set of frames conditioned on the representation frame and a frame rate; interpolate the set of frames based on a second frame rate; generate a first video based on the interpolated set of frames; use a first super-resolution model to increase the resolution of the first video based on spatiotemporal information to generate a second video; and use a second super-resolution model to increase the resolution of the second video based on spatial information to generate an output video.

[0007] According to the present disclosure, these and other embodiments will be apparent.It should be understood that other configurations of the subject technology will be apparent to those skilled in the art according to the following specific embodiments, wherein various configurations of the subject technology are shown and described by way of example.As will be appreciated, the subject technology can have other and different configurations, and several details of these configurations can be modified in various other aspects, all of which do not depart from the scope of the subject technology.Therefore, the drawings and specific embodiments are essentially considered to be illustrative, rather than restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an environment in which the methods, apparatuses, and systems described herein may be implemented according to some embodiments.

[0009] Figure 2 is a diagram showing the Figure 1 A block diagram with details of the multiple devices used in the architecture.

[0010] Figure 3 is a block diagram illustrating a text-to-video generation framework of a text-to-video generation model for generating videos based on text input without requiring paired text-to-video data, according to some embodiments.

[0011] Figure 4 The architecture of a pseudo 3D convolutional layer is shown in accordance with one or more embodiments.

[0012] Figure 5 The architecture and initialization scheme of the pseudo 3D attention layer according to some embodiments are shown.

[0013] Figure 6 An example block diagram of a system for text-to-video generation is shown in accordance with some embodiments.

[0014] Figure 7 A flow chart of a method for text-to-video generation according to some embodiments is shown.

[0015] Figure 8 is a block diagram illustrating a computer system for performing, at least in part, one or more of the operations of the methods disclosed herein, according to some embodiments.

[0016] In the drawings, elements with the same or similar reference numerals are associated with the same or similar properties, unless explicitly stated otherwise. DETAILED DESCRIPTION

[0017] In the following detailed description, many specific details are set forth to provide a comprehensive understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the various embodiments of the present disclosure can be practiced without some of these specific details. In other instances, well-known structures and techniques are not shown in detail to avoid obscuring the present disclosure.

[0018] General Overview

[0019] Recently, large-scale generative networks have been applied to generate images from text, based on image generation frameworks trained on large-scale paired text-image datasets. Similarly, text-video generation frameworks can be trained on paired text-video datasets. However, the successful replication of text-image to video is largely limited by the lack of large-scale datasets of similar size (text, video) with high-quality text-video pairs, and by the complexity of modeling higher-dimensional video data. Limited text-video datasets also lead to misalignment between user text input and generated videos. Therefore, standard video generation focuses on simple domains and is trained on smaller (or private) text-video pairs (such as moving fingers or specific human actions). In addition, training a text-video model from scratch will significantly increase the cost of the framework, and will be an inadequate use of resources when a model capable of generating images already exists. Therefore, a more accessible, adaptable, and faster video generation model is needed that leverages text-image datasets to reduce the cost of the model.

[0020] Embodiments of the present disclosure describe a text-video generation framework that utilizes open source data sets used in text-image generation and unsupervised learning to enable networks to learn from orders of magnitude larger data to generate videos from text. According to various embodiments, the text-video generation model of the framework learns the correspondence between text and the visual world from paired text-image data, and uses unsupervised learning to learn real motion for unlabeled (unpaired) video data. The text-video generation model is pre-trained on images, and the text-video generation model is fine-tuned on videos. A large amount of text-image data is very important for learning the representation of more subtle and less common concepts in the world. The model pre-trained in this way produces much higher performance than the performance when trained separately in a supervised manner. The text-video generation framework according to various embodiments breaks the dependence of text-video generation on text-video pairs. This is a significant advantage because the output of text-video generation is not limited to narrow fields or does not require large-scale paired text-video data. Compared to, for example, freezing the weights, the text-to-video generation of each embodiment obtains the advantage of effectively adjusting the weights of the text-to-image model by fine-tuning on unlabeled video data. In addition, the use of pseudo 3D convolutional layers and temporal attention layers not only better utilizes the text-to-image architecture, but also allows for improved temporal information fusion.

[0021] According to various aspects of the embodiment, the text-video generation framework uses function-preserving transformations to extend the spatial layers in the model initialization phase to include temporal information. The extended spatiotemporal network includes a new attention module that can learn temporal world dynamics from a series of videos. This process significantly speeds up the text-video training process by instantly transferring knowledge from a previously trained text-image network to a new text-video network.

[0022] The text-image network includes convolutional layers and attention layers. The attention layer performs self-attention on the extracted features and can inject text information into several network levels along with other relevant information (e.g., diffusion time step). Using 3D convolutional layers is computationally intensive, and adding the time dimension to the attention layer will result in even more memory consumption. To address this issue, embodiments extend the dimensionality decomposition to each attention layer by stacking spatial attention layers and temporal attention layers.

[0023] In some embodiments, the text-video generation model decomposes the full-time U-Net and attention tensor and approximates them in space and time. The spatiotemporal pipeline is used to generate high-resolution and high-frame-rate videos through a video decoder, a frame interpolation model, and a super-resolution model. According to various embodiments, the super-resolution model and the frame interpolation model are trained to enhance the visual quality of the generated video. This improves the resolution of the generated video and achieves a higher (controllable) frame rate, thereby generating high-definition, high-frame-rate videos given text input.

[0024] Although various embodiments are described with reference to video generation, systems and methods consistent with the systems and methods described herein may be implemented for other tasks such as, but not limited to, image animation, video changes, and the like.

[0025] The embodiments disclosed herein provide a solution rooted in computer technology and derived from the field of text-video generation, that is, generating video from text without using paired text-video data. The disclosed subject technology also provides an improvement to the functionality of the computer itself, because it accelerates the training of the text-video model (because it does not need to learn visual and multimodal representations from scratch). In addition, the text-video model does not require paired text-video data. This allows the computer to use less computing resources and training time, ultimately reducing costs, and thus improving the technical field of text-video generation and user experience by generating high-quality spatiotemporal resolution videos that are faithful to the text input, while improving efficiency. In addition, the text-video generation model / method according to each embodiment also advantageously inherits the extensiveness of the image generation model (aesthetic diversity, fantasy depiction, etc.).

[0026] Example Architecture

[0027] Figure 1A network architecture 100 for implementing text-video generation according to some embodiments is shown. The architecture 100 may include multiple servers 130 and a database 152, which are communicatively coupled to multiple client devices 110 via a network 150. Any of the multiple servers 130 may be hosted on a platform / application (used by one or more of the multiple participants in the network) running on the client device 110. The multiple servers 130 may include a cloud server or a group of cloud servers. In some embodiments, the server 130 may not be cloud-based (i.e., the platform / application may be implemented outside of a cloud computing environment), or may be partially cloud-based. The client device 110 may include any of the following: a laptop computer; a desktop computer; or a mobile device, such as a smartphone, a handheld device, a video player, or a tablet device. In some embodiments, the client device 110 may include a headset or other wearable device (e.g., a virtual reality or augmented reality headset or smart glasses). The database 152 may store backup files from the platform, which include generated videos and inputs.

[0028] The network 150 may include, for example, any one or more of the following: a local area network (LAN); a wide area network (WAN); and the Internet, etc. In addition, the network 150 may include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star bus network, and a tree or hierarchical network, etc.

[0029] Figure 22 is a block diagram 200 showing details of a client device 110 and a server 130 used in a network architecture (e.g., architecture 100) as disclosed herein, according to some embodiments. The client device 110 and the server 130 are communicatively coupled via a network 150 via respective communication modules 218-1 and 218-2 (hereinafter collectively referred to as "communication modules 218"). The communication modules 218 are configured to interface with the network 150 to send and receive information, such as requests, uploads, information, and commands to other devices on the network 150. The communication module 218 can be, for example, a modem or an Ethernet card, and can include radio hardware and software for wireless communication (e.g., via electromagnetic radiation, such as radio frequency (RF), near field communication (NFC), Wi-Fi, and Bluetooth radio technologies). The client device 110 can be coupled to an input device 214 and an output device 216. A user can interact with the client device 110 through the input device 214 and the output device 216. The input device 214 may include a mouse, keyboard, pointer, touch screen, microphone, joystick, virtual joystick, or touch screen display that a user can use to interact with the client device 110. In some embodiments, the input device 214 may include a camera, microphone, and sensor (e.g., touch sensor, acoustic sensor, inertial motion unit (IMU), and other sensors configured to provide input data to a virtual reality (VR) / augmented reality (AR) head-mounted device). The output device 216 may be a screen display, touch screen, speaker, etc.

[0030] The client device 110 may also include a processor 212-1 configured to execute instructions stored in the memory 220-1 and cause the client device 110 to perform at least some operations in the method consistent with the present disclosure. The memory 220-1 may also include an application 222 and a graphical user interface (GUI) 225, which are configured to run in the client device 110 and are coupled to the input device 214 and the output device 216. The application 222 can be downloaded by a user from the server 130 and can be hosted by the server 130. The application 222 includes specific instructions that, when executed by the processor 212-1, cause the operation according to the method described herein to be performed. In some embodiments, the application 222 runs on an operating system (OS) installed in the client device 110. In some embodiments, the application 222 can run outside a web browser. In some embodiments, the processor is configured to control a graphical user interface (GUI) of a user of a client device of a server of a social platform accessed by a plurality of client devices 110.

[0031] The application 222 may include one or more modules configured to perform operations according to various aspects of various embodiments. These modules will be described in detail later.

[0032] The one or more databases 252 may store data and files associated with the social platform from the application 222. In some embodiments, the client device 110 is a mobile phone that is used to collect videos or images and upload them to the server 130 using the video or image collection application 222 for storage in the one or more databases 252.

[0033] The server 130 includes a memory 220-2, a processor 212-2, and a communication module 218-2. Hereinafter, the processors 212-1 and 212-2 and the memories 220-1 and 220-2 will be collectively referred to as "processor 212" and "memory 220", respectively. The processor 212 is configured to execute instructions stored in the memory 220. In some embodiments, the memory 220-2 includes an engine 232. The engine 232 can share or provide features and resources to the GUI 225, including multiple tools associated with a collection application, an acquisition application, or a design application (e.g., application 222) of text, images, or videos, which use images or pictures retrieved using the engine 232. The user can access the engine 232 through the application 222 installed in the memory 220-1 of the client device 110. Therefore, the application 222 (including the GUI 225) can be installed by the server 130 and execute scripts and other routines provided by the server 130 through any one of the multiple tools. The execution of the application 222 may be controlled by the processor 212-1. The server 130 may include an application programming interface (API) layer that controls the applications in the client device 110. The API layer may also provide tutorials about new features in the application 222 to the user of the client device 110.

[0034] The engine 232 may include one or more modules configured to perform operations according to one or more aspects of the embodiments described herein (eg, see later). Figure 6 The engine 232 includes a neural network tool that can be part of one or more machine learning models stored in one or more databases 252. The one or more databases 252 include the following training archives and other data files: the engine 232 can use these training archives and other data files in the training of machine learning models based on user input through the application 222.

[0035] In some embodiments, at least one or more training archives or machine learning models may be stored in any one of the multiple memories 220, or the user may access them through the application 222. The neural network tool may include algorithms trained for the specific purpose of the engine and tools included in the neural network tool. These algorithms may include machine learning algorithms or artificial intelligence algorithms using any linear or nonlinear algorithms, such as neural network algorithms or multivariate regression algorithms. In some embodiments, the machine learning model may include a neural network (NN), a convolutional neural network (CNN), a generative adversarial neural network (GAN), an unsupervised learning algorithm, a deep recurrent neural network (DRNN), a classical machine learning algorithm (e.g., random forest), a diffusion-based text-image model, or any combination thereof. More generally, the machine learning model may include any machine learning model involving a training step and an optimization step. In some embodiments, one or more databases 252 may include a training archive for modifying coefficients according to the expected results of the machine learning model. Therefore, in some embodiments, the engine 232 is configured to access one or more databases 252 to retrieve documents and archive them as inputs to the machine learning model. In some embodiments, the engine 232 , tools contained therein, and at least a portion of the one or more databases 252 may be hosted on different servers accessible by the server 130 or the client device 110 .

[0036] The technology described herein may be implemented as one or more methods performed by one or more physical computing devices; may be implemented as one or more non-transitory computer-readable storage media storing a plurality of instructions that, when executed by one or more computing devices, cause the one or more methods to be performed; or may be implemented as one or more physical computing devices that are specifically configured with a combination of hardware and software that causes the one or more methods to be performed.

[0037] Figure 3 is a block diagram illustrating a text-to-video generation framework 300 of a text-to-video generation model for generating video based on text input without requiring paired text-to-video data, according to one or more embodiments.

[0038] According to various embodiments, the framework 300 may include three main components: (i) a basic text-image model trained on text-image pairs; (ii) a spatiotemporal convolutional layer and an attention layer that extend the building blocks of the network to the time dimension; and (iii) a spatiotemporal super-resolution network consisting of two spatiotemporal layers, and a frame interpolation network for high frame rate generation. This creates a seamless transition from static to temporal vision. In some embodiments, the basic text-image model is a diffusion-based text-image model, and the framework 300 extends the diffusion-based text-image model to text-video generation through a spatiotemporally factorized diffusion model. The framework 300 utilizes a joint text-image prior to bypass the need for paired text-video data, which in turn allows the model to be scaled to larger amounts of video data. Given a text input, the spatiotemporal super-resolution that is approximated in space and time enables the generation of high-definition, high-frame-rate videos.

[0039] Before adding the temporal component, a basic text-image model is trained on a text-image pair dataset. During inference, the prior network P of the text-image model 304 is configured to generate an image embedding for a given text embedding based on the input text 302 and the encoded text tokens (e.g., byte-pair encoded text tokens). The prior network P is the only component in the framework that is trained on paired text-image data. The inference scheme of the framework 300 can be formulated as:

[0040]

[0041] in, is the generated video, SR h is a spatial super-resolution network, is a spatiotemporal super-resolution network, ↑F is a frame interpolation network (i.e., a frame interpolation network implemented by frame interpolator 308), Dt is a spatiotemporal decoder (i.e., decoder 306), P is a prior network, is a text token encoded using Byte Pair Encoding (BPE), C x is the Contrastive Language-Image Pretraining (CLIP) text encoder and x is the input text.

[0042] Given an input text 302, the text-image model 304 may include encoding the input text 302 using a pre-trained text encoder to generate a text embedding. Then, the text-image model 304 converts the input text 302 into an image embedding through a priori network P.

[0043] The decoder 306 generates a frame rate based on the desired frame rate f psand the image embedding to generate a set of frames constituting a low-resolution video. During training, the decoder 306 is first trained on images and then extended to the temporal dimension by adding a spatiotemporal layer. The decoder 306 is further fine-tuned on unlabeled or unannotated video data. The frame rate f ps is a conditional parameter representing the number of frames per second in a low-resolution video. As a non-limiting example, the set of frames may include 16 64×64 frames. In some embodiments, the set of frames may include a low-resolution 64×64 RGB image based on image embedding.

[0044] According to various embodiments, the decoder 306 is trained by minimizing a hybrid loss to optimize the text-to-video generation model. In some implementations, the loss function consists of two terms: a simple loss and a loss L vlb , simple loss learning predicts the added noise, loss L vlb A constraint is added on the estimated variational lower bound (VLB). According to various embodiments, the loss term of the added noise is predicted is defined as:

[0045]

[0046] Where y is the input video (e.g., a low-resolution video), and y0 represents the first frame of the video. y (y0) represents the extracted first frame of CLIP image embedding. t is the noise input added to y at time steps t uniformly sampled from 1 to T during training. ps is the frame rate embedding as described above. ∈ t is represented by ∈ θ The additional noise in the network estimate.

[0047] The frame interpolator 308 interpolates the set of frames to a higher frame rate for generating the first video based on an interpolation model. The interpolation model may take into account the frame rate f ps and the image embedding from the prior network P to generate multiple frames of the first video. Increasing the frame rate before applying spatiotemporal considerations helps to improve the quality of the final generated video. During training, the frame interpolator 308 is fine-tuned from the spatiotemporal decoder 306 based on random masking of different frames. During inference, the frame interpolator 308 applies interpolation performed using a sliding window.

[0048] According to various embodiments, the interpolation model can use the image representation of the two images to complete the interpolation task. As a non-limiting example, the interpolation model can use the two images as the starting frame and the ending frame, and mask the 14 frames in between based on their image representation for generation. As described above, because the frame interpolator 308 is fine-tuned from the spatiotemporal decoder 306, the frame interpolator 308 is able to generate semantically more meaningful interpolated frames (compared to simply providing a smooth transition between frames without a semantic real-world understanding of what is moving). In some embodiments, the frame interpolator 308 uses the average CLIP embedding of all frames from a video as a condition for generating a semantically similar video.

[0049] According to some embodiments, an extrapolation model similar to the interpolation model of the frame interpolator 308 may be applied. For example, the extrapolation model may be configured to obtain the first 4 frames of the group of frames and expand the first 4 frames to 16 frames. This may be repeated for subsequent groups of 4 frames in the group of frames. According to various embodiments, the interpolation model may be constrained and adjusted to perform interpolation based on specific frames in a specific component and / or the position of the frame in the group of frames. For example, assuming that the group of frames includes 16 frames, the frames with the first position, the last position, and the middle position may be considered as reference frames for interpolation. These reference frames are then used to generate the remaining frames that make up the first video. Following the same example, a video with 76 frames of 64×64 resolution is interpolated. Constraining the interpolation model in this manner enables the frame interpolator 308 to effectively and efficiently learn how to extrapolate between frames, interpolate between frames, and infer actions / events from static images.

[0050] The spatiotemporal super-resolution network 310 is a first upsampler that increases the resolution of frames in the first video based on the spatiotemporal super-resolution model. The spatiotemporal super-resolution network 310 is trained on image data and fine-tuned on unlabeled video data to increase the video resolution by considering temporal information. Following the same example above, the spatiotemporal super-resolution network 310 can increase the resolution of a first video of 76×64×64 to generate a second video with a resolution of 256×256 (i.e., 76×256×256). According to various embodiments, the spatiotemporal super-resolution network 310 as a whole increases the resolution of the first video to a higher resolution. Increasing the resolution of the video helps to significantly improve the final video quality.

[0051] The spatial super-resolution network 312 is a second upsampler that increases the resolution of frames in the second video based on the spatial super-resolution model to generate a final video 316. Following the same example above, the spatial super-resolution network 312 can increase the resolution of the 76×256×256 second video to generate a final high spatiotemporal resolution video of a 76×768×768 resolution video. The spatial super-resolution model applies fixed noise 314 to each of the multiple frames of the second video to help reduce artifacts in the final video. The second super-resolution model can be applied independently to each frame with the same sampled frame noise (e.g., fixed noise 314).

[0052] According to various embodiments, the frame-video portion of the text-video generation framework 300 (e.g., frame interpolator 308 and super-resolution network 310 / 312) is trained on a video dataset to learn motion and adjust the spatiotemporal consistency of the entire generated video (i.e., final video 316). Training the model on unlabeled video data enables training on a much larger dataset with higher quality videos, which in turn improves the accuracy and alignment of the generated videos with the text input.

[0053] In some embodiments, the resulting frames of the final video 316 are downsampled to (e.g., to 512) using bicubic interpolation to provide a clearer visual of the frames in the video. In some embodiments, the super-resolution network involves hallucinogenic information. To prevent flickering artifacts, the illusion must be consistent between frames. Therefore, the super-resolution network operates in both the spatial and temporal dimensions, which is superior to per-frame super-resolution. In some embodiments, the spatial super-resolution network operates only along the spatial dimension. However, in order to promote consistent detail illusions between frames, the same noise initialization is used for each frame.

[0054] According to various embodiments, in order to extend the two-dimensional (2D) conditional network (e.g., text-image model) used in text-image generation to the temporal dimension (i.e., 3D) of video generation, the convolutional layers and attention layers of the super-resolution model are modified to process both the spatial dimension and the temporal dimension in order to generate a video. Other layers (e.g., fully connected layers) do not require specific processing when adding additional dimensions because they are unaware of the structured spatial and temporal information. Temporal modifications include, for example: the decoder 306 generates 16 RGB frames, each of which is 64×64 in size; the frame interpolator 308 increases the effective frame rate by interpolating between a set of generated frames; and the spatiotemporal super-resolution network includes an added temporal layer. According to various embodiments, when fine-tuning a video, a temporal layer is added to the spatiotemporal super-resolution model.

[0055] Figure 4The architecture of a pseudo 3D convolutional layer 400 of a spatiotemporal super-resolution network is shown in accordance with one or more embodiments.

[0056] According to various embodiments, each spatial 2D convolution layer is followed by a temporal one-dimensional (1D) convolution layer. Figure 4 As shown, in the time dimension 416, the first 1Dconv(f k+1 )402 is stacked in the first 2D conv(f k-1 )404. Similarly, in the time dimension 416, the second 1D conv(f k )406 is stacked in the second 2D conv(f k )408, and the third 1D conv(f k-1 )410 is stacked on the third 2D conv(f k+1 )412. Stacking 1D convolutional layers after each 2D convolutional layer and factorizing them in time helps share information between the spatial and temporal axes without being constrained by the heavy computational load of 3D convolutional layers. In addition, it creates a clear partition between pre-trained 2D convolutional layers and newly initialized 1D convolutional layers, enabling temporal convolutions to be trained from scratch while retaining previously learned spatial knowledge in the weights of spatial convolutions.

[0057] Given an input tensor Among them, B, C, F, H, and W are batch, channel, frame, height size, and width size respectively, and the pseudo 3D convolution layer (Conv P3D ) is defined as:

[0058]

[0059] Among them, the transpose operator Swap between spatial and temporal dimensions. For smooth initialization, although the 2D convolutional layer (Conv 2D ) is initialized from a pre-trained text-image model, but the 1D convolutional layer (Conv 1D ) is initialized to the identity function, thereby achieving a seamless transition from training only spatial layers to spatiotemporal layers. The initialization of the new temporal layer is important in order not to corrupt the weights of the pre-trained text-to-image model. Upon initialization, the text-to-video generation model will generate K different images (due to random noise), each of which is faithful to the input text, but lacks temporal consistency. In addition, the set of frames (generated at the decoder 306) will have the spatial consistency obtained from the text-image model, however, will likely lack the temporal consistency required for video. Therefore, for each upsampled frame, the text-to-video model must learn how to correct the upsampling and create a temporally consistent upsampling.

[0060] Figure 5 The architecture and initialization scheme 500 of the pseudo 3D attention layer of the spatiotemporal super-resolution network according to one or more embodiments is shown. The pseudo 3D convolution layer and pseudo 3D attention layer enable seamless conversion of the pre-trained text-to-image model to the temporal dimension. According to various embodiments, the temporal convolution layer and the temporal attention layer are initialized using an identity function.

[0061] A temporal attention layer is applied after the spatial attention layer by initializing the temporal projection to zero, resulting in the identity function of the temporal attention block. The temporal attention layer is stacked after each (pre-trained) spatial attention layer (this is the same as the convolutional layer), approximating a full spatiotemporal attention layer.

[0062] The previous residual layer output 502 is input to the spatial attention layer 504. The spatial attention layer 504 may include embedded image tokens. The output of the frame position encoder 508 and the projection 514 of the spatial attention layer 504 are applied to the temporal attention layer 506. The temporal attention layer 506 may include embedded image tokens for each frame. As described above, the projection 516 of the temporal attention layer 506 is initialized to zero (i.e., k0=0), and the next residual layer input 510 is generated based on the projection. In some embodiments, the initialization scheme 500 may include skip connections 512 in the spatiotemporal network.

[0063] Given an input tensor h, the matrix operator flattens the spatial dimensions to h′∈R B×C×F×HW The inverse matrix operator is used to unflatten the spatial dimensions. Therefore, according to some embodiments, the pseudo 3D attention layer ATTN P3D is defined as:

[0064]

[0065] As described similarly to the pseudo 3D convolutional layer, in order to allow smooth spatiotemporal initialization, the pseudo 3D attention layer ATTN is initialized from a pre-trained text-image model. P3D , and the 1D attention layer ATTN 1D Initialized to the identity function. According to various embodiments, temporal and spatial layers are jointly trained, and an additional 3×1×1 convolutional projection is applied (after each 1×3×3) so that temporal information will also pass through each convolutional layer.

[0066] According to some embodiments, the text-to-video generation model is based on the text-to-image model and the frame rate f ps Conditioning on a varying FPS enables the additional augmentation methods to handle the limited amount of available video at training time and provides additional control over the generated video at inference time.

[0067] In addition to the spatiotemporal modifications (presented by the spatiotemporal model), aspects of the embodiments include training a new masked frame interpolation (and extrapolation) network ↑F that is capable of increasing the number of frames of the generated video. In some embodiments, the number of frames of the generated video is increased by frame interpolation, resulting in a smoother generated video. In some embodiments, the number of frames of the generated video is increased by extrapolating the leading / following frames to extend the length of the video. In order to increase the frame rate and computational constraints in memory, the spatiotemporal decoder 306 fine-tunes the task of masked frame interpolation by zero-padding the masked input frames, thereby achieving video upsampling.

[0068] When fine-tuning the masked frame interpolation, an additional 4 channels can be added to the input of the U-Net. For example, the additional channels can include 3 channels for the RGB masked video input and an additional binary channel indicating which frames are masked. Adjusting variable frame skipping (e.g., skip connection 512) and frame rate f ps to enable multiple temporal upsampling rates at inference time. Similar to training the decoder 306, the training objective of the frame interpolator 308 is to minimize the mixing loss under the additional condition of unmasked frames. The frame interpolation network ↑F expands the given video tensor by interpolating the masked frames. As a non-limiting example, a frame interpolation (and extrapolation) network ↑F with 5 frame skips can be applied to upsample a 16-frame video to 76 frames (i.e., (16-1)×5+1). According to various embodiments, the same architecture and interpolation process described can be applied to video extrapolation and / or image animation by masking each frame at the beginning or end of the video.

[0069] In some embodiments, the text-to-video generation model can be used for image animation, where a masked frame interpolation and extrapolation network ↑F is conditioned on the input image and the CLIP image embedding to extrapolate the rest of the video. This allows users to generate videos using their own images, giving them the opportunity to personalize and directly control the generated video.

[0070] Different models (e.g., text-image model, interpolation model, and super-resolution model) described as part of a text-video generation model / framework can be trained independently and applied during inference. The only component that receives text as input is the prior network P of the text-image model, which is trained on paired text-image data. According to various embodiments, the prior network P is not fine-tuned on the video. According to various embodiments, the decoder (i.e., decoder 306) and the two super-resolution models are first trained separately on the images (without aligned text). As described above, the decoder 306 receives the CLIP image embedding as input, and the super-resolution network 310 / 312 receives the downsampled image as input during training. The new temporal layer is added to the second super-resolution model after being trained on the image. The new temporal layer is initialized and fine-tuned on unlabeled video data. In some embodiments, the new temporal layer is initialized and fine-tuned on unlabeled video data with a random f ps 16 frames are sampled from the original video. ps Within a set range (e.g., from 1 to 30). The beta function is used for sampling and when training the decoder, the higher f ps range (i.e., less motion) and then transition to a lower f ps The masked frame interpolation component may be fine-tuned from the temporal decoder.

[0071] According to various embodiments, as shown in reference Figures 3 to 5 As described, the text-video generative model includes several components (e.g., text-image model, interpolator, super-resolution network). In some embodiments, a subset of the multiple model components can be used when any one or more of the multiple model components are excluded. As a non-limiting example, the text-video generative model can implement a super-resolution model (without previous interpolation). It should be understood that other variations of the text-video generative model are within the scope of the present disclosure.

[0072] The described embodiments provide advantages over other systems, including but not limited to: providing an efficient method to extend diffusion-based text-image models to text-video through a spatiotemporal factorized diffusion model; utilizing joint text-image priors to bypass the need for paired text-video data, which in turn allows scaling to larger amounts of video data; and providing a super-resolution strategy for generating high-definition, high-frame-rate video in space and time given user-provided text input. In addition, the systems and methods according to the embodiments generate state-of-the-art results in both quantitative and qualitative measures, thereby achieving enhanced zero-shot performance in terms of video quality and text-video fidelity, and improved generalization capabilities to generate more coherent videos. The methods according to the embodiments can advantageously outperform when there are large differences between frames, where real-world knowledge of how objects move is critical.

[0073] Figure 6 An example block diagram of a system 600 for text-to-video generation according to one or more embodiments is shown. The apparatus 600 may include one or more computing platforms that may be configured by machine-readable instructions. These machine-readable instructions may include one or more instruction modules. These instruction modules may include computer program modules. Figure 6 As shown, these instruction modules may include one or more of the following: a receiving module 510; a determining module 520; a connecting module 530; a mapping module 540; and a generating module 550; and / or other instruction modules.

[0074] In some embodiments, one or more of the modules 610, 620, 630, 640, 650, 660, 670, and 680 may be included in the client device 110 (e.g., in the application 222) and executed by one or more processors (e.g., the processor 212-1). In some embodiments, one or more of the modules 610, 620, 630, 640, 650, 660, 670, and 680 may be included in the server 130 (e.g., included in the engine 232) and executed by one or more processors (e.g., the processor 212-2). In some embodiments, one or more of the modules 610, 620, 630, 640, 650, 660, 670, and 680 may be included in and executed by a combination of a client device and a server.

[0075] The receiving module 610 is configured to receive text input at a user device (eg, the client device 110 ).

[0076] The frame representation generation module 620 is configured to generate a representation frame or image embedding based on the text input. The representation frame is generated using a text-to-image model trained on a text-image pair dataset. The text-to-image generation model may also include encoding the text input, extracting a text embedding from the text input, and generating an image embedding based on the text embedding.

[0077] The decoding module 630 is configured to generate a set of frames based on the representative frame and a first frame rate. The first frame rate may be predefined by the system. The set of frames may be combined to generate a low-resolution video.

[0078] The interpolation module 640 is configured to interpolate the set of frames to a second frame rate. The second frame rate is a frame rate greater than the first frame rate. In this way, multiple frames of the low-resolution video are interpolated to a higher frame rate. In some embodiments, the system 600 includes extrapolating multiple frames to predict future frames given several specified frames.

[0079] According to various embodiments, the system 600 may further include: interpolating / extrapolating each frame based on the set of frames in the following manner: identifying a preset number of frames and / or frame positions (for example, three frames: a first frame, an intermediate frame, and a last frame) from the set of frames, and generating additional intermediate frames or future frames (for example, intermediate frames between each of the first frame, the intermediate frames, and the last frame) using an interpolation model.

[0080] The video generation module 650 is configured to generate a first video with an increased frame rate based on the interpolated set of frames.

[0081] The first super-resolution module 660 is configured to: based on a spatiotemporal super-resolution model, consider the spatiotemporal information to increase the resolution of the first video to a first resolution value. According to various embodiments, the spatiotemporal super-resolution model is trained on an image dataset and fine-tuned on unlabeled video data. The spatiotemporal super-resolution model generates a second video with a higher resolution. The resolution of the first video can be increased to a higher resolution overall.

[0082] According to various embodiments, the spatiotemporal super-resolution model includes a spatiotemporal convolution layer and an attention layer. The system 600 may also include stacking a temporal convolution layer on each spatial convolution layer and stacking a temporal attention layer on each spatial attention layer. The system 600 may even include initializing the temporal projection of the temporal attention layer to zero and initializing the temporal convolution layer using an identity function.

[0083] The second super-resolution module 670 is configured to: based on a spatial super-resolution model, consider the spatial information to improve the resolution of the second video to a second resolution value. The spatial super-resolution model is trained on the image data set. The resolution of the second video is improved on a frame-by-frame basis, and a fixed noise is applied to each frame in the plurality of frames of the second video. A high spatiotemporal resolution video is generated based on the results of the spatial super-resolution model.

[0084] The output module 680 is configured to output the high spatiotemporal resolution video to a client device and / or, for example, a display of a client device.

[0085] although Figure 6 Several example blocks of system 600 are shown, but in some implementations, system 600 may include more Figure 6 The blocks depicted in the figure are more blocks, fewer blocks, and Figure 6 Those blocks depicted are different blocks or blocks arranged in a different manner. Additionally or alternatively, two or more of the multiple blocks of the system may be combined.

[0086] Figure 7 is a flow chart of a method 700 for text-to-video generation according to one or more embodiments.

[0087] In some embodiments, one or more of the operations / steps in method 700 may be performed by one or more of modules 610, 620, 630, 640, 650, 670, and 680. In some embodiments, Figure 7 One or more operation blocks of the method 700 may be performed by a processor circuit executing instructions stored in a memory circuit, a client device, a remote server, or a database, which are coupled via network communication. In some embodiments, a method consistent with the present disclosure may include at least one or more operations in method 700, which are performed in different orders, simultaneously, quasi-simultaneously, or overlapping in time.

[0088] like Figure 7 As shown, at operation 710, method 700 includes receiving text input at a user device.

[0089] At operation 720, method 700 includes generating a representation frame or image embedding based on the text input.

[0090] At operation 730 , the method 700 includes generating a set of frames based on the representative frame and the first frame rate.

[0091] At operation 740, method 700 includes interpolating the set of frames to a second frame rate. In some embodiments, method 700 may further include extrapolating each frame to predict future frames given several specified frames. Method 700 may further include extrapolating each frame based on the set of frames by identifying a preset number of frames and / or positions of these frames (e.g., a first frame, an intermediate frame, and a last frame) from the set of frames, and generating additional intermediate frames or future frames (e.g., intermediate frames between the first frame, the intermediate frame, and the last frame) using an interpolation model. Method 700 may even include generating future frames by expanding the set of frames based on a subset of the set of frames (e.g., the first 4 frames are expanded to 16 frames).

[0092] At operation 750 , the method 700 includes generating a first video having an increased frame rate based on the interpolated set of frames.

[0093] At operation 760, method 700 includes: based on a spatiotemporal super-resolution model, increasing the resolution of the first video to a first resolution value taking into account the spatiotemporal information. The spatiotemporal super-resolution model is trained on image data and fine-tuned on unlabeled video data. The spatiotemporal super-resolution model generates a second video with a higher resolution. The resolution of the first video is improved overall.

[0094] At operation 770, method 700 includes: based on a spatial super-resolution model, taking into account the spatial information, increasing the resolution of the second video to a second resolution value. The spatial super-resolution model is trained on the image data set. The resolution of the second video is increased on a frame-by-frame basis, and fixed noise is applied to each frame of the plurality of frames of the second video.

[0095] At operation 780 , method 700 includes generating a high spatiotemporal resolution video based on the results of the spatial super-resolution model.

[0096] At operation 790 , method 700 includes outputting the high spatiotemporal resolution video to a user device and / or, for example, a display of a user device.

[0097] According to various embodiments, the spatiotemporal super-resolution model and the spatial super-resolution model are fine-tuned on unlabeled video data.

[0098] According to various embodiments, the spatiotemporal super-resolution model includes a spatiotemporal convolution layer and an attention layer. Method 700 may also include: stacking a temporal convolution layer on each spatial convolution layer, and stacking a temporal attention layer on each spatial attention layer. Method 700 may even include: initializing the temporal projection of the temporal attention layer to zero, and initializing the temporal convolution layer using an identity function.

[0099] although Figure 7Several example blocks of method 700 are shown, but in some embodiments, method 700 may include more Figure 7 The boxes depicted in more boxes, fewer boxes, and Figure 7 In the embodiment of the present invention, the method may be performed in a different manner or in a different way than those depicted in the method. Additionally or alternatively, two or more of the multiple blocks of the method may be executed in parallel.

[0100] Hardware Overview

[0101] Figure 8 It shows that it can be achieved Figure 1 and Figure 2 7000, and a block diagram of an exemplary computer system 800 for the client and server of the method 7000. In some aspects, the computer system 800 can be implemented using hardware, or a combination of software and hardware in a dedicated server, or integrated into another entity, or distributed across multiple entities. The computer system 800 may include a desktop computer, a laptop computer, a tablet computer, a tablet phone, a smart phone, a feature phone, a server computer, or other computers. The server computer may be remotely located in a data center or stored locally.

[0102] The computer system 800 (e.g., the client 110 and the server 130) includes a bus 808 or other communication mechanism for transmitting information, and a processor 802 (e.g., the processor 212) coupled to the bus 808 for processing information. For example, the computer system 800 can be implemented with one or more processors 802. The processor 802 can be a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate logic, a plurality of discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.

[0103] In addition to the hardware, the computer system 800 may also include code for an execution environment created for the mentioned computer programs, such as code constituting the following items stored in the included memory 804 (e.g., memory 220): processor firmware; protocol stack; database management system; operating system; or a combination of one or more of them, such as random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable PROM (Erasable PROM, EPROM), registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or any other suitable storage device, which is coupled to the bus 808 for storing information and storing instructions to be executed by the processor 802. The processor 802 and the memory 804 may be supplemented by or incorporated in special purpose logic circuitry.

[0104] Instructions may be stored in memory 804 and may be implemented in one or more computer program products, such as one or more modules of multiple computer program instructions encoded on a computer-readable medium for execution by computer system 800 or for controlling the operation of computer system 800, and according to any method well known to those skilled in the art, such as, but not limited to, the following computer languages: such computer languages ​​are, for example, data-oriented languages ​​(e.g., structured query language (SQL), dBase), system languages ​​(e.g., C, object-oriented programming language (Objective-C) extending C, C++, assembly), structural languages ​​(e.g., Java, .NET), and application languages ​​(e.g., PHP, Ruby, Perl, Python). The instructions may also be implemented in computer languages ​​such as array languages, aspect-oriented languages, assembly languages, authoring languages, command line interface languages, compiled languages, concurrent languages, curly-bracket languages, data flow languages, data structured languages, declarative languages, esoteric languages, extension languages, fourth generation languages, functional languages, interactive mode languages, interpreted languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multiparadigm languages, numerical analysis, non-English-based languages, class-based object-oriented languages, prototype-based object-oriented languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax handling languages, visual languages, wirth languages, and xml-based languages. The memory 804 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 802 .

[0105] Computer programs as discussed herein do not necessarily correspond to files in a file system. A program may be stored in a portion of a file storing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program mentioned, or multiple collaborative files (e.g., a file storing one or more modules, one or more subroutines, or partial codes). A computer program may be deployed to execute on one or more computers, which are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification may be executed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating outputs.

[0106] The computer system 800 also includes a data storage device 606 coupled to the bus 808 for storing information and instructions, such as a disk or optical disk. The computer system 800 can be coupled to various devices via an input / output module 810. The input / output module 810 can be any input / output module. An exemplary input / output module 810 includes a data port (e.g., a USB port). The input / output module 810 is configured to be connected to a communication module 812. An exemplary communication module 812 (e.g., communication module 218) includes a network interface card (e.g., an Ethernet card and a modem). In some aspects, the input / output module 810 is configured to be connected to multiple devices, such as an input device 814 (e.g., input device 214) and / or an output device 816 (e.g., output device 216). An exemplary input device 814 includes a keyboard and a pointing device (e.g., a mouse or a trackball), and a user can provide input to the computer system 800 through the keyboard and the pointing device. Other types of input devices 814 may also be used to provide interaction with the user, such as tactile input devices, visual input devices, audio input devices, or brain-computer interface devices. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form including acoustic input, voice input, tactile input, or brain wave input. Exemplary output devices 816 include display devices (e.g., liquid crystal display (LCD) monitors) for displaying information to the user.

[0107] According to one aspect of the present disclosure, the client device 110 and the server 130 can be implemented using the computer system 800 in response to the processor 802 executing one or more sequences of one or more instructions contained in the memory 804. Such instructions can be read into the memory 804 from another machine-readable medium (e.g., a data storage device 806). The execution of the sequence of instructions contained in the main memory 804 causes the processor 802 to perform the various process steps described herein. One or more processors in a multi-processing arrangement can also be used to execute the sequence of instructions contained in the memory 804. In alternative aspects, hard-wired circuits can be used instead of software instructions, or hard-wired circuits can be used in combination with software instructions to implement various aspects of the present disclosure. Therefore, aspects of the present disclosure are not limited to any specific combination of hardware circuits and software.

[0108] Various aspects of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). The communication network (e.g., network 150) may include, for example, any one or more of the following: a LAN, a WAN, and the Internet, etc. In addition, the communication network may include, for example, but is not limited to, any one or more of the following tool topologies, including, for example, a bus network, a star network, a ring network, a mesh network, a star bus network, or a tree network or a hierarchical network, etc. The communication module may be, for example, a modem or an Ethernet card.

[0109] Computer system 800 can include client and server.Client and server are usually far away from each other, and usually interact through communication network.The relationship between client and server is due to the computer program running on respective computers and having client-server relationship between each other and generating.Computer system 800 can be, for example, but not limited to desktop computer, laptop computer or tablet computer.Computer system 800 can also be embedded in another device, and this another device is, for example, but not limited to mobile phone, personal digital assistant (PDA), mobile audio player, global positioning system (GlobalPositioning System, GPS) receiver, video game console and / or TV set-top box.

[0110] As used herein, the term "machine-readable storage medium" or "computer-readable medium" refers to any one or more media that participate in providing instructions to processor 802 for execution. Such media can take a variety of forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media (e.g., data storage device 806) include, for example, optical disks or magnetic disks. Volatile media include dynamic memory (e.g., memory 804). Transmission media include coaxial cables, copper wires, and optical fibers, including the wires that form bus 808. Common forms of machine-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, flash erasable programmable read-only memories (FLASH EPROMs), any other memory chips or cartridges, or any other media form that a computer can read. The machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.

[0111] To illustrate the interchangeability of hardware and software, multiple items (e.g., various illustrative blocks, modules, components, methods, operations, instructions, and algorithms) have been generally described in terms of their functionality. Whether such functionality is implemented as hardware, software, or a combination of hardware and software depends on the specific application and design constraints imposed on the overall system. A skilled artisan may implement the described functionality in varying ways for each specific application.

[0112] As used herein, the phrase "at least one of" following a list of items, together with the terms "and" or "or" used to separate any of those items, modifies the list as a whole, rather than modifying each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one item; rather, the phrase is meant to include at least one of any of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" all refer to: only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0113] To the extent the terms "including," "having," and the like are used in this specification or claims, the terms are intended to be open ended in a manner similar to the term "comprising," as "comprising" is interpreted when used as a transitional word in a claim. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0114] Unless otherwise specified, reference to an element in the singular is not intended to mean "one and only one", but rather "one or more". All structural and functional equivalents of the elements of the various configurations described in this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be included in the subject technology. Moreover, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the description above. Unless the element is explicitly described using the phrase "device for...", or in the case of a method clause, the phrase "step for..." is used to describe the element, no clause element shall be interpreted under the provisions of the sixth paragraph of Section 112 of Title 35 of the United States Code.

[0115] Although this specification contains many details, these details should not be interpreted as limitations on the scope of the claimed content, but should be interpreted as descriptions of specific embodiments of this theme. Certain features described in the context of different examples in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although the above may describe features as working in certain combinations, even initially claimed as such, in some cases, one or more features in the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variation of a sub-combination.

[0116] The subject matter of this specification has been described in particular aspects, but other aspects can be implemented and fall within the scope of the following claims. For example, although the operations are depicted in a particular order in the drawings, this should not be understood as requiring the operations to be performed in the particular order shown or in a continuous order, or should not be understood as requiring the execution of all the operations shown to achieve the desired results. The actions recorded in the claims can be performed in different orders and still achieve the desired results. As an example, the process depicted in the drawings does not necessarily require the particular order shown or the continuous order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the above-mentioned multiple aspects should not be understood as requiring such separation in all aspects, but it should be understood that the described program components and systems can be integrated together in a single software product or packaged in multiple software products as a whole. Other variations are within the scope of the appended claims.

Claims

1. A computer-implemented method for text-to-video generation performed by at least one processor, the method comprising: Receive text input; generating a representation frame based on a text embedding of the text input; generating a set of frames based on the representative frames and a first frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; Improving the resolution of the first video based on the spatiotemporal information to generate a second video; as well as An output video is generated by increasing the resolution of the second video based on the spatial information.

2. The computer-implemented method of claim 1 , wherein: The set of frames are combined to generate a low resolution video.

3. The computer-implemented method of claim 1 , wherein: The second frame rate is greater than the first frame rate.

4. The computer-implemented method of claim 1 , wherein: The representation frames are generated using a text-to-image model trained on a dataset of text-image pairs.

5. The computer-implemented method of claim 1 , wherein: The resolution of the second video is increased on a frame-by-frame basis.

6. The computer-implemented method of claim 1 , wherein: Interpolating the set of frames further comprises: identifying a particular frame from the set of frames based on at least one of a preset number of frames and positions of the frames; and An additional frame is generated based on the specific frame using an interpolation model to interpolate the specific frame.

7. The computer-implemented method of claim 1 , wherein: The second video is generated based on a first super-resolution model, and the output video is generated based on a second super-resolution model, wherein the first super-resolution model is fine-tuned on unlabeled video data, and the second super-resolution model applies fixed noise to each frame in the second video.

8. The computer-implemented method of claim 7, wherein: The first super-resolution model includes spatiotemporal convolutional layers and attention layers, wherein a temporal convolutional layer is stacked on each spatial convolutional layer, and a temporal attention layer is stacked on each spatial attention layer.

9. The computer-implemented method of claim 8, further comprising: Initializing the temporal projection of the temporal attention layer to zero; as well as The temporal convolutional layer is initialized as a recognition function.

10. A system for text-to-video generation, the system comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to: Receive text input; generating a representation frame based on a text embedding of the text input; generating a set of frames based on the representative frames and a first frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; Using a first super-resolution model to increase the resolution of the first video based on spatiotemporal information to generate a second video; as well as An output video is generated by increasing the resolution of the second video based on spatial information using a second super-resolution model.

11. The system according to claim 10, further comprising: A low-resolution video is generated based on the set of frames.

12. The system according to claim 10, wherein: The second frame rate is greater than the first frame rate.

13. The system according to claim 10, wherein: The representation frames are generated using a text-to-image model trained on a dataset of text-image pairs.

14. The system according to claim 10, wherein: The first super-resolution model is fine-tuned on unlabeled video data.

15. The system according to claim 10, wherein: The resolution of the second video is increased on a frame-by-frame basis.

16. The system of claim 10, wherein: The one or more processors further execute instructions for: identifying a particular frame from the set of frames based on at least one of a preset number of frames and positions of the frames; and An additional frame is generated based on the specific frame using an interpolation model to interpolate the specific frame.

17. The system according to claim 10, wherein: The second super-resolution model applies fixed noise to each frame in the second video.

18. The system of claim 10, wherein: The first super-resolution model includes spatiotemporal convolutional layers and attention layers, wherein a temporal convolutional layer is stacked on each spatial convolutional layer, and a temporal attention layer is stacked on each spatial attention layer.

19. The system of claim 18, wherein: The one or more processors also execute instructions for: Initializing the temporal projection of the temporal attention layer to zero; and The temporal convolutional layer is initialized to the identity function.

20. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for text-to-video generation and causing the one or more processors to: Receive text input; generating a representation frame based on a text embedding of the text input; Decoding a set of frames conditioned on the representation frame and the frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; Using a first super-resolution model, the resolution of the first video is increased based on spatiotemporal information to generate a second video; as well as The second video is enhanced in resolution based on the spatial information using a second super-resolution model to generate an output video.

Images