Video generation method, and training method and device of video generation model

Abstract

The embodiment of the invention discloses a video generation method and a training method and device of a video generation model. According to the main technical scheme, the method comprises the steps of obtaining input data, wherein the input data comprises first noise data and a control condition; performing de-noising processing on the first noise data based on the control condition by using a first video generation model to generate an initial hidden space representation; the spatial resolution represented by the initial hidden space is smaller than the original resolution of the first noise data; converting the initial hidden space representation into a target hidden space representation by using a second video generation model; the spatial resolution of the target hidden space representation is greater than the spatial resolution of the initial hidden space representation; and decoding the target hidden space representation to generate a target video. According to the invention, the generation speed of the video can be improved while the generation quality of the high-resolution video is not reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, a training method and a device for a video generation model. Background Art

[0002] With the continuous advancement of science and technology, high-resolution video generation technology has shown broad application potential in many fields, such as film and television production, video game development, virtual reality content creation, and the advertising industry. However, current high-resolution video generation technology faces a series of urgent challenges that seriously restrict its further development and popularization.

[0003] Traditional methods for generating high-resolution videos often rely on complex computational processes and enormous computing resources. These methods typically require high-precision rendering and processing of each frame, resulting in extremely slow generation. Furthermore, while some methods have emerged to accelerate video generation, these methods often improve speed at the expense of video quality. This results in significantly reduced visual quality and a significant impact on user experience.

[0004] Therefore, how to significantly improve the generation speed while maintaining the quality of high-resolution video generation has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present application provides a video generation method, a video generation model training method and a device, which are used to increase the video generation speed without reducing the quality of high-resolution video generation.

[0006] This application provides the following solutions:

[0007] In a first aspect, a video generation method is provided, the method comprising:

[0008] Acquiring input data, wherein the input data includes first noise data and a control condition;

[0009] Using a first video generation model, denoising the first noise data based on the control condition to generate an initial latent space representation; the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data;

[0010] Using a second video generation model, converting the initial latent space representation into a target latent space representation; wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation;

[0011] The target latent space representation is decoded to generate a target video.

[0012] Optionally, the using the first video generation model to denoise the first noise data based on the control condition to generate an initial latent space representation includes:

[0013] performing, using the first video generation model, an initial denoising process for k time steps on the first noise data at an original resolution of the first noise data based on the control condition to obtain a first intermediate latent space representation, where k is a positive integer less than T, and T is a maximum time step of the denoising process;

[0014] Based on the first intermediate latent space representation, estimating the noise-free latent space representation corresponding to the k-th time step;

[0015] spatially downsampling the noise-free latent space representation to generate a second intermediate latent space representation; wherein the spatial resolution of the second intermediate latent space representation is less than the original resolution of the noisy data;

[0016] Injecting second noise data corresponding to the k-th time step into the second intermediate latent space representation to generate a third intermediate latent space representation;

[0017] The first video generation model is used to further perform denoising on the third intermediate latent space representation at the spatial resolution of the second intermediate latent space representation to generate the initial latent space representation.

[0018] Optionally, the converting the initial latent space representation into a target latent space representation by using the second video generation model includes:

[0019] Using the second video generation model, performing spatial upsampling mapping processing on the initial latent space representation to obtain a fourth intermediate latent space representation; the spatial resolution of the fourth intermediate latent space representation is greater than the spatial resolution of the initial latent space representation;

[0020] The fourth intermediate latent space representation is optimized using a three-dimensional self-attention layer to obtain the target latent space representation.

[0021] Optionally, optimizing the fourth intermediate latent space representation using a three-dimensional self-attention layer to obtain the target latent space representation includes:

[0022] Dividing the fourth intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows containing latent space representations of a fixed number of frame sequences;

[0023] Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows;

[0024] fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fifth intermediate latent space representation;

[0025] Performing a cyclic shift operation on the fifth intermediate latent space representation and re-dividing the fifth intermediate latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences;

[0026] Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window;

[0027] The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

[0028] Optionally, performing three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window includes:

[0029] Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

[0030] Optionally, a parameter scale of the first video generation model is greater than a parameter scale of the second video generation model.

[0031] In a second aspect, a method for training a video generation model is provided, the method comprising:

[0032] Acquire training data including a plurality of training samples, wherein the training samples include input data samples and target video samples, wherein the input data samples include control conditions and noise samples, and the noise samples are obtained by adding noise to the target video samples;

[0033] denoising the noise sample based on the control condition using a first video generation model to generate an initial latent space representation, wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the target video sample;

[0034] Converting the initial latent space representation into a target latent space representation using a second video generation model, wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation;

[0035] Decoding the target latent space representation to generate a target video;

[0036] A loss function is determined using the target video and the target video sample, and the model parameters of the second video generation model are updated using the loss function.

[0037] Optionally, the second video generation model includes a mapping layer and a three-dimensional self-attention layer;

[0038] The converting the initial latent space representation into a target latent space representation by using a second video generation model includes:

[0039] Performing spatial upsampling mapping processing on the initial latent space representation using the mapping layer to obtain an intermediate latent space representation; the spatial resolution of the intermediate latent space representation is greater than the spatial resolution of the initial latent space representation;

[0040] The intermediate latent space representation is optimized using the three-dimensional self-attention layer to obtain the target latent space representation.

[0041] Optionally, optimizing the intermediate latent space representation using the three-dimensional self-attention layer to obtain the target latent space representation includes:

[0042] Dividing the intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows contains the latent space representation of a fixed number of frame sequences;

[0043] Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows;

[0044] Fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fused latent space representation;

[0045] Performing a cyclic shift operation on the fused latent space representation and re-dividing the fused latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences;

[0046] Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window;

[0047] The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

[0048] Optionally, performing three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window includes:

[0049] Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

[0050] According to a third aspect, a video generation device is provided, the device comprising:

[0051] a data acquisition unit configured to acquire input data, wherein the input data includes first noise data and a control condition;

[0052] a first processing unit configured to, using a first video generation model, perform denoising on the first noise data based on the control condition to generate an initial latent space representation; wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data;

[0053] a second processing unit configured to convert the initial latent space representation into a target latent space representation using a second video generation model; wherein a spatial resolution of the target latent space representation is greater than a spatial resolution of the initial latent space representation;

[0054] The video generation unit is configured to decode the target latent space representation to generate a target video.

[0055] In a fourth aspect, a training device for a video generation model is provided, the device comprising:

[0056] a sample acquisition unit configured to acquire training data comprising a plurality of training samples, wherein the training samples include input data samples and target video samples, wherein the input data samples include control conditions and noise samples, and the noise samples are obtained by adding noise to the target video samples;

[0057] The model training unit is configured to use the first video generation model to denoise the noise sample based on the control condition to generate an initial latent space representation, wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the target video sample; use the second video generation model to convert the initial latent space representation into a target latent space representation, wherein the spatial resolution of the target latent space representation is larger than the spatial resolution of the initial latent space representation; decode the target latent space representation to generate a target video; use the target video and the target video sample to determine a loss function, and use the loss function to update the model parameters of the second video generation model.

[0058] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed, the steps of the method of the first aspect are implemented.

[0059] In a sixth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method of the first aspect above.

[0060] In a seventh aspect, a computer program product is provided, comprising a computer program, which implements the steps of the method of the first aspect when executed.

[0061] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0062] 1) This application first uses the first video generation model to denoise the first noise data based on control conditions to generate a low-resolution initial latent space representation, which can effectively reduce the amount of data and computational complexity, and then uses the second video generation model to further convert the low-resolution initial latent space representation into a high-resolution target latent space representation. Since the second video generation model does not need to generate content from scratch, but directly converts the initial latent space representation, this process can refine video details and improve video resolution with fewer denoising steps, ultimately generating higher-quality video content. This method can significantly improve the efficiency of video generation without reducing the quality of the generated video through a two-stage processing flow.

[0063] 2) In the embodiment of the present application, the first video generation model performs partial denoising at the original resolution and then performs downsampling and other operations, so that the generated initial latent space vector can retain the global structure and semantic information while reducing the data volume and computational complexity.

[0064] 3) In the embodiment of the present application, the initial latent space representation is first upsampled and mapped by the second video generation model to improve the resolution, and then further optimized using the three-dimensional self-attention layer to generate a target latent space representation with rich details and high resolution.

[0065] 4) The embodiment of the present application limits the scope of attention calculation by limiting the window, and at the same time combines cyclic shift to cover the time correlation across windows, thereby reducing the computational complexity while ensuring the optimization effect.

[0066] 5) The embodiment of the present application effectively prevents information interference between discontinuous frames by masking the discontinuous frames in the time dimension before the three-dimensional self-attention calculation, improves the accuracy of time association, and ensures the smoothness and naturalness of the generated video.

[0067] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A schematic diagram of a system architecture applicable to embodiments of the present application;

[0070] Figure 2 A flowchart of a video generation method provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of a video generation framework provided in an embodiment of the present application;

[0072] Figure 4 A flowchart of a method for training a video generation model provided in an embodiment of the present application;

[0073] Figure 5 A schematic block diagram of a video generation device provided in an embodiment of the present application;

[0074] Figure 6 A schematic block diagram of a training device for a video generation model provided in an embodiment of the present application;

[0075] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0077] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0078] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0079] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0080] Traditional high-resolution video generation methods often rely on complex computational processes and enormous computing resources. These methods typically require high-precision rendering and processing of each frame, resulting in extremely slow generation speeds. For example, current video generation models take approximately 50 minutes to generate a 5-second video at 720P resolution. To address this issue, several methods have been proposed to accelerate video generation, but these methods often improve speed at the expense of video quality. As a result, the visual quality of the video is significantly reduced, and the user experience is severely impacted.

[0081] In view of this, the present application provides a new approach. To facilitate understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 This is a schematic diagram of the system architecture applicable to the embodiments of the present application, such as Figure 1 As shown in , the system architecture may include: user equipment, a video generation device located on the server side, and a device for training a video generation model.

[0082] The user equipment and the server can communicate with each other. The user equipment and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0083] User devices may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices may include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include smart TVs and smart refrigerators. Wearable devices may include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0084] A server can be a standalone server, a server cluster, or even a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0085] Before performing a video generation task, the device for training a video generation model can adopt the method provided in the embodiment of the present application to train a video generation model.

[0086] Users can input data through their user devices, which then send the input data over the network to a video generation device on the server side. The video generation device uses a trained video generation model to generate a target video and returns it to the user device over the network. The user device then displays the received target video to the user.

[0087] Apart from Figure 1 In addition to the architecture shown, a computer terminal device with strong computing power can also use the method provided in the embodiment of the present application to train the video generation model and / or generate the target video.

[0088] It should be understood that Figure 1 The number of user devices, apparatuses for training video generation models, video generation apparatuses, and video generation models in the embodiment is merely illustrative. Any number of user devices, apparatuses for training video generation models, video generation apparatuses, and video generation models may be provided as needed.

[0089] It should be noted that the limitations such as "first" and "second" involved in this disclosure do not have restrictions on size, order and quantity, but are only used to distinguish them in name. For example, "first noise data" and "second noise data" are used to distinguish two types of noise data, and "first video generation model" and "second video generation model" are used to distinguish two video generation models, and so on.

[0090] Figure 2 This is a flowchart of a video generation method provided in an embodiment of the present application. The method can be performed by Figure 1 The video generation device in the system shown is executed. Figure 2 As shown in , the method may include the following steps:

[0091] Step 201: Acquire input data, where the input data includes first noise data and control conditions.

[0092] Step 202: Using the first video generation model, denoising is performed on the first noise data based on a control condition to generate an initial latent space representation; the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data.

[0093] Step 203: Using the second video generation model, the initial latent space representation is converted into a target latent space representation; the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation.

[0094] Step 204: Decode the target latent space representation to generate a target video.

[0095] As can be seen from the above process, this application first uses the first video generation model to denoise the first noise data based on the control conditions to generate a low-resolution initial latent space representation, which can effectively reduce the amount of data and computational complexity, and then uses the second video generation model to further convert the low-resolution initial latent space representation into a high-resolution target latent space representation. Since the second video generation model does not need to generate content from scratch, but directly converts the initial latent space representation, this process can refine video details and improve video resolution with fewer denoising steps, ultimately generating higher-quality videos. This method can significantly improve the efficiency of video generation without reducing the quality of the generated video.

[0096] The following describes in detail the steps in the above process and the effects that can be further produced in conjunction with the embodiments.

[0097] First, the above step 201, namely "obtaining input data, where the input data includes first noise data and control conditions", is described in detail with reference to an embodiment.

[0098] The method provided in the embodiment of the present application is applicable to a video generation task. In the video generation task, the input data may include first noise data and a control condition.

[0099] The first noise data is usually randomly generated Gaussian noise, which serves as the initial input for video generation and has a certain dimension and structure, and can be gradually transformed into video content through subsequent denoising steps. Specifically, the first noise data can be expressed as Z1∈R b×c×f×h×w , where b represents the batch size, i.e. the number of noise data samples processed at one time, c is the number of channels, corresponding to the color channel of the video, f represents the number of frames, which determines the duration and smoothness of the video, and h×w represents the resolution, which reflects the clarity of the video.

[0100] Control conditions are typically user input or selections that guide the direction and details of video generation. Common control conditions include, but are not limited to, the following:

[0101] Text prompt data: Users can input text descriptions to define the video's theme, plot, action, and scene. For example, users can input "a cat wearing sunglasses and a swim ring floating on the water, or a cute raccoon playing guitar on a boat in the ocean." These text prompts provide semantic information to the model, helping it generate video content that meets user expectations.

[0102] Image prompt data: Users can provide one or more images as references, and the model will generate similar videos based on the visual features of these images. For example, if a user uploads a photo of a landscape, the model will generate a video related to that landscape.

[0103] Other conditions: In addition to text and images, control conditions can also include audio, video clips, motion trajectory data, etc. For example, users can provide an audio clip as background music for a video, or provide a video clip as a reference style.

[0104] Next, in conjunction with an embodiment, the above-mentioned step 202, namely, "using the first video generation model to denoise the first noise data based on the control condition to generate an initial latent space representation; the spatial resolution of the initial latent space representation is less than the original resolution of the first noise data" is described in detail.

[0105] After analyzing the existing video generation process, it was found that the key factors affecting video generation speed are resolution and the number of denoising steps. Therefore, the embodiment of the present application divides the video generation process into two stages. In the first stage, a low-resolution video preview is generated, namely the initial latent space representation. Then, in the second stage, the low-resolution video preview is converted into a high-resolution target video using fewer denoising steps.

[0106] To efficiently complete the first phase, this embodiment of the application uses a pre-trained model as the first video generation model. This model can use its learned features to denoise the first noisy data in the initial phase, quickly identifying and preserving important structural and semantic information in the video while removing noise. This capability ensures that the initial latent space representation retains the prior knowledge of the pre-trained model.

[0107] As an implementable manner, step 202 may be implemented as follows:

[0108] The first video generation model receives the first noise data Z1∈R b×c×f×h×wand control conditions as input. In the initial stage, the first video generation model denoises the first noisy data for k time steps based on the control conditions at its original resolution, generating a first intermediate latent space representation. Here, k is a positive integer smaller than the maximum time step T of the denoising process. This means that the first video generation model does not complete all denoising steps at once, but instead determines the overall content and structure of the video at an early stage.

[0109] In the embodiment of the present application, before reaching the kth time step, the first noise data z1 may be denoised based on a flow matching method of an ODE (Ordinary Differential Equation), as shown in formula (1):

[0110]

[0111] Among them, u θ (z i ,i) represents the direction function predicted by the first video generation model at time step t = i. The value of time step t ranges from [0, 1]. During the denoising process, time step t gradually decreases from t = 1 to t = 0. In other words, the smaller the value of t, the closer the denoising process is to completion. When reaching the kth time step, the value of time step t is between 0 and 1, depending on the position of k. Finally, when t = 0, the denoising process is complete, and the noise-free latent space representation z0 is obtained.

[0112] After denoising at the kth time step, the first intermediate latent space is represented as z k .

[0113] It should be noted that the original resolution here is also the optimal resolution of the first video generation model, which can avoid prior losses. In addition, k in the embodiment of the present application can be set manually, and it is only necessary to ensure that the first intermediate latent space representation obtained at the kth time step already contains the global semantic information and structure of the video. The specific value of k is not limited in the embodiment of the present application.

[0114] Then, the first video generation model is based on the first intermediate latent space representation z k , and estimate the noise-free latent space representation corresponding to the k-th time step. This estimation process is to predict the potential representation at the k-th time step through the first video generation model, that is, the pure signal after removing the noise.

[0115] Specifically, the noise-free latent space representation can be determined using the following formula (2):

[0116]

[0117] Among them, σk is the standard deviation at the kth time step, is the noise-free latent space estimation representation.

[0118] Then, this noiseless latent space is represented as Perform spatial downsampling to generate a second intermediate latent space representation. Spatial downsampling reduces the data size and computational complexity by lowering the data resolution, making subsequent processing more efficient. The spatial resolution of the downsampled data is smaller than that of the original noisy data.

[0119] Specifically, it can be determined using the following formula (3):

[0120]

[0121] in, It is the second intermediate latent space representation.

[0122] Then, the first video generation model re-injects the second noise data corresponding to the k-th time step into the second intermediate latent space representation to obtain a third intermediate latent space representation, where the second noise data refers to the noise data corresponding to the k-th time step.

[0123] Specifically, it can be determined using the following formula (4):

[0124]

[0125] Among them, z k-1 It is the third intermediate latent space representation. is the second noise data injected.

[0126] Finally, the third intermediate latent space representation is further denoised at the spatial resolution of the second intermediate latent space representation to generate the initial latent space representation.

[0127] This process involves performing partial denoising on the first video generation model at the original resolution and then downsampling, so that the generated initial latent space vector can retain the global structure and semantic information while reducing the amount of data and computational complexity.

[0128] Next, the above step 203, namely "using the second video generation model to convert the initial latent space representation into a target latent space representation; the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation" is described in detail with reference to an embodiment.

[0129] After the first video generation model outputs the initial latent space representation, it serves as the input to the second video generation model. The second video generation model first performs spatial upsampling on the initial latent space representation to obtain a fourth intermediate latent space representation with a higher spatial resolution than the initial latent space representation. The fourth intermediate latent space representation is then optimized using a 3D self-attention layer to obtain the target latent space representation.

[0130] Among them, when performing spatial upsampling, the low-resolution latent space representation (denoted as z lr ) and high-resolution latent space representation (denoted as z h ). Specifically, replace z1 in the original formula (1) with z lr , replace z0 with z hr The modified flow matching equation can map the initial latent space representation to the high-resolution space, thereby obtaining the fourth intermediate latent space representation, namely z hr .

[0131] In this process, lr represents the low-resolution initial latent space representation, which is the output generated by the first stage. It has a low spatial resolution but contains the global structure and semantic information of the video. hr It represents a high-resolution latent space representation, which is the output that the second stage hopes to generate. It has a higher spatial resolution and can accommodate richer details and finer structures.

[0132] After obtaining the fourth intermediate latent space representation, it needs to be further optimized to improve the detail accuracy of the generated video. The existing technology generally uses the full attention mechanism to achieve optimization, but as the number of video frames increases, the computational complexity of the full attention mechanism increases with the square of the number of frames. For example, if the video contains 100 frames, the full attention mechanism needs to calculate 100×100=10,000 attention pairs. If the number of video frames increases to 200 frames, the computational complexity rises to 200×200=40,000 times, which will lead to a significant increase in calculation time and seriously affect the efficiency of the video generation model.

[0133] Therefore, an embodiment of the present application proposes an optimization strategy that combines a cyclic shift operation and a three-dimensional self-attention mechanism, which limits the attention calculation range by limiting the window, and combines cyclic shift to cover the time correlation across windows, thereby reducing the computational complexity while ensuring the optimization effect.

[0134] Specifically, the fourth intermediate latent space representation is first divided into multiple non-overlapping first windows according to the time dimension, and each first window contains the latent space representation of a fixed number of frame sequences; then, a three-dimensional self-attention calculation is performed on the latent space representation in each first window to obtain the first optimized latent space representation corresponding to each first window; then, the first optimized latent space representation corresponding to each first window is fused to obtain the fifth intermediate latent space representation; then, a cyclic shift operation is performed on the fifth intermediate latent space representation, and it is re-divided into multiple non-overlapping second windows, and each second window contains the latent space representation of a fixed number of frame sequences; then, a three-dimensional self-attention calculation is performed on the latent space representation in each second window to obtain the second optimized latent space representation corresponding to each second window; finally, the second optimized latent space representation corresponding to each second window is fused to generate the target latent space representation.

[0135] Furthermore, before performing three-dimensional self-attention calculation on the latent space representation in each second window, the two frames that are discontinuous in the time dimension in each second window are masked. This can avoid information interference between discontinuous frames on the time axis, thereby improving the accuracy of temporal association and ensuring the smoothness and naturalness of the generated video.

[0136] In practical applications, the first video generation model and the second video generation model together constitute a video generation framework, such as Figure 3 The figure shows a schematic diagram of the video generation framework. In the downsampling rendering stage, the model is first pre-trained (i.e., the first video generation model) to denoise the initial noise at the original resolution (480p). This process uses the initial Gaussian noise and control conditions as input. When the k-th time step is reached, the noise-free latent space representation is estimated and spatially downsampled (from 480p to 240p) to reduce the amount of data and computational complexity. The predicted noise is then re-injected into the downsampled latent space representation, and denoising is continued at a low resolution, ultimately generating a low-resolution initial latent space representation (such as a 240p video preview).

[0137] In the upsampling mapping stage, the low-resolution initial latent space representation (such as 240p) is processed through spatial upsampling mapping to generate a high-resolution fourth intermediate latent space representation (such as 1080p). This step is achieved by modifying the flow matching equation to map the initial latent space representation to the high-resolution space. In the high-resolution space, a lightweight second video generation model (such as DiT Transformer) is used to further denoise the fourth intermediate latent space representation, and it is sequentially processed by modules such as RMSNorm (root mean square normalization, used to normalize data and accelerate model convergence), Scale (scaling operation) and 3D Self-Attention (three-dimensional self-attention mechanism, used to capture the feature relationship of the video in the spatiotemporal dimension). These operations further extract and process the video data, optimize the quality of video generation, and finally output the processed video data.

[0138] Taking the fourth intermediate latent space representation as an example, where the total number of frames is 28 and the frame order is 1-28, it is divided into seven first windows according to the time dimension W = 4 (window 1: frames 1-4, window 2: frames 5-8, window 3: frames 9-12, window 4: frames 13-16, window 5: frames 17-20, window 6: frames 21-24, window 7: frames 25-28). A three-dimensional self-attention calculation is performed on the latent space representation within each first window, and the self-attention results of the first window are then weighted fused to generate the fifth intermediate latent space vector.

[0139] If only the first window is used for division, the time boundary information between windows will be lost. For example, frame 4 in window 1 and frame 5 in window 2 are continuous in the time dimension, but are not covered by the same window, which may weaken the time correlation across windows. Therefore, the fifth intermediate latent space representation is further cyclically shifted in the time dimension. Forward shift or backward shift can be used here. In the embodiment of the present application, the time dimension of the forward shift W / 2=2 frames is shifted as an example. The first two frames are moved to the end. After the cyclic shift, the frame order becomes 3-28-1-2, and is redivided into 7 second windows (window A: frames 3-6, window B: frames 7-10, window C: frames 11-14, window D: frames 15-18, window E: frames 19-22, window F: frames 23-26, window G: frames 27-28-1-2).

[0140] Before the self-attention calculation of the second window, in an embodiment of the present application, the discontinuous frames in the time dimension can be masked. For example, in window G (frames 27-28-1-2) of the second window, frame 28 and frame 1 are non-continuous frames in the original frame sequence. At this time, the attention weights of frames 28 and 1 need to be suppressed, and they can be masked by setting them to a minimum value or 0. After masking, the three-dimensional self-attention calculation is performed on the latent space representation in the second window. Such a mask design can avoid the association of erroneous information between discontinuous frames in the time dimension, thereby improving the accuracy of time association.

[0141] Finally, the self-attention results of the second window are weighted fused to generate the final target latent space representation.

[0142] In the embodiment of the present application, by dividing the local window, the attention calculation is reduced from global to local, which reduces the computational complexity limit. The cyclic shift strategy covers the time correlation across windows, which improves the accuracy of time correlation. The optimized target latent space representation can simultaneously retain the motion trajectory of local continuous frames and the global time consistency across windows, so that the details of the final generated video are also significantly improved. At the same time, the mask processing suppresses the erroneous interference of non-continuous frames, avoiding the problems of "window boundary information loss" or "erroneous association".

[0143] After upsampling mapping, the denoising step is greatly reduced, and the denoising effect can be achieved in a single-digit time step, achieving good video generation effects.

[0144] Next, the above step 204, namely "decoding the target latent space representation to generate the target video", is described in detail with reference to an embodiment.

[0145] After obtaining the target latent space representation, the target latent space representation is given to the decoding network in the second video generation model, and the decoding network decodes it to obtain the target video.

[0146] In addition, in an embodiment of the present application, in order to make the second video generation model more advantageous in terms of computational efficiency and memory usage, so as to be able to complete the denoising and optimization tasks more efficiently, the second video generation model can adopt a lightweight model. In other words, the parameter scale of the second video generation model is smaller than the parameter scale of the first video generation model. For example, the first video generation model adopts parameters on the scale of tens of billions, while the second video generation model adopts parameters on the scale of billions. Because the parameter scale of the model can directly affect its feature extraction capability and computational complexity, the second video generation model in the embodiment of the present application does not need to generate content from scratch, but directly optimizes the initial latent space representation. Since the initial latent space representation already contains the global semantic information and structure of the video, the second video generation model does not need to repeatedly learn the underlying features, but only needs to focus on "optimization", so its parameter scale can be smaller.

[0147] Figure 4 This is a flowchart of a method for training a video generation model provided in an embodiment of the present application. The method can be performed by Figure 1 The system shown is executed by a device for training a video generation model. Figure 4 As shown, the method may include the following steps:

[0148] Step 401: Acquire training data including a plurality of training samples. The training samples include input data samples and target video samples. The input data samples include control conditions and noise samples. The noise samples are obtained by adding noise to the target video samples.

[0149] In an embodiment of the present application, the input data sample may include a control condition and a noise sample. The control condition may be obtained by manually labeling the target video sample, and the noise sample may be obtained by adding noise to the target video sample (i.e., adding T time steps of noise).

[0150] Step 402: De-noising the noise sample based on the control condition using the first video generation model to generate an initial latent space representation, where the spatial resolution of the initial latent space representation is smaller than the original resolution of the target video sample.

[0151] The first video generation model may be a pre-trained model. The specific details of the model performing denoising processing on noise samples to generate an initial latent space representation may be found in the relevant records in the above step 202 and will not be repeated here.

[0152] Step 403: Utilize the second video generation model to convert the initial latent space representation into a target latent space representation, wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation.

[0153] In an embodiment of the present application, the second video generation model may include a mapping layer and a three-dimensional self-attention layer, wherein:

[0154] The initial latent space representation is spatially upsampled and mapped using a mapping layer to obtain an intermediate latent space representation; the spatial resolution of the intermediate latent space representation is greater than that of the initial latent space representation;

[0155] The intermediate latent space representation is optimized using a three-dimensional self-attention layer to obtain the target latent space representation.

[0156] Here, the mapping layer can establish a low-resolution latent space representation (denoted as z) by modifying the flow matching equation mentioned above (such as formula (1)). lr ) and high-resolution latent space representation (denoted as z hr ). Specifically, replace z1 in the original formula (1) with the initial latent space representation, and replace z0 with the intermediate latent space representation. The modified flow matching equation can map the initial latent space representation to the high-resolution space, thereby obtaining the intermediate latent space representation.

[0157] In this process, the initial latent space representation is the output of the first video generation model, which has a lower spatial resolution but contains the global structure and semantic information of the video. The intermediate latent space representation has a higher spatial resolution and can accommodate richer details and finer structures.

[0158] In addition, the three-dimensional self-attention layer in the embodiment of the present application is combined with a cyclic shift strategy, specifically including:

[0159] Divide the intermediate latent space representation into multiple non-overlapping first windows according to the time dimension, each first window contains the latent space representation of a fixed number of frame sequences;

[0160] Perform three-dimensional self-attention calculation on the latent space representation in each first window to obtain the first optimized latent space representation corresponding to each first window;

[0161] Fusing the first optimized latent space representations corresponding to each first window to obtain a fused latent space representation;

[0162] Performing a cyclic shift operation on the fused latent space representation and re-dividing it into multiple non-overlapping second windows, each second window containing the latent space representation of a fixed number of frame sequences;

[0163] Perform three-dimensional self-attention calculation on the latent space representation in each second window to obtain the second optimized latent space representation corresponding to each second window;

[0164] The second optimized latent space representations corresponding to each second window are fused to generate the target latent space representation.

[0165] Furthermore, when performing three-dimensional self-attention calculation on the latent space representation in each second window, the two discontinuous frames in each second window in the time dimension can be masked first, and then the three-dimensional self-attention calculation is performed to obtain the second optimized latent space representation corresponding to each second window.

[0166] The specific details of optimizing the intermediate latent space representation using the three-dimensional self-attention layer to obtain the target latent space representation can be found in the relevant records in the above step 203 and will not be repeated here.

[0167] Step 404: Decode the target latent space representation to generate a target video.

[0168] The specific details of decoding the target latent space representation to generate the target video can be found in the relevant records in the above step 204, which will not be repeated here.

[0169] Step 405: Determine a loss function using the target video and the target video sample, and use the loss function to update model parameters of the second video generation model.

[0170] The target video sample refers to real-world video data with high resolution (e.g., 1080p), which serves as a reference for model generation and is used to measure the quality of the generated results. The target video is the output of the second video generation model. The loss function is a quantitative metric that measures the difference between the model output and the target output.

[0171] In this embodiment of the present application, a loss function can be used to supervise the training of the second video generation model, so that it learns the mapping relationship from the low-resolution preview (i.e., the initial latent space representation) to the high-resolution target (the target latent space representation). This process not only ensures the visual quality of the generated video, but also maintains the prior knowledge of the pre-trained model, ultimately achieving efficient and high-quality high-resolution video generation.

[0172] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0173] Under the condition of the same input data and video resolution, the video generation effect of the method provided by the embodiment of the present application is compared with the traditional video generation method (for example, based on the Wan2.1 model): at a video resolution of 1080p, the method provided by the embodiment of the present application can shorten the video generation time of 81 frames from 29530 seconds to 643.5 seconds compared with the traditional video generation method. At a video resolution of 720p, the method provided by the embodiment of the present application can shorten the video generation time of 81 frames from 5381.5 seconds to 586.2 seconds compared with the traditional video generation method. However, the quality scores of the videos are almost the same.

[0174] According to an embodiment of another aspect, a video generating apparatus is provided. Figure 5 This is a schematic block diagram of a video generation device provided in an embodiment of the present application. Figure 5 As shown, the apparatus 500 mainly includes: a data acquisition unit 501, a first processing unit 502, a second processing unit 503 and a video generation unit 504. The main functions of each component unit are as follows:

[0175] The data acquisition unit 501 is configured to acquire input data, where the input data includes first noise data and a control condition;

[0176] A first processing unit 502 is configured to use a first video generation model to denoise the first noise data based on the control condition to generate an initial latent space representation; the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data;

[0177] The second processing unit 503 is configured to convert the initial latent space representation into a target latent space representation using a second video generation model; the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation;

[0178] The video generation unit 504 is configured to decode the target latent space representation to generate a target video.

[0179] Optionally, the first processing unit 502 is specifically configured to:

[0180] performing, using the first video generation model, an initial denoising process for k time steps on the first noise data at an original resolution of the first noise data based on the control condition to obtain a first intermediate latent space representation, where k is a positive integer less than T, and T is a maximum time step of the denoising process;

[0181] Based on the first intermediate latent space representation, estimating the noise-free latent space representation corresponding to the k-th time step;

[0182] spatially downsampling the noise-free latent space representation to generate a second intermediate latent space representation; wherein the spatial resolution of the second intermediate latent space representation is less than the original resolution of the noisy data;

[0183] Injecting second noise data corresponding to the k-th time step into the second intermediate latent space representation to generate a third intermediate latent space representation;

[0184] The first video generation model is used to further perform denoising on the third intermediate latent space representation at the spatial resolution of the second intermediate latent space representation to generate the initial latent space representation.

[0185] Optionally, the second processing unit 503 is specifically configured to:

[0186] Using the second video generation model, performing spatial upsampling mapping processing on the initial latent space representation to obtain a fourth intermediate latent space representation; the spatial resolution of the fourth intermediate latent space representation is greater than the spatial resolution of the initial latent space representation;

[0187] The fourth intermediate latent space representation is optimized using a three-dimensional self-attention layer to obtain the target latent space representation.

[0188] Optionally, the second processing unit 503 optimizes the fourth intermediate latent space representation using a three-dimensional self-attention layer to obtain the target latent space representation, and is specifically configured as follows:

[0189] Dividing the fourth intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows containing latent space representations of a fixed number of frame sequences;

[0190] Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows;

[0191] fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fifth intermediate latent space representation;

[0192] Performing a cyclic shift operation on the fifth intermediate latent space representation and re-dividing the fifth intermediate latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences;

[0193] Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window;

[0194] The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

[0195] Furthermore, the second processing unit 503 performs three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window, and is specifically configured as follows:

[0196] Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

[0197] Optionally, a parameter scale of the first video generation model is greater than a parameter scale of the second video generation model.

[0198] Figure 6 A schematic block diagram of a training device for a video generation model provided in an embodiment of the present application, such as Figure 6 As shown in FIG, the apparatus 600 may include: a sample acquisition unit 601 and a model training unit 602. The main functions of each component unit are as follows:

[0199] The sample acquisition unit 601 is configured to acquire training data including a plurality of training samples, wherein the training samples include input data samples and target video samples, wherein the input data samples include control conditions and noise samples, and the noise samples are obtained by adding noise to the target video samples;

[0200] The model training unit 602 is configured to use the first video generation model to denoise the noise sample based on the control condition to generate an initial latent space representation, wherein the spatial resolution of the initial latent space representation is less than the original resolution of the target video sample; use the second video generation model to convert the initial latent space representation into a target latent space representation, wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation; decode the target latent space representation to generate a target video; use the target video and the target video sample to determine a loss function, and use the loss function to update the model parameters of the second video generation model.

[0201] Optionally, the second video generation model includes a mapping layer and a three-dimensional self-attention layer;

[0202] The model training unit 602 converts the initial latent space representation into a target latent space representation, including:

[0203] Performing spatial upsampling mapping processing on the initial latent space representation using the mapping layer to obtain an intermediate latent space representation; the spatial resolution of the intermediate latent space representation is greater than the spatial resolution of the initial latent space representation;

[0204] The intermediate latent space representation is optimized using the three-dimensional self-attention layer to obtain the target latent space representation.

[0205] Optionally, the model training unit 602 optimizes the intermediate latent space representation using the three-dimensional self-attention layer to obtain the target latent space representation, and is specifically configured as follows:

[0206] Dividing the intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows contains the latent space representation of a fixed number of frame sequences;

[0207] Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows;

[0208] Fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fused latent space representation;

[0209] Performing a cyclic shift operation on the fused latent space representation and re-dividing the fused latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences;

[0210] Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window;

[0211] The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

[0212] Optionally, the model training unit 602 performs three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window, and is specifically configured as follows:

[0213] Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

[0214] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0216] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0217] And an electronic device comprising:

[0218] one or more processors; and

[0219] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0220] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed.

[0221] in, Figure 7The schematic block diagram of an electronic device provided in an embodiment of the present application may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730. The input / output interface 713 may also be referred to as an I / O interface 713.

[0222] Among them, the processor 710 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0223] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. In addition, a web browser 723, a data storage management system 724, a video generation device 500, a device 600 for training a video generation model, etc. can also be stored. The above-mentioned video generation device 500 and the device 600 for training a video generation model can be the application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0224] The input / output interface 713 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0225] The network interface 714 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0226] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710 , the video display adapter 711 , the disk drive 712 , the input / output interface 713 , the network interface 714 , and the memory 720 ).

[0227] It should be noted that although the above device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0228] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0229] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A video generation method, characterized in that: The method comprises: Acquiring input data, wherein the input data includes first noise data and a control condition; Using a first video generation model, denoising the first noise data based on the control condition to generate an initial latent space representation; the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data; Using a second video generation model, converting the initial latent space representation into a target latent space representation; wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation; The target latent space representation is decoded to generate a target video.

2. The method according to claim 1, characterized in that The step of using the first video generation model to denoise the first noise data based on the control condition to generate an initial latent space representation includes: performing, using the first video generation model, an initial denoising process for k time steps on the first noise data at an original resolution of the first noise data based on the control condition to obtain a first intermediate latent space representation, where k is a positive integer less than T, and T is a maximum time step of the denoising process; Based on the first intermediate latent space representation, estimating the noise-free latent space representation corresponding to the k-th time step; spatially downsampling the noise-free latent space representation to generate a second intermediate latent space representation; wherein the spatial resolution of the second intermediate latent space representation is less than the original resolution of the noisy data; Injecting second noise data corresponding to the k-th time step into the second intermediate latent space representation to generate a third intermediate latent space representation; The first video generation model is used to further perform denoising on the third intermediate latent space representation at the spatial resolution of the second intermediate latent space representation to generate the initial latent space representation.

3. The method according to claim 1, characterized in that The converting the initial latent space representation into a target latent space representation by using the second video generation model includes: Using the second video generation model, performing spatial upsampling mapping processing on the initial latent space representation to obtain a fourth intermediate latent space representation; the spatial resolution of the fourth intermediate latent space representation is greater than the spatial resolution of the initial latent space representation; The fourth intermediate latent space representation is optimized using a three-dimensional self-attention layer to obtain the target latent space representation.

4. The method according to claim 3, characterized in that The optimizing the fourth intermediate latent space representation by using the three-dimensional self-attention layer to obtain the target latent space representation includes: Dividing the fourth intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows containing latent space representations of a fixed number of frame sequences; Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows; fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fifth intermediate latent space representation; Performing a cyclic shift operation on the fifth intermediate latent space representation and re-dividing the fifth intermediate latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences; Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window; The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

5. The method according to claim 4, characterized in that The performing three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window includes: Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

6. The method according to any one of claims 1 to 5, characterized in that The parameter scale of the first video generation model is greater than the parameter scale of the second video generation model.

7. A method for training a video generation model, characterized in that: The method comprises: Acquire training data including a plurality of training samples, wherein the training samples include input data samples and target video samples, wherein the input data samples include control conditions and noise samples, and the noise samples are obtained by adding noise to the target video samples; denoising the noise sample based on the control condition using a first video generation model to generate an initial latent space representation, wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the target video sample; Converting the initial latent space representation into a target latent space representation using a second video generation model, wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation; Decoding the target latent space representation to generate a target video; A loss function is determined using the target video and the target video sample, and the model parameters of the second video generation model are updated using the loss function.

8. The method according to claim 7, characterized in that The second video generation model includes a mapping layer and a three-dimensional self-attention layer; The converting the initial latent space representation into a target latent space representation by using a second video generation model includes: Performing spatial upsampling mapping processing on the initial latent space representation using the mapping layer to obtain an intermediate latent space representation; the spatial resolution of the intermediate latent space representation is greater than the spatial resolution of the initial latent space representation; The intermediate latent space representation is optimized using the three-dimensional self-attention layer to obtain the target latent space representation.

9. The method according to claim 8, characterized in that The optimizing the intermediate latent space representation by using the three-dimensional self-attention layer to obtain the target latent space representation includes: Dividing the intermediate latent space representation into a plurality of non-overlapping first windows according to the time dimension, each of the first windows contains the latent space representation of a fixed number of frame sequences; Performing a three-dimensional self-attention calculation on the latent space representation in each of the first windows to obtain a first optimized latent space representation corresponding to each of the first windows; Fusing the first optimized latent space representations corresponding to each of the first windows to obtain a fused latent space representation; Performing a cyclic shift operation on the fused latent space representation and re-dividing the fused latent space representation into a plurality of non-overlapping second windows, each of the second windows containing the latent space representation of the fixed number of frame sequences; Performing a three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window; The second optimized latent space representation corresponding to each of the second windows is fused to generate the target latent space representation.

10. The method according to claim 9, characterized in that The performing three-dimensional self-attention calculation on the latent space representation in each second window to obtain a second optimized latent space representation corresponding to each second window includes: Masking is performed on two discontinuous frames in the time dimension in each of the second windows, and then three-dimensional self-attention calculation is performed to obtain a second optimized latent space representation corresponding to each of the second windows.

11. A video generating device, characterized in that: The device comprises: a data acquisition unit configured to acquire input data, wherein the input data includes first noise data and a control condition; a first processing unit configured to, using a first video generation model, perform denoising on the first noise data based on the control condition to generate an initial latent space representation; wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the first noise data; a second processing unit configured to convert the initial latent space representation into a target latent space representation using a second video generation model; wherein a spatial resolution of the target latent space representation is greater than a spatial resolution of the initial latent space representation; The video generation unit is configured to decode the target latent space representation to generate a target video.

12. A training device for a video generation model, characterized in that: The device comprises: a sample acquisition unit configured to acquire training data comprising a plurality of training samples, wherein the training samples include input data samples and target video samples, wherein the input data samples include control conditions and noise samples, and the noise samples are obtained by adding noise to the target video samples; The model training unit is configured to use the first video generation model to denoise the noise sample based on the control condition to generate an initial latent space representation, wherein the spatial resolution of the initial latent space representation is smaller than the original resolution of the target video sample; use the second video generation model to convert the initial latent space representation into a target latent space representation, wherein the spatial resolution of the target latent space representation is greater than the spatial resolution of the initial latent space representation; decode the target latent space representation to generate a target video; use the target video and the target video sample to determine a loss function, and use the loss function to update the model parameters of the second video generation model.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the steps of the method according to any one of claims 1 to 10 are implemented.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being configured to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed, the steps of the method according to any one of claims 1 to 10 are implemented.

Images