Video generation method and apparatus, and method and apparatus for training video generation model

By combining the feature representation of spatial and temporal conditions and the noise sequence for denoising and combining the cross-attention processing of global conditional feature expression, the problem of insufficient diversity and flexibility in existing video generation technologies is solved, and the video quality is significantly improved and controllability.

WO2024248736A1PCT designated stage expired Publication Date: 2024-12-05ALIBABA INNOVATION PRIVATE LIMITED

Patent Information

Application Number
PCT/SG2024/050361
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-29
Filing Date
2024-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing text-based video generation technologies lack diversity and flexibility, resulting in lower quality of generated videos.

Method used

The video generation method is adopted to integrate the feature representation of spatial conditions and temporal conditions with the noise sequence, and denoising the diffusion model is used to perform cross-attention processing, and to generate videos by combining global conditional feature expression.

Benefits of technology

It improves the controllability and diversity of video generation, and the generated video is more flexible and high-quality, which can effectively capture inter-frame dynamics and ensure consistency of time and space perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050361_05122024_PF_FP_ABST
    Figure SG2024050361_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the embodiments of the present disclosure are a video generation method and apparatus, and a method and apparatus for training a video generation model. The main technical solution includes: acquiring a local condition, wherein the local condition comprises a space condition and / or a time condition; encoding the local condition, so as to obtain a local condition feature representation; integrating a noise sequence and the local condition feature representation, so as to obtain a noise hidden vector sequence; using a diffusion model to perform denoising processing on the noise hidden vector sequence, so as to obtain a denoised hidden vector sequence; and using the denoised hidden vector sequence to perform decoding processing, so as to obtain a video.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Video Generation Method, Method and Apparatus for Training Video Generation Model Cross-Reference This disclosure claims priority to Chinese Patent Publication No. 202310618707.8, filed with the Patent Office of China on May 29, 2023, entitled "Video Generation Method, Method and Apparatus for Training Video Generation Model," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of computer vision technology, and more particularly to a video generation method, a method and apparatus for training a video generation model. Background The continuous development of artificial intelligence technology has provided new opportunities for computer vision technology, with the ability to automatically generate videos becoming increasingly powerful. Text-based video generation technology provides new tools for many content creators, making the creation of video content, which previously required professional expertise and expensive equipment, easier and more cost-effective. However, most current technologies use text as a guide, resulting in a lack of diversity and flexibility in the generated videos, resulting in low video quality. SUMMARY In view of this, this disclosure provides a video generation method, a method and apparatus for training a video generation model, to improve the quality of generated videos. The present disclosure provides the following solutions: In a first aspect, a video generation method is provided, comprising: obtaining local conditions, wherein the local conditions include spatial conditions and / or temporal conditions; encoding the local conditions to obtain local condition feature representations; integrating a noise sequence with the local condition feature representations to obtain a noise latent vector sequence; denoising the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; and decoding the denoised latent vector sequence to obtain a video. According to an implementation of an embodiment of the present disclosure, the method further comprises: obtaining global conditions, wherein the global conditions include at least one of text conditions, style conditions, and color conditions; encoding the global conditions to obtain a global condition feature representation; and denoising the noise latent vector sequence using a diffusion model comprises: performing a cross-attention process on the noise latent vector sequence using the global condition feature representations to predict noise, and performing denoising using the predicted noise. According to an implementable manner in an embodiment of the present disclosure, the spatial condition includes at least one of a single image and a single semantic sketch; and the temporal condition includes at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence, and a grayscale map sequence.According to one possible implementation of the embodiments of the present disclosure, encoding the global condition to obtain a global condition feature representation includes: performing text encoding on the text condition to obtain a text feature representation, and performing image encoding on the style condition and / or color condition to obtain an image feature representation; and integrating the text feature representation and the image feature representation to obtain a global condition feature representation. According to one possible implementation of the embodiments of the present disclosure, encoding the local condition to obtain a local condition feature representation includes: using each spatiotemporal condition encoder to encode each local condition to obtain a feature tensor corresponding to each local condition, wherein the spatiotemporal condition encoder corresponds to the local condition in a one-to-one manner; and fusing the feature tensors corresponding to each local condition to obtain the local condition feature representation. According to an implementable method in an embodiment of the present disclosure, encoding each local condition separately using a spatiotemporal condition encoder includes: using the spatiotemporal condition encoder to perform spatial feature encoding on the local condition to obtain a spatial feature representation of the local condition; if the local condition is a sequence, performing temporal self-attention processing on the spatial feature representation of the local condition to obtain a feature tensor corresponding to the local condition; otherwise, copying the spatial feature representation of the local condition in a temporal sequence to generate a temporal spatial feature representation, performing temporal self-attention processing on the temporal spatial feature representation, and obtaining a feature tensor corresponding to the local condition. In a second aspect, a method for training a video generation model is provided, the method comprising: obtaining first training data comprising multiple first training samples, the first training samples comprising local condition samples, the local condition samples comprising spatial condition samples and / or temporal condition samples; using the first training data to train a video generation model, the video generation model comprising: a spatiotemporal condition encoder, a diffusion model, and a decoder; wherein the spatiotemporal condition encoder encodes the local condition samples to obtain a local condition feature representation; the diffusion model denoises a noisy latent vector sequence to obtain a denoised latent vector sequence, the noisy latent vector sequence being obtained by integrating the noise sequence with the local condition feature representation; the decoder performs decoding processing on the denoised latent vector sequence to obtain a video; the training objectives include: minimizing the difference between the noise predicted by the diffusion model during denoising and Gaussian noise.According to one possible implementation of the embodiments of the present disclosure, the first training samples further include global condition samples, which include at least one of text condition samples, style condition samples, and color condition samples. The video generation model further includes a global encoder, which encodes the global condition samples to obtain a global condition feature representation. De-noising the noisy latent vector sequence by the diffusion model includes: the diffusion model using the global condition feature representation to perform cross-attention processing on the noisy latent vector sequence to predict noise, and performing de-noising using the predicted noise. According to one possible implementation of the embodiments of the present disclosure, the global encoder and decoder use parameters obtained through pre-training. In each iteration of the training, the parameters of the spatiotemporal condition encoder and diffusion model are updated using a loss function corresponding to the training objective. According to an implementable manner in an embodiment of the present disclosure, obtaining first training data including multiple first training samples includes: obtaining a video sample; obtaining a descriptive text, a style image and / or a color histogram of the video sample as the text condition sample, the style condition sample and / or the color condition sample, respectively; extracting at least one of a single image and a single semantic sketch as the spatial condition sample; and extracting at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence and / or a grayscale map sequence as the temporal condition sample. According to an implementable method of an embodiment of the present disclosure, before using the first training data to train the video generation model, the method further includes pre-training the diffusion model; further using the first training data to train the video generation model based on the parameters of the diffusion model obtained through pre-training; wherein, pre-training the diffusion model includes: obtaining second training data including multiple second training samples, the second training samples including: extracting descriptive text from video samples as text samples; encoding the text samples using a global encoder to obtain text feature representations; inputting the text feature representations and a noise sequence into a diffusion model to train the diffusion model, wherein the diffusion model denoises the noise sequence using the text feature representations; the training objectives include: minimizing the difference between the noise predicted by the diffusion model when performing denoising at each time step and the Gaussian noise. According to an implementable manner in an embodiment of the present disclosure, encoding the local condition samples by the spatiotemporal condition encoder to obtain the local condition feature representation includes: using each spatiotemporal condition encoder to respectively encode each local condition sample to obtain a feature tensor corresponding to each local condition sample, wherein the spatiotemporal condition encoder corresponds to the local condition; and fusing the feature tensors corresponding to each local condition sample to obtain the local condition feature representation.According to an implementable manner in an embodiment of the present disclosure, encoding each local conditional sample using each spatiotemporal conditional encoder includes: using the spatiotemporal conditional encoder to perform spatial feature encoding on the local conditional sample to obtain a spatial feature representation of the local conditional sample; if the local conditional sample is a sequence, performing temporal self-attention processing on the spatial feature representation of the local conditional sample to obtain a feature tensor corresponding to the local conditional sample; otherwise, copying the spatial feature representation of the local conditional sample in a temporal sequence to generate a temporal spatial feature representation, performing temporal self-attention processing on the temporal spatial feature representation, and obtaining a feature tensor corresponding to the local conditional sample. In a third aspect, a video generation method is provided, which is executed by a cloud server. The method includes: obtaining local conditions from a user terminal, where the local conditions include spatial conditions and / or temporal conditions; encoding the local conditions to obtain a local condition feature representation; integrating a noise sequence with the local condition feature representation to obtain a noise latent vector sequence; denoising the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; decoding the denoised latent vector sequence to obtain a video; and sending the video to the user terminal for display. In a fourth aspect, a video generation device is provided, comprising: a condition acquisition unit configured to acquire local conditions, wherein the local conditions include spatial conditions and / or temporal conditions; a video generation unit configured to encode the local conditions to obtain a local condition feature representation; integrate a noise sequence with the local condition feature representation to obtain a noise latent vector sequence; denoise the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; and decode the denoised latent vector sequence to obtain a video.In a fifth aspect, a device for training a video generation model is provided. The device comprises: a sample acquisition unit configured to acquire first training data comprising multiple first training samples, wherein the first training samples comprise local condition samples, including spatial condition samples and / or temporal condition samples; and a model training unit configured to train a video generation model using the first training data. The video generation model comprises: a spatiotemporal condition encoder, a diffusion model, and a decoder. The spatiotemporal condition encoder encodes the local condition samples to obtain a local condition feature representation; the diffusion model denoises a noisy latent vector sequence to obtain a denoised latent vector sequence, wherein the noisy latent vector sequence is obtained by integrating the noise sequence with the local condition feature representation; and the decoder decodes the denoised latent vector sequence to obtain a video. The training objective comprises minimizing the difference between noise predicted by the diffusion model during denoising and Gaussian noise. In a sixth aspect, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the steps of the method described in any one of aspects 1 to 3 above. According to a seventh aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions. When read and executed by the one or more processors, the program instructions execute the steps of any of the methods described in any of the first to third aspects. According to specific embodiments provided herein, the present disclosure provides the following technical effects:

[0002] 1) This disclosure incorporates spatial and / or temporal conditions as local conditions. This local conditional feature representation is integrated with a noise sequence to generate a noise latent vector sequence. This latent vector sequence is then denoised to obtain a denoised latent vector sequence, which is then decoded to generate a video. This approach no longer limits video generation to textual conditions. Instead, spatial and / or temporal conditions are introduced to guide video generation, enabling more flexible and diverse video generation and improving video quality.

[0003] 2) This disclosure can generate the desired video by combining multiple global conditions and spatial and temporal conditions from multiple local conditions. A spatiotemporal conditional encoder is used to encode, align, and fuse the various local conditions, integrating them with the noise sequence to achieve local control of the video. A global encoder encodes and integrates the global conditions, and a diffusion model uses the global conditional feature representation to perform cross-attention processing on the noise latent vector sequence, achieving global control of the video. This approach significantly improves the controllability of video generation, making the generated videos more flexible and diverse.

[0004] 3) This disclosure innovatively introduces a motion vector sequence into the temporal condition. By incorporating the control signal of the motion vector sequence, the video generation model can capture the inter-frame dynamics, thereby achieving control over the internal motion in the video.

[0005] 4) The spatiotemporal conditional encoder provided by this disclosure first extracts local spatial information and then performs temporal modeling, facilitating explicit embedding in the temporal domain. It also provides a unified interface for different local conditions, enhancing inter-frame consistency. For non-temporal spatial conditions, such as single images or single semantic sketches, local conditions are fused by replicating them in the temporal dimension to ensure consistency with the temporal condition. This encoding and fusion process enables greater controllability of the resulting video in terms of both temporal and spatial perception.

[0006] 5) Before training the video generation device, the present disclosure can pre-train the diffusion model using training data from text-generated videos. The video generation model can then be further trained based on the pre-trained diffusion model parameters, thereby improving the effectiveness and efficiency of the video generation model. Of course, implementing any product of the present disclosure does not necessarily require achieving all of the advantages described above. BRIEF DESCRIPTION OF THE DRAWINGS To more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly describes the drawings used in the embodiments. Obviously, the drawings described below represent some embodiments of the present disclosure. Those skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 is a diagram of the system architecture applicable to embodiments of the present disclosure; Figure 2 is a flow chart of a video generation method provided by an embodiment of the present disclosure; Figure 3 is a schematic diagram of the conditions required for video generation provided by an embodiment of the present disclosure; Figure 4 is a schematic diagram of the principle of a video generation model provided by an embodiment of the present disclosure; Figure 5 is a schematic diagram of the structure of a spatiotemporal conditional encoder provided by an embodiment of the present disclosure; Figure 6 is a schematic diagram of the principle of a diffusion model provided by an embodiment of the present disclosure at various time steps; Figures 72 to 7d are four example diagrams of generated videos provided by an embodiment of the present disclosure; Figure 8 is a flow chart of a method for training a video generation model provided by an embodiment of the present disclosure; Figure 9 is a schematic block diagram of a video generation device provided by an embodiment of the present disclosure; Figure 10 is a schematic block diagram of an apparatus for training a video generation model provided by an embodiment of the present disclosure; and Figure 11 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. It should be understood that the described embodiments represent only a portion of the embodiments of the present disclosure, but not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure are intended to fall within the scope of protection of the present disclosure. The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are intended to include the plural forms, unless the context clearly indicates otherwise. It should be understood that the term "and / or" as used herein describes an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " herein generally indicates that the associated objects are in an "or" relationship.Depending on the context, the term "if" as used herein can be interpreted as "at the time of," "when," "in response to determining," or "in response to detecting." Similarly, depending on the context, the phrases "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when determined," "in response to determining," "when (a stated condition or event) is detected," or "in response to detecting (a stated condition or event)." To facilitate understanding of the present disclosure, a brief description of the system architecture to which the present disclosure applies is first provided. Figure 1 illustrates an exemplary system architecture to which embodiments of the present disclosure can be applied. As shown in Figure 1, the system architecture includes a user terminal, as well as a model training device and a video generation device located on a server. The model training device on the server uses the method provided in embodiments of the present disclosure to perform model training in an offline phase to obtain a video generation model. The user terminal can exchange information with the video generation device on the server via a network. For example, a user terminal can send the condition information for video generation input by the user to a video generation device via a network. The condition information may include global conditions and local conditions, as described in detail in subsequent embodiments. The video generation device can use a trained video generation model to generate a video under the constraints of the global and local conditions, and send the generated video to the user terminal via a network for display. Such user terminals may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, smart medical devices, and PCs (Personal Computers). Smart mobile devices may include mobile phones, tablet computers, laptop computers, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include smart home appliances, such as smart TVs and smart refrigerators, that have video playback capabilities. Wearable devices may include smart watches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality) devices, mixed reality devices (i.e., devices that support both VR and AR), and the like. The model training device and video generation device may be configured as separate servers, on the same server or server group, or on separate or shared cloud servers.A cloud server, also known as a cloud computing server or cloud host, is a host product within a cloud computing service system. It addresses the management difficulties and weak service scalability of traditional physical hosts and virtual private servers (VPS) services. The model training device and video generation device can also be installed on a computer terminal with strong computing power. It should be noted that the aforementioned video generation device can also generate video offline, in addition to performing video generation online. It should be understood that the number of user terminals, model training devices, video generation devices, and video generation models shown in Figure 1 is illustrative. Any number of user terminals, model training devices, video generation devices, and video generation models can be used depending on implementation requirements. Figure 2 is a flow chart of a video generation method provided in an embodiment of the present disclosure. This method can be performed by the video generation device in the system architecture shown in Figure 1. As shown in Figure 2, the method may include the following steps: Step 202: Obtaining local conditions, which may include spatial conditions and / or temporal conditions. Step 204: Encoding the local conditions to obtain a local condition feature representation. Step 206: Integrate the noise sequence with the local conditional feature representation to obtain a noise latent vector sequence; denoise the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence. Step 208: Decode the denoised latent vector sequence to obtain a video. As can be seen from the above process, the present disclosure incorporates spatial and / or temporal conditions as local conditions, integrates the local conditional feature representation with the noise sequence to obtain a noise latent vector sequence, and denoises the noise latent vector sequence using the global conditional feature representation to obtain a denoised latent vector sequence, which is then decoded to obtain a video. This approach eliminates the need for textual conditions in video generation, introducing spatial and / or temporal conditions to guide video generation, thereby enabling more flexible and diverse video generation and improving video quality. Each step in the above process is described in detail below. First, step 202, "obtaining local conditions," is described in detail in conjunction with an embodiment. Existing video generation methods typically generate videos guided by textual conditions, resulting in limited flexibility and diversity. In the disclosed embodiments, new combinatorial conditions are introduced to improve the controllability of video generation. These conditions include at least local conditions and may further include global conditions. Global conditions primarily include text conditions, style conditions, and / or color conditions. Text conditions primarily describe the video content, globally reflecting the content of the video to be generated.Style conditions typically employ a style image. The goal is to generate a video with stylistic features consistent with the style image, meaning that the video as a whole possesses the style expressed in the style image. Color conditions can take the form of a color histogram, for example, and their goal is to guide the color distribution of the generated video. Local conditions primarily include spatial conditions and / or temporal conditions. Spatial conditions can include at least one of a single image and a single semantic sketch. A single image is a single image that constrains the content and structure of the image frames in a video. Semantic sketches, also known as hand-drawn sketches or sketches, are a method for describing the basic semantics of objects in an image. They are a generalized representation technique that primarily depicts the edge (or outline) features of an object, ignoring details and redundant features while retaining the key information. As can be seen, spatial conditions primarily guide video content based on spatial features. Temporal conditions include at least one of a motion vector sequence, a depth map sequence, a mask map sequence, and a semantic sketch sequence. Since videos are image sequences, temporal conditions provide more detailed guidance along the temporal dimension. The motion vector sequence contains motion vectors between adjacent frames. Motion vectors indicate the direction of motion of each token (element) between adjacent frames, including horizontal and vertical directions. Therefore, motion vectors can be represented as two-dimensional vectors. They explicitly express the movement of each token between two adjacent image frames, ensuring that the video generated by the video generation model has motion controllability. Each token in an image refers to the element that constitutes the image. For an image, if the image is divided into non-overlapping blocks, the blocks and start symbols in the image are all tokens. The depth map sequence consists of the depth map of each frame. The depth map contains the depth information of each token in the image frame and is used to guide the depth information of each frame in the video. The mask map sequence consists of the mask map of each frame. The mask map masks the content of a certain area in the image frame, enabling the model to predict the content of the masked area. The masked area can be user-specified or randomly determined. The semantic sketch sequence consists of the semantic sketch of each frame. Compared with a single semantic sketch in spatial conditions, sketch sequences provide more control details. The grayscale image sequence consists of grayscale images of each frame. The grayscale image contains the grayscale information of each token in the image frame and is used to guide the grayscale information of each frame in the video. The length of each sequence in the above time conditions is consistent and equal to the length of the generated video, for example, denoted as T1.For example, the text condition is "a child standing next to a santa claus tree." The style condition is a style image, which is used to constrain the overall style of the generated video, that is, to transfer the style features of the style image to the generated video. The color condition is a color histogram, as shown in Figure 3, which is used to constrain the color distribution of the generated video. The single image in the spatial condition can be an image of a child and a Christmas tree, which is used to constrain the appearance of the child and the Christmas tree in the generated video, as shown in Figure 3. The semantic sketch can be a diagram depicting the basic lines of the child and the Christmas tree, which is used to constrain the basic shape and position of the child and the Christmas tree. The motion vector sequence in the temporal condition contains the motion vectors between adjacent frames, schematically represented by small arrows in the figure. The depth map sequence contains the depth map of each frame, the mask map sequence contains the mask map of each frame, and the semantic sketch sequence contains the semantic sketch of each frame. The length of each sequence is T1, where T1 is the length of the generated video. As one possible implementation method, a condition input interface can be provided to the user, where the user can select and enter the above-mentioned global and local conditions. For example, the user can enter text conditions in the text input box. For another example, a user can use the image drawing or editing tools provided on the interface to input a single image and / or a single semantic sketch as a spatial condition, and at least one of a motion vector sequence, depth map sequence, mask map sequence, and semantic map sequence as a temporal condition. A style image can also be input by uploading a self-selected image or selecting an image from the image library provided on the interface. When inputting a motion vector sequence, the user can select a specific area in the image and generate a motion trajectory using specific gestures, mouse traces, or other methods to indicate the specific area's motion path in the video. The server automatically generates a motion vector sequence based on the motion trajectory. In addition to the conditional input methods listed above, other conditional input methods can also be used, and these are not exhaustive. The following describes in detail step 204, namely, "encoding the local condition to obtain a local conditional feature representation, encoding the global condition to obtain a global conditional feature representation, and encoding the local condition to obtain a local conditional feature representation," in conjunction with an embodiment. Steps 204 through 208 in FIG. 2 can be implemented using a pre-trained video generation model. As shown in FIG4 , the video generation model may include a spatiotemporal conditional encoder, a diffusion model, and a decoder, and may further include a global encoder. This step is primarily performed by the spatiotemporal conditional encoder. If global conditions are used, this step is further performed by the global encoder. The global encoder is used to encode the global conditions to obtain a global conditional feature representation.If the global condition includes a text condition, a text encoder can be used to encode the text condition to obtain a text feature representation, which is then used as the global condition feature representation. If the global condition includes both a text condition and a style image, a text encoder can be used to encode the text condition to obtain a text feature representation, and an image encoder can be used to encode the style condition and / or color condition to obtain an image feature representation. The text feature representation and image feature representation are then integrated to obtain the global condition feature representation. The text encoder can be implemented using a pretrained language model, such as BERT (Bidirectional Encoder Representation from Transformers), XLNet (an autoregressive model that implements bidirectional contextual information through a permutation language model), GPT (Generative Pre-Training) model, or CLIP (an encoding model that enables multimodal encoding). The text encoder effectively obtains a semantic embedding of the text condition, namely, a text feature representation. The image encoder can use VIT (Vision Transformer), CLIP, etc. Through the image encoder, the semantic embedding of the style image, that is, the image feature representation, can be obtained. The spatiotemporal conditional encoder is used to encode local conditions to obtain local conditional feature representations. Since local conditions include spatial and / or temporal conditions, and since temporal conditions are sequential, they contain rich and complex spatiotemporal relationships, posing challenges to controllable video guidance. In view of this, in the embodiments of the present disclosure, a corresponding spatiotemporal conditional encoder is deployed for each local condition. As one feasible approach, a one-to-one correspondence exists between the local conditions and the spatiotemporal conditional encoders. Each spatiotemporal conditional encoder is used to encode each local condition, obtaining a feature tensor corresponding to each local condition. These feature tensors are then fused to obtain a local conditional feature representation. The structure of the spatiotemporal conditional encoder can be shown in Figure 5. The spatiotemporal conditional encoder first encodes the spatial features of the local condition to obtain a spatial feature representation of the local condition. As shown in Figure 5, the spatiotemporal conditional encoder can consist of two two-dimensional convolutions (Conv2D), two activation layers (e.g., using the SiLU activation function), and an average pooling layer. If the local condition input to the spatiotemporal conditional encoder is a sequence, temporal self-attention is performed on the spatial feature representation of this local condition (i.e., in temporal order) to obtain a feature tensor corresponding to the local condition. This process is performed by the temporal Transformer shown in Figure 5. If the local condition input to the spatiotemporal conditional encoder is not a sequence, such as a single image or a single semantic sketch, the spatial feature representation of this local condition can first be replicated in temporal order to generate a temporal spatial feature representation. This process is performed by the "tooth" in Figure 5, aligning it with the spatial feature representation corresponding to the sequence-type local condition. The temporal Transformer then performs temporal self-attention on the temporal spatial feature representation to obtain a feature tensor corresponding to the local condition. As one feasible approach, the feature tensors corresponding to the local conditions can be fused by element-wise addition. It can be seen that the spatiotemporal conditional encoder actually first extracts local spatial information and then performs temporal modeling, which facilitates explicit embedding in the temporal domain. It also provides a unified interface for different local conditions, enhancing inter-frame consistency. For non-temporal spatial conditions, such as single images or semantic sketches, local conditions are fused by replicating them in the temporal dimension to ensure consistency with the temporal conditions. This encoding and fusion process enables greater controllability of the resulting video in terms of both temporal and spatial perception. In particular, the motion vector sequence introduced in the temporal condition captures inter-frame dynamics, enabling direct control of internal motion within the video.The following describes in detail step 206, "integrating the noise sequence with the local conditional feature representation to obtain a noise latent vector sequence; and denoising the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence," with reference to an embodiment. In this disclosed embodiment, a noise sequence & can be randomly generated. This randomly generated noise sequence conforms to a normal distribution, i.e., Gaussian noise, and its length is consistent with the length of the video to be generated, both being T1. After encoding and integrating the local conditions using a spatiotemporal conditional encoder, the local conditional feature representation and & have the same spatial shape. The noise sequence and the local conditional feature representation are then integrated, for example, by concatenating them along the channel dimension, to obtain a noise latent vector sequence z,. This noise latent vector sequence serves as a control signal for video generation. Diffusion models have been widely used in image generation due to their more stable training and generation flexibility, but have not yet been well utilized in video generation. To efficiently process video data, the disclosed embodiments introduce an IDM (latent diffusion model) to maintain local fidelity. This involves projecting the initial video onto a latent representation, which is then mapped back into pixel space via a decoder to produce the final video. The initial video is the noise sequence, and the final video is the generated video. In addition to the IDM, other types of diffusion models can also be used. Diffusion model processing can be understood as predicting normally distributed noise and performing denoising at each time step to restore the true video content. This process simulates the reverse process of a Markov chain of length . Where T is the total time step of the diffusion model. A longer T yields better denoising results, but also increases the impact on computational performance. Therefore, a trade-off between these two factors is necessary. An empirical or experimental value can be used, for example, 1000c. At each time step, the diffusion model uses the noisy latent vector sequence obtained at the previous time step (at the first time step, the diffusion model uses the noisy latent vector sequence input to predict the noise at the current time step). The predicted noise is then used to denoise the noisy latent vector sequence, resulting in the noisy latent vector sequence at the current time step. Furthermore, if the input conditions include global conditions, the diffusion model uses the global condition feature representation to perform a cross-attention process on the noisy latent vector sequence to predict the noise, and then uses the predicted noise for denoising. Specifically, as shown in Figure 6, at the first time step, the diffusion model uses the global condition feature representation to perform a cross-attention process on the noisy latent vector sequence to predict the noise at the current time step. The predicted noise is then used to denoise the noisy latent vector sequence, resulting in the noisy latent vector sequence at the current time step.At other time steps, the noisy latent vector sequence obtained in the previous time step is subjected to a cross-attention process using the global conditional feature representation to predict the noise in the current time step. The predicted noise is then used to denoise the noisy latent vector sequence obtained in the previous time step, resulting in the noisy latent vector sequence for the current time step. The diffusion model can use a three-dimensional UNet as its backbone network. The UNet is an encoder-decoder network that incorporates skip connections. Named for its structure resembling the letter U, it combines deep, high-level features with shallow, low-level features. Since it is a well-known network, it will not be described in detail here. The above process can be viewed as applying the denoising function scoop (•,• / ) to the noisy latent vector sequence z and the condition (e.g., including global and local conditions) c, where / represents 1,...,0. During the denoising process, a cross-attention mechanism is used to inject global conditions, allowing them to guide video generation from an overall semantic perspective. Finally, the denoised latent vector sequence obtained in the final step of the extended model is input into the decoder in step 208. The decoder then decodes the denoised latent vector sequence to produce the video. The decoder can utilize the decoder of existing video generation models and will not be described in detail here. The above-described methods provided by the embodiments of this disclosure allow for flexible combination of various conditions to generate more diverse videos. Several examples illustrate the controllability of video generation in this disclosure. Example 1: The text condition in the global condition can be combined with the temporal condition in the local condition. As shown in Figure 7a, the user inputs the text condition "a rotated perspective of a long-haired woman standing in a forest" and a sequence of semantic sketches. In this example, the sequence length is six frames. The resulting video is shown in Figure 7a, which illustrates the individual frames in the video. Example 2: The spatial and temporal conditions in the local condition can be combined. The user inputs a single image and a depth map sequence, resulting in the video shown in Figure 7b. Example 3: The style condition in the global condition can be combined with the depth map sequence and semantic sketch sequence in the local condition. The user inputs a style image, a depth map sequence, and a semantic sketch sequence, resulting in the video shown in Figure 7c. Example 4: Global conditions can combine text and style conditions with local conditions such as a semantic sketch and a motion vector sequence. A user enters the text condition "a moving golden moon," a style image, and a semantic sketch. The user can also hand-draw the direction of the moon's motion on the semantic sketch. The server's video generation device can automatically generate a motion vector sequence based on the moon's motion direction drawn on the semantic sketch, thereby generating a video using the methods provided by the embodiments of this disclosure.As shown in Figure 7d, the moon in the generated video moves along the direction of the user's hand-drawn motion. The training process of the video generation model described above will be described in detail below in conjunction with embodiments. Figure 8 is a flow chart of a method for training a video generation model according to an embodiment of the present disclosure. This method can be executed by the model training device in the system architecture shown in Figure 1. As shown in Figure 8, the method may include the following steps: Step 802: Obtain first training data comprising multiple first training samples. The first training samples include local condition samples, where the local conditions include spatial condition samples and / or temporal condition samples. Furthermore, the first training data samples may also include global condition samples, where the global condition samples may include at least one of text condition samples, style condition samples, and color condition samples. As one possible implementation, a video sample may be first obtained, and a descriptive text of the video sample may be obtained as a text condition sample. A style image of the video sample may be obtained as a style condition sample, and a color histogram of the video sample may be obtained as a color condition sample. For example, the description text of the video sample can be obtained using an existing video description text generation model, or it can be obtained from the webpage where the video sample originated, or it can be manually added. For example, the first frame of the video sample can be used as the style image. For another example, the color histogram corresponding to the first frame of the video sample can be used as the color condition sample. Then, at least one of a single image and a single semantic sketch can be extracted from the video sample as the spatial condition sample. For example, the first frame of the video sample can be used as the single image. For another example, edge extraction can be performed on the first frame of the video sample, and a single semantic sketch can be generated using the edge information extracted from the image. Alternatively, a single semantic sketch can be generated for the first frame of the video sample using a hand-drawn drawing generation tool. Then, at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence, and a grayscale map sequence can be extracted from the video sample as the temporal condition sample. For example, the motion vectors between adjacent frames of the video sample can be extracted to form a motion vector sequence. The depth map sequence can be extracted from each frame of the video sample to form a depth map sequence. Masking is performed on each frame in the video sample to obtain a mask image sequence. Semantic sketches are generated for each frame image in the video sample to obtain a semantic sketch sequence. Grayscale images are generated for each frame image in the video sample to obtain a grayscale image sequence. In addition to the above method, other methods can also be used to obtain the above conditional samples, which are not listed here.It should be noted that the definitions of "first" and "second" in the embodiments of the present disclosure do not have restrictions on size, order, or quantity, but are used to distinguish between the names. For example, "first training sample" and "second training sample" are used to distinguish between two training samples. Step 804: Train a video generation model using the first training data. The video generation model includes: a spatiotemporal conditional encoder, a diffusion model, and a decoder. The spatiotemporal conditional encoder encodes the local conditional samples to obtain a local conditional feature representation. The diffusion model denoises the noisy latent vector sequence to obtain a denoised latent vector sequence. The noisy latent vector sequence is obtained by integrating the noise sequence with the local conditional feature representation. The decoder uses the denoised latent vector sequence to perform decoding to obtain a video. The training objective includes minimizing the difference between the noise predicted by the diffusion model during denoising and Gaussian noise. Furthermore, the video generation model may further include a global encoder. The global encoder encodes the global conditional samples to obtain a global conditional feature representation. Accordingly, The diffusion model can use the global conditional feature representation to perform cross-attention processing on the noisy latent vector sequence to predict noise, and use the predicted noise for denoising. As one possible implementation, if the global condition sample includes a text condition sample, the global encoder can use a text encoder to perform text encoding on the text condition sample to obtain a text feature representation, and use this text feature representation as the global conditional feature representation. As another possible implementation, if the global condition includes style condition samples and / or color condition samples in addition to the text condition sample, the global encoder can use a text encoder to perform text encoding on the text condition sample to obtain a text feature representation, and use an image encoder to perform image encoding on the style condition sample and / or color condition sample to obtain an image feature representation; the text feature representation and image feature representation are then integrated to obtain the global conditional feature representation. As another possible implementation, each local condition sample can be encoded using a spatiotemporal condition encoder to obtain a feature tensor corresponding to each local condition sample, where the spatiotemporal condition encoder corresponds one-to-one with the local condition; the feature tensors corresponding to each local condition sample are then fused to obtain the local condition feature table 7F o, where: First, a spatiotemporal conditional encoder may be used to perform spatial feature encoding on the local conditional sample to obtain a spatial feature representation of the local conditional sample.If the local conditional sample is a sequence, temporal self-attention is performed on the spatial feature representation of the local conditional sample to obtain a feature tensor corresponding to the local conditional sample. Otherwise, the spatial feature representation of the local conditional sample is replicated in time to generate a temporal spatial feature representation, which is then subjected to temporal self-attention to obtain a feature tensor corresponding to the local conditional sample. As one possible implementation, the diffusion model can use the global conditional feature representation to perform cross-attention on the noisy latent vector sequence at the first time step to predict the noise at the current time step. The predicted noise is then used to denoise the noisy latent vector sequence to obtain the noisy latent vector sequence at the current time step. At other time steps, the diffusion model uses the global conditional feature representation to perform cross-attention on the noisy latent vector sequence obtained at the previous time step to predict the noise at the current time step. The predicted noise is then used to denoise the noisy latent vector sequence obtained at the previous time step to obtain the noisy latent vector sequence at the current time step. The specific structure and detailed description of the video generation model can be found in the relevant descriptions of Figures 4, 5, and 6 in the previous method embodiments and are not further elaborated here. In the disclosed embodiments, a loss function can be constructed based on the aforementioned training objectives. The loss function value can be used in each iteration of training the video generation model to update model parameters using methods such as gradient descent until a preset training termination condition is met. The training termination condition can include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. In addition, in addition to the loss function shown in the above formula (1), other loss functions used within the spirit of this principle are also within the scope of protection of this disclosure. For example, in each round of iteration, the difference between the noise predicted at multiple time steps and the Gaussian noise is selected to obtain the loss function. As one of the achievable methods, during the training of the video generation model, the global encoder and decoder can use the parameters obtained by pre-training. In each round of training iteration, the parameters of the spatiotemporal conditional encoder and the diffusion model are updated using the loss function corresponding to the training target. The pre-training of the global encoder can use other tasks, such as image generation tasks, text classification tasks, etc. The pre-training of the decoder can also use other tasks, such as image generation tasks, image classification tasks, etc. These are currently available training tasks and will not be described in detail here. Before training the video generation model, the parameters of the diffusion model in the image generation model can be used for initialization. This method alleviates the training difficulty to a certain extent and speeds up the training speed, but it still has difficulties in learning to process temporal features and generate videos under multiple conditions. In view of this, the present disclosure provides a more preferred embodiment, namely, a two-stage training strategy. Before training the video generation model using the first training data, the diffusion model is first pre-trained. This pre-training utilizes a process of generating videos based on text conditions to enable the diffusion model to learn. Then, based on the parameters of the diffusion model obtained through pre-training, the video generation model is further trained using the first training data (i.e., including multiple conditions such as global and local conditions). The pre-training of the diffusion model may include the following steps: first, obtaining second training data comprising multiple second training samples. The second training samples include: extracting descriptive text from video samples as text samples. The descriptive text of the video samples can be obtained using an existing video description text generation model, from the webpage where the video samples originated, or manually added. The text samples are then encoded using a global encoder to obtain text feature representations. The resulting text feature representation and noise sequence are then input into the diffusion model to train the diffusion model. The diffusion model uses the text feature representation to denoise the noise sequence. The training objective is to minimize the difference between the noise predicted by the diffusion model and the Gaussian noise at each time step during denoising. The loss function used in this pre-training is the same as that in formula (1), except that the condition C involved in the pre-training process includes the text condition. In other words, when the input condition includes the text condition, the video generation process is used to train the diffusion model.The parameters of the diffusion model obtained through training are then used as the initialization parameters for the diffusion model when training the video generation model in step 804. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous. According to another embodiment, a video generation device is provided. Figure 9 shows a schematic block diagram of a video generation device according to one embodiment, which corresponds to the model training device in the system shown in Figure 1. As shown in Figure 9, the apparatus 900 includes a condition acquisition unit 901 and a video generation unit 902. The main functions of each component unit are as follows: the condition acquisition unit 901 is configured to acquire local conditions, which include spatial conditions and / or temporal conditions; the video generation unit 902 is configured to encode the local conditions to obtain local condition feature representations; integrate the noise sequence with the local condition feature representations to obtain a noise latent vector sequence; denoise the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; and decode the denoised latent vector sequence to obtain a video. Furthermore, the condition acquisition unit 901 can also be configured to acquire global conditions, which include at least one of text conditions, style conditions, and color conditions. Accordingly, the video generation unit 902 can also be configured to encode the global conditions to obtain a global condition feature representation. The diffusion model can use the global condition feature representation to perform cross-attention processing on the noise latent vector sequence to predict noise, and then use the predicted noise to perform denoising. The spatial condition can include at least one of a single image and a single semantic sketch. The temporal condition can include at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence, and a grayscale map sequence. The video generation unit 902 can be implemented using the video generation model shown in FIG3 . The global encoder encodes the global condition to obtain a global condition feature representation. The spatiotemporal condition encoder encodes the local condition to obtain a local condition feature representation. The diffusion model denoises the noisy latent vector sequence to obtain a denoised latent vector sequence. The decoder uses the denoised latent vector sequence for decoding to obtain a video.As one possible implementation, a global encoder in a video generation model performs text encoding on the text condition to obtain a text feature representation, and performs image encoding on the style condition and / or color condition to obtain an image feature representation. The text feature representation and the image feature representation are then integrated to obtain a global condition feature representation. As another possible implementation, each spatiotemporal condition encoder encodes each local condition separately to obtain a feature tensor corresponding to each local condition, where the spatiotemporal condition encoder corresponds one-to-one to the local condition. The feature tensors corresponding to each local condition are then fused to obtain a local condition feature representation. Specifically, the spatiotemporal condition encoder can use the spatiotemporal condition encoder to perform spatial feature encoding on the local condition to obtain a spatial feature representation of the local condition. If the local condition is a sequence, a temporal self-attention process is performed on the spatial feature representation of the local condition to obtain a feature tensor corresponding to the local condition. Otherwise, the spatial feature representation of the local condition is replicated in time to generate a temporal spatial feature representation, and a temporal self-attention process is performed on the temporal spatial feature representation to obtain a feature tensor corresponding to the local condition. According to another embodiment, a device for training a video generation model is provided. Figure 10 shows a schematic block diagram of the device for training a video generation model according to one embodiment. As shown in Figure 10, the device 1000 includes a sample acquisition unit 1001 and a model training unit 1002, and may further include a pre-training unit 1003. The main functions of each component unit are as follows: The sample acquisition unit 1001 is configured to acquire first training data comprising a plurality of first training samples, wherein the first training samples include locally conditioned samples, which include spatially conditioned samples and / or temporally conditioned samples. The model training unit 1002 is configured to train a video generation model using first training data. The video generation model includes a spatiotemporal conditional encoder, a diffusion model, and a decoder. The spatiotemporal conditional encoder encodes local conditional samples to obtain local conditional feature representations. The diffusion model denoises a noisy latent vector sequence to obtain a denoised latent vector sequence, where the noisy latent vector sequence is obtained by integrating the noise sequence with the local conditional feature representations. The decoder decodes the denoised latent vector sequence to obtain a video. The training objective includes minimizing the difference between noise predicted by the diffusion model during denoising and Gaussian noise. Furthermore, the first training samples also include global conditional samples, where the global conditional samples include at least one of text conditional samples, style conditional samples, and color conditional samples. The video generation model also includes a global encoder, which encodes the global conditional samples to obtain a global conditional feature representation.Accordingly, the diffusion model uses a global conditional feature representation to perform cross-attention on the noisy latent vector sequence to predict noise, and uses the predicted noise for denoising. As one possible implementation, the global encoder and decoder use pre-trained parameters. In each training iteration, the parameters of the spatiotemporal conditional encoder and diffusion model are updated using a loss function corresponding to the training objective. As one possible implementation, the sample acquisition unit 1001 can acquire a video sample; acquire descriptive text, a style image, and / or a color histogram of the video sample as a text conditional sample, a style conditional sample, and / or a color conditional sample, respectively; extract at least one of a single image and a single semantic sketch as a spatial conditional sample; and extract at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence, and / or a grayscale image sequence as a temporal conditional sample. As one possible implementation, the pre-training unit 1003 can be configured to pre-train the diffusion model in the following manner: obtaining second training data comprising multiple second training samples, the second training samples comprising: extracting descriptive text from video samples as text samples; encoding the text samples using a global encoder to obtain text feature representations; inputting the text feature representations and noise sequences into a diffusion model to train the diffusion model, wherein the diffusion model denoises the noise sequence using the text feature representations; the training objective includes minimizing the difference between the noise predicted by the diffusion model during denoising at each time step and Gaussian noise. Accordingly, the model training unit 1002 further trains the video generation model using the first training data based on the parameters of the diffusion model pre-trained by the pre-training unit 1003. As one possible implementation, the spatiotemporal conditional encoder can encode each local conditional sample using each spatiotemporal conditional encoder to obtain a feature tensor corresponding to each local conditional sample, wherein the spatiotemporal conditional encoder corresponds one-to-one to the local condition; and fusing the feature tensors corresponding to each local conditional sample to obtain a local conditional feature representation. As one possible implementation, the spatiotemporal conditional encoder can perform spatial feature encoding on the local conditional sample to obtain a spatial feature representation of the local conditional sample. If the local conditional sample is a sequence, temporal self-attention processing is performed on the spatial feature representation of the local conditional sample to obtain a feature tensor corresponding to the local conditional sample. Otherwise, the spatial feature representation of the local conditional sample is replicated in time sequence to generate a temporal spatial feature representation, and temporal self-attention processing is performed on the temporal spatial feature representation to obtain a feature tensor corresponding to the local conditional sample.The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments may be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant details, refer to the description of the method embodiments. The device embodiments described above are illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one location or distributed across multiple network elements. Some or all of these modules may be selected based on actual needs to achieve the objectives of the present embodiments. Persons of ordinary skill in the art will be able to understand and implement the present embodiments without inventive effort. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of these data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. In addition, embodiments of the present disclosure further provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the aforementioned method embodiments. Furthermore, an electronic device includes: one or more processors; and a memory associated with the one or more processors, the memory configured to store program instructions that, when read and executed by the one or more processors, execute the steps of any of the aforementioned method embodiments. The present disclosure further provides a computer program product including a computer program that, when executed by a processor, implements the steps of any of the aforementioned method embodiments. FIG. 11 exemplarily illustrates the architecture of an electronic device, which may include a processor 1110, a video display adapter 1111, a disk drive 1112, an input / output interface 1113, a network interface 1114, and a memory 1120. The processor 1110, the video display adapter 1111, the disk drive 1112, the input / output interface 1113, the network interface 1114, and the memory 1120 may be communicatively connected via a communication bus 1130.The processor 1110 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the present disclosure. The memory 1120 can be implemented using ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, and the like. The memory 1120 can store an operating system 1121 configured to control the operation of the electronic device 1100 and a basic input / output system (BIOS) 1122 configured to control low-level operations of the electronic device 1100. Furthermore, the memory 1120 can store a web browser 1123, a data storage management system 1124, and a video generation device / model training device 1125, among others. The video generation device / model training device 1125 can be an application program that specifically implements the aforementioned steps in the embodiments of the present disclosure. In summary, when implementing the technical solutions provided by the present disclosure through software or firmware, the relevant program code is stored in the memory 1120 and is called and executed by the processor 1110. The input / output interface 1113 is configured to connect to an input / output module to implement information input and output. The input / output module can be configured as a component within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc. The network interface 1114 is configured to connect to a communication module (not shown in the figure) to implement communication interaction between the device and other devices. The communication module can achieve communication through a wired method (e.g., USB, network cable, etc.) or a wireless method (e.g., mobile network, Wi-Fi, Bluetooth, etc.). The bus 1130 comprises a path for transmitting information between various components of the device (e.g., the processor 1110, the video display adapter 1111, the disk drive 1112, the input / output interface 1113, the network interface 1114, and the memory 1120). It should be noted that although the above device shows the processor 1110, the video display adapter 1111, the disk drive 1112, the input / output interface 1113, the network interface 1114, and the memory 112,, bus 1130, etc. However, during specific implementation, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will appreciate that the aforementioned device may also include the components necessary to implement the disclosed solution, rather than all of the components shown in the figures. From the above description of the embodiments, it is clear that those skilled in the art will clearly understand that the disclosed solution can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the disclosed solution, or the portion that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments or portions thereof. The technical solution provided by the disclosed solution has been described in detail above. Specific examples have been used herein to illustrate the principles and implementations of the disclosed solution. The description of the above embodiments is intended only to facilitate understanding of the disclosed solution and its core concepts. Furthermore, those skilled in the art will appreciate that, based on the concepts of the disclosed solution, variations in the specific implementation and scope of application are possible. In summary, the contents of this specification should not be construed as limiting the present disclosure. Industrial Applicability: Embodiments of the present disclosure provide a video generation method, comprising: obtaining local conditions, including spatial and / or temporal conditions; encoding the local conditions to obtain a local condition feature representation; integrating a noise sequence with the local condition feature representation to obtain a noise latent vector sequence; denoising the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; and decoding the denoised latent vector sequence to obtain a video. The present disclosure incorporates spatial and / or temporal conditions as local conditions, integrating the local condition feature representation with the noise sequence to obtain a noise latent vector sequence; denoising the noise latent vector sequence to obtain a denoised latent vector sequence; and decoding the denoised latent vector sequence to obtain a video. This approach eliminates the limitation of video generation to textual conditions and instead introduces spatial and / or temporal conditions to guide video generation, thereby enabling more flexible and diverse video generation and improving video quality.

Claims

Claims 1. A video generation method, the method comprising: Acquiring a local condition, wherein the local condition includes a spatial condition and / or a temporal condition; encoding the local condition to obtain a local condition feature representation; The noise sequence is integrated with the local conditional feature representation to obtain a noise latent vector sequence; the noise latent vector sequence is denoised using a diffusion model to obtain a denoised latent vector sequence; and the denoised latent vector sequence is decoded to obtain a video.

2. The method according to claim 1, wherein: The method also includes: obtaining a global condition, wherein the global condition includes at least one of a text condition, a style condition, and a color condition; encoding the global condition to obtain a global condition feature representation; and denoising the noisy latent vector sequence using a diffusion model, including: performing cross-attention processing on the noisy latent vector sequence using the diffusion model to predict noise, and performing denoising using the predicted noise.

3. The method according to claim 1, wherein: The spatial condition includes at least one of a single image and a single semantic sketch; the temporal condition includes at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence and a grayscale map sequence.

4. The method according to claim 2, wherein: Encoding the global condition to obtain a global condition feature representation includes: performing text encoding on the text condition to obtain a text feature representation, and performing image encoding on the style condition and / or color condition to obtain an image feature representation; integrating the text feature representation and the image feature representation to obtain a global condition feature representation.

5. The method according to any one of claims 1 to 4, wherein: The encoding process of the local conditions to obtain the local condition feature representation includes: using each spatiotemporal condition encoder to encode each local condition respectively to obtain a feature tensor corresponding to each local condition, wherein the spatiotemporal condition encoder corresponds to the local condition one-to-one; and fusing the feature tensors corresponding to each local condition to obtain the local condition feature representation.

6. The method according to claim 5, wherein: The method of encoding each local condition respectively by using the spatiotemporal condition encoder includes: using the spatiotemporal condition encoder to perform spatial feature encoding on the local condition to obtain a spatial feature representation of the local condition; if the local condition is a sequence, performing a time-domain self-attention process on the spatial feature representation of the local condition to obtain a feature tensor corresponding to the local condition; otherwise, copying the spatial feature representation of the local condition in time sequence to generate a spatial feature representation in time sequence, performing a time-domain self-attention process on the spatial feature representation in time sequence, and obtaining a feature tensor corresponding to the local condition.

7. A method for training a video generation model, the method comprising: Acquire first training data including a plurality of first training samples, wherein the first training samples include local condition samples, and the local condition samples include spatial condition samples and / or temporal condition samples; The video generation model is trained using the first training data, and the video generation model includes: a spatiotemporal conditional encoder, a diffusion model and a decoder; wherein the spatiotemporal conditional encoder encodes the local conditional samples to obtain a local conditional feature representation; the diffusion model denoises the noise latent vector sequence to obtain a denoised latent vector sequence, and the noise latent vector sequence is obtained by integrating the noise sequence with the local conditional feature representation; the decoder uses the denoised latent vector sequence to perform decoding processing to obtain a video; the training objectives include: minimizing the difference between the noise predicted by the diffusion model when performing denoising and the Gaussian noise.

8. The method according to claim 7, wherein: The first training sample further includes a global condition sample, and the global condition sample includes at least one of a text condition sample, a style condition sample, and a color condition sample; the video generation model further includes a global encoder, and the global encoder encodes the global condition sample to obtain a global condition feature representation; The diffusion model performs denoising on the noisy latent vector sequence, including: the diffusion model performs cross-attention processing on the noisy latent vector sequence using the global conditional feature representation to predict noise, and performs denoising using the predicted noise.

9. The method according to claim 8, wherein: The global encoder and the decoder use parameters obtained through pre-training, and in each round of iteration of the training, the parameters of the spatiotemporal conditional encoder and the diffusion model are updated using a loss function corresponding to the training objective.

10. The method according to claim 8, wherein: The step of acquiring first training data including a plurality of first training samples comprises: acquiring a video sample; acquiring a description text, a style image and / or a color histogram of the video sample as the text condition sample, the style condition sample and / or the color condition sample respectively; extracting at least one of a single image and a single semantic sketch as the spatial condition sample; and extracting at least one of a motion vector sequence, a depth map sequence, a mask map sequence, a semantic sketch sequence and / or a grayscale map sequence as the temporal condition sample.

11. The method according to any one of claims 7 to 10, wherein: Before using the first training data to train the video generation model, the method further includes pre-training the diffusion model; further using the first training data to train the video generation model based on the parameters of the diffusion model obtained by pre-training; wherein the pre-training of the diffusion model includes: obtaining second training data including a plurality of second training samples, wherein the second training samples include: extracting description text from the video sample as a text sample; encoding the text sample using a global encoder to obtain a text feature representation; The text feature representation and the noise sequence are input into a diffusion model to train the diffusion model, and the diffusion model performs denoising on the noise sequence using the text feature representation; the training objectives include: minimizing the difference between the noise predicted by the diffusion model when performing denoising at each time step and the Gaussian noise.

12. The method according to any one of claims 7 to 10, wherein: The spatiotemporal condition encoder encodes the local condition samples to obtain the local condition feature representation, including: using each spatiotemporal condition encoder to encode each local condition sample respectively to obtain a feature tensor corresponding to each local condition sample, wherein the spatiotemporal condition encoder corresponds to the local condition one-to-one; and fusing the feature tensors corresponding to each local condition sample to obtain the local condition feature representation.

13. The method according to claim 12, wherein: The method of encoding each local conditional sample using each spatiotemporal conditional encoder includes: using the spatiotemporal conditional encoder to perform spatial feature encoding on the local conditional sample to obtain a spatial feature representation of the local conditional sample; if the local conditional sample is a sequence, performing a temporal self-attention process on the spatial feature representation of the local conditional sample to obtain a feature tensor corresponding to the local conditional sample; otherwise, copying the spatial feature representation of the local conditional sample in time sequence to generate a spatial feature representation in time sequence, performing a temporal self-attention process on the spatial feature representation in time sequence, and obtaining a feature tensor corresponding to the local conditional sample.

14. A video generation method, executed by a cloud server, comprising: Acquiring a local condition from a user terminal, the local condition including a spatial condition and / or a temporal condition; encoding the local condition to obtain a local condition feature representation; Integrate the noise sequence with the local conditional feature representation to obtain a noise latent vector sequence; perform denoising on the noise latent vector sequence using a diffusion model to obtain a denoised latent vector sequence; perform decoding using the denoised latent vector sequence to obtain a video; The video is sent to the user terminal for display.

15. A video generating device, comprising: A condition acquisition unit, configured to acquire a local condition, where the local condition includes a spatial condition and / or a temporal condition; A video generating unit, configured to encode the local condition to obtain a local condition feature representation; The noise sequence is integrated with the local conditional feature representation to obtain a noise latent vector sequence; the noise latent vector sequence is denoised using a diffusion model to obtain a denoised latent vector sequence; and the denoised latent vector sequence is decoded to obtain a video.

16. A device for training a video generation model, the device comprising: a sample acquisition unit configured to acquire first training data including a plurality of first training samples, wherein the first training samples include local condition samples, and the local condition samples include spatial condition samples and / or temporal condition samples; a model training unit configured to train a video generation model using the first training data, wherein the video generation model includes: a spatiotemporal condition encoder, a diffusion model, and a decoder; wherein the spatiotemporal condition encoder trains the local condition samples; The method comprises the steps of: encoding the local conditional samples to obtain the local conditional feature representation; performing denoising on the noise latent vector sequence by a diffusion model to obtain the denoised latent vector sequence, wherein the noise latent vector sequence is obtained by integrating the noise sequence with the local conditional feature representation; and performing decoding processing on the decoder using the denoised latent vector sequence to obtain the video. The training objectives include: minimizing the difference between the noise predicted by the diffusion model during the denoising process and the Gaussian noise.

17. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

18. An electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Image generation method and device and electronic equipment

    CN115965791A

  • Diffusion models having improved accuracy and reduced consumption of computational resources

    WO2022265992A1

Cited By

  • Method and device for generating video by using image, equipment and medium

    CN119364135A

  • Cross-view gait recognition method and system based on diffusion model under missing view

    CN119693971A

  • Video generation method and device and electronic equipment

    CN119741222A

  • Video generation method, apparatus and electronic device

    CN119741222B

  • Text-guided zero-sample transparent layer and layered image generation method

    CN120070638A