Video generation method and device based on latent consistency model

Abstract

The application provides a video generation method and device based on a latent consistency model, comprising: obtaining information and a text description of a to-be-processed video; performing spatiotemporal condition processing on a first frame of the to-be-processed video, adopting a consistency distillation strategy in image generation, combining the spatiotemporal condition processing, optimizing an existing latent diffusion model, and performing latent consistency model training on the processed information and the text description; wherein the latent consistency model training performs a forward diffusion operation to generate a noisy video; the generated noisy video is input into a student model and a teacher model to predict a de-noised video, and finally a video is generated. The application can effectively reduce the need for a large number of sampling steps in the video generation process, and maintain the consistency of the video in space and motion, thereby ensuring the efficient synthesis of high-quality videos.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a video generation method and device based on a latent consistency model. BACKGROUND

[0002] The rapid development of contemporary science and technology has promoted the vigorous development of the field of video generation, which has shown great potential and wide application in many fields. Video generation refers to the use of computer algorithms and technologies to generate virtual video content, which can be to create completely new videos from scratch or to edit and enhance existing videos. The development of this technology benefits from the progress in the fields of deep learning, computer vision and image processing, providing new solutions for video production, virtual reality, game development, etc.

[0003] In the field of image generation, the most common generation models include Generative Adversarial Networks (GAN) and Variational Autoencoders (VAE), each with unique characteristics and advantages. However, in the past few years, a new generation model, Denoising Diffusion Probabilistic Model (DDPM), also known as diffusion model, has begun to attract attention and gradually become one of the research hotspots. The diffusion model was proposed in response to the limitations and challenges of existing generation models. Although GAN and VAE have made significant achievements in image generation, they have certain defects in some cases, such as blurred, distorted or discontinuous images. Diffusion model addresses these issues by introducing a denoising diffusion process, thereby improving the quality and realism of generated images. In diffusion, the process of generating images is modeled as a gradual denoising process, with the model gradually improving the quality of generated images through multiple iterations of inference, making the generated images clearer, more realistic and detailed. Through these models, computers can automatically generate realistic human images, scene backgrounds and dynamic effects, thereby realizing a highly automated video generation process. However, these models require multiple iterations of inference when generating videos, making the generation process both time-consuming and computationally intensive. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a video generation method and device based on a latent consistency model.

[0005] In a first aspect, the present application provides a video generation method based on a latent consistency model, comprising:

[0006] obtaining information and text description of the video to be processed;

[0007] The first frame of the to-be-processed video is subjected to spatio-temporal condition processing, a consistency distillation strategy in image generation is adopted, and the existing latent diffusion model is optimized in combination with the spatio-temporal condition processing; the processed information and the text description are subjected to latent consistency model training; wherein the latent consistency model training performs a forward diffusion operation to generate a noisy video; the generated noisy video is input into a student model and a teacher model to predict a denoised video, and finally a video is generated.

[0008] Further, further comprising:

[0009] A time convolution block is constructed to capture spatio-temporal information in a video sequence to make it compatible with a video generation task;

[0010] Based on the constructed time convolution block, an absolute position embedding is realized by using a rotation matrix technique to represent position information in the time dimension;

[0011] The key vector and the value vector in the self-attention layer in the model are modified for spatial feature adjustment;

[0012] A consistency distillation strategy is adopted for optimization, and finally a high-quality video can be quickly generated.

[0013] In a second aspect, the present application provides a video generation device based on a latent consistency model, comprising:

[0014] An acquisition module is configured to acquire information of a to-be-processed video and a text description;

[0015] A video generation module is configured to input the spatio-temporal processed video as an input, combine the text description, adopt a consistency distillation strategy in image generation, optimize the existing latent diffusion model in combination with the spatio-temporal condition processing, and finally quickly generate a high-quality video. Wherein the consistency model training performs a forward diffusion operation to generate a noisy video; the generated noisy video is input into a student model and a teacher model to predict a denoised video, and finally a video is generated.

[0016] Further, further comprising a spatio-temporal processing module:

[0017] The spatio-temporal processing module is configured to construct a time convolution block to capture spatio-temporal information in a video sequence to make it compatible with a video generation task;

[0018] Based on the constructed time convolution block, an absolute position embedding is realized by using a rotation matrix technique to represent position information in the time dimension;

[0019] The key vector and the value vector in the self-attention layer in the model are modified for spatial feature adjustment;

[0020] Adopting a consistent distillation strategy optimization, a high-quality video can be quickly generated.

[0021] Advantages:

[0022] The application optimizes the existing latent diffusion model by performing spatio-temporal condition processing on the first frame of the video to be processed and adopting a consistent distillation strategy in image generation, and combining the spatio-temporal condition processing to perform latent consistency model training on the processed information; wherein the latent consistency model training performs a forward diffusion operation to generate a noise video; the generated noise video is input into a student model and a teacher model to predict a denoised video, and finally generate a video. The spatio-temporal condition processing can effectively reduce the need for a large number of sampling steps in the video generation process and maintain the consistency of the video in space and motion, thereby ensuring the efficient synthesis of high-quality videos. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 The flowchart of the video generation method based on the latent consistency model provided by an embodiment of the present application;

[0025] Figure 2 The network structure diagram of the video generation method based on the latent consistency model provided by an embodiment of the present application;

[0026] Figure 3 The structure diagram of the video generation device based on the latent consistency model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. The video generation method based on the latent consistency model provided by the present application will be explained and described in detail through specific embodiments.

[0028] As Figure 1As shown, an embodiment of the present application provides a video generation method based on a latent consistency model, including the following steps:

[0029] Step 101: obtaining information and text description of a to-be-processed video;

[0030] Step 102: performing spatio-temporal condition processing on the first frame of the to-be-processed video;

[0031] Step 103: using a consistency distillation strategy in image generation, combining the spatio-temporal condition processing, optimizing an existing latent diffusion model, training a latent consistency model video for the processed information and text description, and training the video latent consistency model; wherein the latent consistency model training performs a forward diffusion operation to generate a noisy video; the generated noisy video is input into a student model and a teacher model to predict a denoised video, and finally generate a video.

[0032] In this embodiment, for the step 102, it is necessary to note that the first frame of the to-be-processed video is taken as input, represented as latent features, and the latent features are taken as a condition signal input into a model; at the same time, a time convolution block is constructed to capture spatio-temporal information in a video sequence, making it compatible with a video generation task; based on the constructed time convolution block, a rotation matrix technique is used to realize absolute position embedding to represent position information in the time dimension.

[0033] In this embodiment, for the video generation stage step 103, it is necessary to note that the powerful knowledge in a large-scale pre-trained video diffusion model is fully utilized, a consistency distillation strategy is used to optimize the model, a forward diffusion operation is performed on the processed video to add noise to the video to generate a noisy video, which is input into a student model and a teacher model to predict a denoised video.

[0034] Wherein, the forward diffusion operation simulates the denoising process of video data; the noisy video is shared by the student model and the teacher model with the same network structure, but at the beginning of training, the student model uses the parameters of the teacher model for initialization; the teacher model is responsible for estimating the denoised frame; the entire consistency distillation process is carried out in the latent space, and all operations are carried out at the level of the latent representation of the video.

[0035] From the above technical solutions, the video generation method based on a latent consistency model provided by the embodiment of the present application can obtain information and a text description of a to-be-processed video; perform spatio-temporal condition processing on a first frame of the to-be-processed video, adopt a consistency distillation strategy in image generation, combine the spatio-temporal condition processing, optimize an existing latent diffusion model, and perform latent consistency model video training on the processed information and the text description; wherein the latent consistency model training performs a forward diffusion operation to generate a noise video; the generated noise video is input into a student model and a teacher model to predict a denoised video, and finally a video is generated. The present application is based on the existing latent diffusion model in video generation, and trains a video latent consistency model using the consistency distillation idea, effectively synthesizes a video in the least steps, and maintains high quality at the same time.

[0036] In the embodiment, it should be noted that the constructed time convolution block is one-dimensional and has the same structure as the spatial convolution block in the model, can capture the spatio-temporal information in the video sequence, is compatible with the video generation task, simplifies the design and understanding of the model, and improves the scalability and generalization ability of the model. The latent diffusion model is a large-scale pre-training diffusion model, which includes a series of spatial upsampling and downsampling modules with skip links, and each upsampling / downsampling block is composed of a spatial convolution and a spatial attention layer.

[0037] In the embodiment, it should be noted that the absolute position embedding is realized by using a rotation matrix technology, which has the effect of being independent of a fixed sequence length, so that the model can process input sequences of different lengths and can be combined with a linear self-attention structure, which helps to improve the calculation efficiency and the performance of the model.

[0038] In the embodiment, it should be noted that the consistency distillation is based on the existing latent diffusion model in video generation, and trains a video latent consistency model using the consistency distillation idea, reduces the calculation amount and quickly generates a high-quality video.

[0039] As shown in Figure 2 The video processing stage of the embodiment of the present application uses a large-scale pre-training diffusion model to perform spatio-temporal processing on the obtained video, performs latent consistency distillation training on the processed video, and generates a high-quality video. The spatio-temporal processing step 102 is specifically divided into the following steps:

[0040] Step S1: using an encoder in a large-scale pre-training diffusion model to encode the first frame of the input video as a latent feature, and inputting the latent feature as a conditional signal to a latent diffusion model;

[0041] Step S2: modifying the potential diffusion model, constructing a time convolution block, designing its structure to be the same structure as the spatial convolution block in the model, inserting the constructed one-dimensional time convolution block after each spatial convolution block and the time attention block at a specific attention resolution, to make it compatible with the video generation task;

[0042] Step S3: based on the constructed time convolution block, using rotation matrix technology to realize absolute position embedding to represent the position information of the time dimension.

[0043] Step S4: further, adjust the spatial features, given the middle hidden state of the first frame , the self-attention operation is expressed as , which represents the attention between different spatial positions:

[0044] ;

[0045] ;

[0046] where, q represents the query vector, k represents the key vector, v represents the value vector, , , Wq, Wk and Wv are learnable projection matrices used to create query, key and value vectors from input, and the superscript T represents the transpose of the matrix. Dq and Dk are the dimensions of the query vector and the key vector. The softmax function is a mathematical function commonly used in machine learning, especially in neural networks and natural language processing. It converts the values of a vector into a probability distribution. Each value in the input vector is converted into a probability, and the sum of these probabilities is 1.

[0047] In order to achieve better visual coherence in the video, the key vector and value vector in the self-attention layer are modified to include the features of the first frame:

[0048] Q s = W s Q n i , K s ' = W s k [n i , n 1 ], V s ' = W s v [n i , n 1 ] ;

[0049] wherein, denotes the middle hidden state of the first frame, [·] denotes a concatenation operation such that the modified key vector and the value vector have a sequence length that is doubled compared to the original key vector and the value vector . In this way, every spatial position in all frames has access to the full information of the first frame, allowing for spatial feature conditioning in the spatial attention layer.

[0050] In the step 103 of the video generation of the embodiment of the present application, the powerful knowledge in the large-scale pre-trained video diffusion model is fully utilized, the above-mentioned spatio-temporal processed video is taken as a model input, and a consistency distillation strategy is used to optimize the model in combination with the text description. Specifically, a forward diffusion operation is performed on the processed video, noise is added to the video, thereby generating a noisy video, and the generated noisy video is input into a student model and a teacher model to predict a de-noised video. The spatio-temporal processing stage is specifically divided into the following steps:

[0051] Step S1: performing a forward diffusion operation on the processed video, adding noise to the video, thereby generating a noisy video. A solver is used to estimate the de-noising process, which can be represented as:

[0052] ;

[0053] wherein, denotes a conditional input, which can be a text prompt in text-to-video generation or multiple combined signals in a synthetic video task. denotes a noise input at time step , denotes a noise estimation value at time step . By adding noise at each time step, the de-noising process of the video is simulated, and the de-noised frame is predicted.

[0054] Step S2: optimizing the latent diffusion model using a consistency distillation strategy, the noisy video is shared by the student model and the teacher model using the same network structure, but at the beginning of training, the student model is initialized using the parameters of the teacher model; the teacher model is responsible for estimating the de-noised frame.

[0055] Since the classifier-free guidance is crucial for synthesizing high-quality content, the present application utilizes the classifier-free guidance in the consistency distillation stage, and uses a factor​ To control the scale of guidance:

[0056] ;

[0057] where the factor is variable and can be fed into the network for modulation, but this changes the structure of the original network because the module for encoding needs to be added. Refers to the parameters of the solver, used to optimize the solving process.

[0058] Specifically, the present application uses an existing pre-trained video diffusion model as a teacher model, and fixes the network parameters in the consistency distillation process; the student model shares the same network structure as the teacher model. The noise video generated by the teacher model is the target, and the student model predicts the denoised video. The student model not only learns the denoising skills, but also benefits from the knowledge of the teacher model. The consistency distillation strategy proposed helps to improve the generalization ability of the model. The entire consistency distillation process is carried out in the latent space, and all operations are carried out at the level of the latent representation of the video. Improve the effect of video denoising, so that the generated video is clearer and more realistic.

[0059] As shown in Figure 3 An embodiment of the video generation device based on the latent consistency model provided by the present application comprises an acquisition module 301, a space-time processing module 302, and a video generation module 303.

[0060] The acquisition module 301 is used to acquire video information to be processed and text description.

[0061] The space-time processing module 302 is used to construct a time convolution block to capture the space-time information in the video sequence, so that it is compatible with the video generation task; based on the constructed time convolution block, the absolute position embedding is realized by using the rotation matrix technology to represent the position information of the time dimension; the key vector and the value vector in the self-attention layer in the model are modified to adjust the spatial features.

[0062] The video generation module 303 is used to input the space-time processed video, optimize it by using the consistency distillation strategy, and finally quickly generate high-quality video.

[0063] The video generation device based on the latent consistency model provided by the embodiment of the present application can be specifically used to execute the video generation method based on the latent consistency model of the above-mentioned embodiment, and the technical principles and advantages are similar. For details, please refer to the above-mentioned embodiment, which will not be repeated here.

[0064] The embodiment of the present application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the video generation method based on the latent consistency model when executing the program.

[0065] The embodiment of the present application further provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the video generation method based on the latent consistency model.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software. Based on such an understanding, the above technical solutions essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method of each embodiment or some parts of the embodiment.

[0067] In addition, in the present application, terms such as "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0068] In addition, in the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0069] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video generation method based on a latent consistency model, characterized in that, include: Obtain information about the video to be processed; The first frame of the video to be processed is subjected to spatiotemporal condition processing. A consistency distillation strategy in image generation is adopted. Combined with the spatiotemporal condition processing, the latent diffusion model is optimized. The processed information and text description are trained to form a latent consistency model. The latent consistency model training performs a forward diffusion operation to generate a noisy video. The generated noisy video is input into the student model and the teacher model to predict the denoised video and finally generate the video. That is: use an existing pre-trained video diffusion model as the teacher model and fix the network parameters during the consistency distillation process; the student model shares the same network structure with the teacher model, and the noisy video generated by the teacher model is used as the target, so that the student model can predict the denoised video.

2. The video generation method based on a potential consistency model according to claim 1, characterized in that, Also includes: A spatiotemporal feature adjustment method is constructed to capture spatiotemporal information in video sequences, making it compatible with video generation tasks; Based on the constructed spatiotemporal feature adjustment method, the rotation matrix technique is used to realize absolute position embedding to represent the position information in the time dimension; Modify the key vectors and value vectors in the self-attention layer of the model to adjust spatial features; By employing a consistent distillation strategy for optimization, high-quality videos can be generated quickly.

3. The video generation method based on a potential consistency model according to claim 2, characterized in that, The spatiotemporal feature adjustment method further includes: Construct temporal convolutional blocks, perform convolution operations in the time dimension, and extract local features from the time series; A one-dimensional temporal convolutional block is inserted after each spatial convolutional block and a temporal attention block with a specific attention resolution; wherein the attention block and spatial convolutional block have the exact same structure as the temporal convolutional block, except that the temporal convolution and attention operations are performed along the temporal dimension.

4. The video generation method based on a potential consistency model according to claim 2, characterized in that, The optimization of the uniform distillation strategy also includes: By leveraging the concept of consistency models in image generation, videos can be synthesized efficiently with minimal steps. The existing pre-trained video diffusion model was used as the teacher model, and the network parameters were fixed during the consistency distillation process. The student model and the teacher model share the same network structure; The teacher model is responsible for estimating the denoised frames; The entire uniform distillation process takes place in the latent space, and all operations are performed at the latent representation level of the video.

5. A video generation device based on a latent consistency model, characterized in that, include: The acquisition module is used to acquire video information and text descriptions to be processed; The spatiotemporal processing module is used to construct temporal convolutional blocks to capture spatiotemporal information in video sequences, making them compatible with video generation tasks. Based on the constructed temporal convolutional blocks, rotation matrix technology is used to achieve absolute position embedding to represent the positional information in the temporal dimension. The key vectors and value vectors in the self-attention layer of the model are modified to adjust spatial features. The video generation module takes the spatiotemporally processed video as input, optimizes it using a consistent distillation strategy, and can quickly generate a high-quality video. Specifically, it uses an existing pre-trained video diffusion model as the teacher model and fixes the network parameters during the consistent distillation process. The student model shares the same network structure with the teacher model. The noisy video generated by the teacher model is used as the target, and the student model predicts the denoised video.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a video generation method based on a potential consistency model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of a video generation method based on a potential consistency model as described in any one of claims 1 to 4.

Images