Video generation method and device based on diffusion model, equipment and medium

Abstract

The embodiment of the invention provides a video generation method and device based on a diffusion model, equipment and a medium, and relates to the technical field of video generation, and the method comprises the steps: firstly, splitting a training video containing a face into a plurality of video frames according to the frame number, selecting a first video frame containing the face as a condition picture, and carrying out the segmentation of the condition picture; inputting the plurality of video frames into a variational auto-encoder, adding Gaussian noise to obtain noisy feature representation, encoding a condition picture by using a CLIP picture encoder to obtain picture embedding representation, and inputting the noisy feature representation and the picture embedding representation into a de-noising U-Net for splicing to obtain a first video, and inputting the first video into the diffusion model to obtain the second video, and inputting the picture into the diffusion model to realize the purpose of generating a high-quality and long-period video from a single picture, and improve the naturalness of face generation and the fluency of video actions.

Description

Technical Field

[0001] The present application relates to the technical field of video generation, and in particular to a method, device, equipment and medium for video generation based on a diffusion model. Background Art

[0002] Existing video generation technology is based on machine learning and artificial intelligence, especially the progress of deep learning, which uses deep neural networks to generate visual content such as faces and human body movements. This involves complex models such as generative adversarial networks (GANs) and variational autoencoders (VAEs), which can simulate the data distribution of the real world to a certain extent and create new sequences of face images or full-body action videos. Combine multi-view images or depth sensor data to build a three-dimensional model of the scene or character to generate a video. This technology helps to improve the stereoscopic sense and realism of the synthesized video.

[0003] Despite significant progress, video generation technology still faces a series of challenges and limitations. For example, although the algorithm can generate dynamic facial images based on existing images or videos, it is a difficult task to maintain the consistency of facial features (such as expression, skin tone, and lighting conditions) between consecutive frames. This may cause the face in the video to flicker and texture incoherence, affecting the viewing experience. There is also a lack of naturalness in the movements. It is often difficult to accurately capture and replicate the subtle details and smoothness of human movements, especially complex body movements and micro-expressions. Therefore, the generated character movements may appear stiff, mechanical or unnatural, lacking the soft transitions and natural rhythm of real human movements. And when dealing with human body movements and changing perspectives, due to the inaccuracy of 3D reconstruction and posture estimation, the character's body parts are often distorted or out of proportion, such as abnormal stretching of hands and feet, dislocation of joints, etc., which seriously damages the realism of the video.

[0004] Therefore, how to study a video generation method to solve the above technical problems becomes a challenge that technicians in this field have to face. Summary of the invention

[0005] The present application provides a method, apparatus, device and medium for video generation based on a diffusion model. By inputting an image into the diffusion model, the goal of generating high-quality, long-duration video from a single image is achieved, and the naturalness of face generation and the smoothness of video motion are improved.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a video generation method based on a diffusion model, the method is applied to the diffusion model, the diffusion model includes a variational autoencoder, a CLIP picture encoder and a denoising U-Net, the method includes:

[0008] Split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image;

[0009] Input multiple video frames into the variational autoencoder and add Gaussian noise to obtain noisy feature representation;

[0010] Use CLIP image encoder to encode the conditional image to obtain image embedding representation;

[0011] The noisy feature representation and the image embedding representation are input into the denoising U-Net for concatenation to obtain the first video;

[0012] The first video is input into a diffusion model to obtain a second video, wherein the second video has a higher video quality than the first video.

[0013] In a possible embodiment, the step of inputting a plurality of video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation includes:

[0014] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0015] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0016] In a possible embodiment, the step of inputting the first video into a diffusion model to obtain the second video includes:

[0017] Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation;

[0018] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0019] Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features;

[0020] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0021] In one possible embodiment, the image embedding representation is input into the denoising U-Net in the form of cross attention.

[0022] In a second aspect, the embodiment of the present application further proposes a video generation device based on a diffusion model, the device comprising:

[0023] A splitting module is used to split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as a conditional image;

[0024] A first encoding module, used for inputting a plurality of video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation;

[0025] A second encoding module, used for encoding the conditional image using a CLIP image encoder to obtain an image embedding representation;

[0026] A splicing module, used for inputting the noisy feature representation and the image embedding representation into a denoising U-Net for splicing to obtain a first video;

[0027] The video optimization module is used to input the first video into the diffusion model to obtain a second video, wherein the second video has a higher video quality than the first video.

[0028] In a possible embodiment, the first encoding module is further used for:

[0029] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0030] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0031] In a possible embodiment, the step of inputting the first video into a diffusion model to obtain the second video includes:

[0032] Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation;

[0033] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0034] Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features;

[0035] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0036] In one possible embodiment, the image embedding representation is input into the denoising U-Net in the form of cross attention.

[0037] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the video generation method as described in any one of the first aspects.

[0038] In a fourth aspect, the present application further proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video generation method as described in any one of the first aspects.

[0039] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0040] Compared with the prior art, the embodiments of the present application propose a video generation method, device, equipment and medium based on a diffusion model, which relates to the field of video generation technology. First, a training video containing a face is split into multiple video frames according to the number of frames, and the first video frame containing a face is selected as a conditional image. The multiple video frames are input into a variational autoencoder and Gaussian noise is added to obtain a noisy feature representation. The conditional image is encoded using a CLIP image encoder to obtain an image embedding representation. The noisy feature representation and the image embedding representation are input into a denoising U-Net for splicing to obtain a first video. The first video is input into a diffusion model to obtain a second video. By inputting the image into the diffusion model, the goal of generating high-quality, long-duration videos from a single image is achieved, and the naturalness of face generation and the smoothness of video movements are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flow chart of a method for generating a video based on a diffusion model proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0046] Variational Auto Encoder (VAE): VAE learns efficient representations of data (usually called latent space representations), which are used to capture key features in images and reconstruct images or generate new, similar data from these features. Specifically for the task of generating videos from images, it helps the model understand and extract the basic structure, texture and other information of people or objects in the image.

[0047] CLIP Image Encoder: CLIP (Contrastive Language-Image Pre-training) is a pre-training model that learns cross-modal representations by comparing image and text data. Here, the role of the CLIP image encoder is to further enhance the model's ability to understand the content of the image, especially the relevance of the image to the context or concept. This helps the generated video not only to be visually coherent, but also to match the underlying semantic information to some extent, making the generated actions or scenes more logical and expected.

[0048] Denoising U-Net: U-Net is a convolutional neural network architecture commonly used for image segmentation and generation tasks, known for its unique encoder-decoder structure and jump connection. In this technical solution, its variant, Denoising U-Net, is used. Its main task is to deal with potential noise in the process of generating videos and improve the clarity and continuity of videos. By gradually adding details and reducing unnecessary interference, the conversion from pictures to videos is ensured to be smooth and realistic.

[0049] In order to solve the problem that the videos generated in the prior art are difficult to achieve consistency and integrity of human faces, and human movements are unnatural and distorted, the embodiments of the present application propose a video generation method, device, equipment and medium based on a diffusion model. By setting up a diffusion model and training it, it can predict and generate a series of continuous frames based on the input picture, thereby forming a short but coherent video clip. In this way, the elements in the static picture are given life and dynamic expression is achieved. At the same time, considering the hardware resource limitations, such as the size of the GPU video memory, the parameters of the number of generated frames are set to balance the computing requirements and the generation quality, which will be described in detail below.

[0050] Please refer to Figure 1 , Figure 1 A flow chart of a method for video generation based on a diffusion model proposed in an embodiment of the present application is shown. The method is applied to a diffusion model, and the diffusion model includes a variational autoencoder, a CLIP picture encoder, and a denoising U-Net. The method includes the following steps:

[0051] Step S1: split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image.

[0052] First, in order to ensure that the model can be exposed to diverse data samples, a video is randomly selected from the entire training set for processing in each iteration. Each frame of the selected video is checked one by one to see if there is any face. Because the project goal clearly sets the need to focus on high-quality generation of faces. If all frames in the video do not contain faces, the video will not be used as a sample for this training and will be discarded.

[0053] Once it is confirmed that at least one frame in the video contains a face, a specified number (e.g., N_frames) of video frames will be selected as training data for the current batch according to certain rules (which can be random and evenly spaced). Among the N_frames video frames, the first frame with a face needs to be determined as the conditional image. The model's task is to understand how to use the given conditional frame (i.e., the initial frame containing the face) to reconstruct or generate the remaining video sequence.

[0054] In one possible embodiment, due to GPU memory limitations, a fixed number of frames (such as 25 frames) is set as the target length of each generation during actual training. The model only needs to process a fixed number of video frames per round, reducing the computational burden. This enables the denoising U-Net to effectively recover clear information from noisy data while ensuring that the generated video content matches the provided conditional image. Reducing the difference between the output and the original noise-free latent variable prompts the model to learn to remove noise efficiently. Using the embedded representation of the conditional image to guide the generation process ensures the quality and consistency of the generated video, especially when complex objects such as faces are involved.

[0055] Step S2: input multiple video frames into a variational autoencoder and add Gaussian noise to obtain a noisy feature representation.

[0056] N_frames video frames are first encoded using a variational autoencoder to obtain a latent space feature representation with a lower resolution (downsampled 8 times). Then a Gaussian noise of random intensity is added to the latent space feature representation to obtain a noisy feature representation, which is used as the input of the denoising U-Net.

[0057] Step S2 includes:

[0058] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0059] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0060] The N_frames video frames extracted from the selected video are first fed into a pre-trained VAE model. VAE converts these high-resolution original frames into low-dimensional (downsampled 8 times) latent space representations. In order to start the denoising learning mechanism during training, Gaussian noise of random intensity is added to the obtained latent variable representation. The latent variable after adding noise is called noisy feature representation and becomes the input data of the denoising U-Net network.

[0061] Step S3: Use the CLIP picture encoder to encode the conditional picture to obtain a picture embedding representation.

[0062] The conditional image is converted into an embedding vector through the image encoder in CLIP, which captures the key visual features of the image and is added to the denoising U-Net in the form of cross attention to ensure that the generated video content is consistent with the provided conditional image.

[0063] Step S4: input the noisy feature representation and the image embedding representation into the denoising U-Net for splicing to obtain the first video.

[0064] In addition to being used to generate embedded representations, the conditional image is also encoded by VAE, and its encoding results are replicated N_frames times. These repeated encoding results are then concatenated with the above-mentioned noisy feature representations and used as the input of the denoising U-Net.

[0065] The diffusion model is designed to enable the denoising U-Net to learn how to recover clear information from noisy data. During the training process, the difference between the denoising U-Net output and the original unnoised latent variable is minimized, and the model can retain or enhance useful signals while removing noise.

[0066] Previously, a single conditional image was converted into a fixed embedding vector (Embedding) through the CLIP image encoder. After the conditional image is encoded by VAE, the encoding result is copied N_frames times and then concatenated with the noisy feature representation to recover effective information from the noise and generate continuous and high-quality video frames.

[0067] Step S5: input the first video into the diffusion model to obtain the second video.

[0068] The second video has higher video quality than the first video.

[0069] The first video is a video with a shorter duration and a lower frame rate, while the second video is a video with a longer duration and a higher resolution. Unlike the previous use of a single conditional image, step S5 uses a series of conditional images as input, which are key frames sparsely distributed throughout the target video. Each conditional image is encoded to obtain multiple embedding vectors to form a sequence.

[0070] VAE encoding is performed on each conditional image, and each encoding result is replicated a certain number of times as needed to ensure that the final total number reaches N_frames. This better preserves the visual information at different time points or perspectives. Then, these encoded latent variables are concatenated with the noisy feature representation as the input of the denoising U-Net.

[0071] The goal is to restore a complete and coherent video sequence from sparsely given key frames. The interpolation capability is emphasized, and newly generated transition frames are inserted between some known frames to create smooth and natural dynamic images.

[0072] Step S5 includes:

[0073] Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation;

[0074] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0075] Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features;

[0076] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0077] The image embedding representation is input into the denoising U-Net via cross attention.

[0078] In order to recover dense video frames from sparse video frames, achieve the effect of interpolation, and at the same time improve the resolution of the video to generate higher quality videos. The input is the first video, which contains several frames (for example, N_frames frames). Make sure that these frames contain faces or other key features for interpolation and resolution improvement. Then extract a series of conditional pictures from the input video as key frames. These conditional pictures are usually frames containing important information or features. Perform VAE encoding operations on each conditional picture to obtain a low-resolution latent space feature representation.

[0079] According to the actual situation, these encoding results are appropriately copied to ensure that the total number is equal to the predetermined N_frames value. For example, if the original conditional image has M frames, but N_frames frames need to be generated, then the encoding of each conditional image may need to be copied N_frames / M times. The sparse video frames are encoded into low-resolution latent space feature representations through VAE, and Gaussian noise of random intensity is added to these latent space feature representations to obtain noisy feature representations. Each conditional image is encoded using the CLIP image encoder to obtain its corresponding second image embedding representation to form a sequence. The VAE encodings of these conditional images are concatenated with the above-mentioned noisy feature representations. The obtained CLIP embedding sequence is integrated into the denoising U-Net through the cross-attention mechanism to ensure that the generated content is consistent and coherent with the provided conditional image.

[0080] The noisy latent space features are input into the denoising U-Net to generate denoised latent space features. The goal of the denoising U-Net is to make the output as close as possible to the latent space representation before noise addition, thereby improving the quality and stability of the generated video. The denoised latent space features are input into the VAE decoder to generate intermediate resolution video frames. The generated intermediate resolution video frames are processed using a super-resolution model (such as ESRGAN, SRGAN, etc.) to further improve the resolution. The super-resolution model can convert low-resolution images into high-resolution images and improve the overall quality of the video. The interpolated and resolution-enhanced video frames are combined into the final video. The generated video has a longer duration and higher resolution.

[0081] During the training process, the parameters are continuously adjusted so that the denoising U-Net can accurately reconstruct the original unnoised video frames from the noisy data, while ensuring that the generated content remains consistent and coherent with the provided conditional images.

[0082] When processing sequence data, the cross-attention mechanism allows the model to focus on related information in one sequence (query) in another sequence (key and value). In the denoising U-Net, the currently processed data (such as noisy feature representation) is used as the query, and the image embedding vector is used as the key and value. The denoising U-Net can take into account the key information of the conditional image when processing each layer of features.

[0083] Therefore, in an embodiment of the present application, a user provides a conditional picture, which is first converted into a low-dimensional latent variable representation by a VAE encoder, and this latent variable representation is copied N_frames times (for example, 25 times) to form an initial noisy feature representation. This noisy feature representation is usually a standard Gaussian noise. The denoising U-Net receives this noisy feature representation and gradually removes the noise. Each forward propagation removes part of the noise from the noisy feature representation. At the same time, the embedding vector generated by the CLIP picture encoder is injected into the denoising U-Net through the cross-attention mechanism to ensure that the generated video content is consistent with the conditional picture. After several denoising, a short-duration video with a low frame rate is finally obtained. Although this video already has basic dynamic effects, it has fewer frames and a relatively low resolution.

[0084] The short-duration, low-frame-rate video generated above is used as input. The key frames in this video will be used as conditional images in the second stage. Each key frame is encoded by the VAE encoder to obtain multiple latent variable representations. Each latent variable representation is replicated a certain number of times as needed to ensure that the total number reaches N_frames. These latent variable representations are then combined with new standard Gaussian noise to form a new noisy feature representation. The denoising U-Net processes these noisy feature representations again to gradually remove the noise. The multiple embedding vectors generated by the CLIP image encoder are injected into the denoising U-Net through the cross-attention mechanism to ensure that the generated video content is consistent with multiple conditional images. After multiple denoising, a video with a longer duration, higher frame rate and higher resolution is finally obtained. This video not only retains the main visual elements of the original conditional image, but also makes the video smoother and more natural through interpolation technology.

[0085] Compared with the prior art, the video generation method proposed in the embodiment of the present application has the following technical effects:

[0086] First, by excluding video samples without faces during the training data screening phase, we ensure that model training focuses on understanding and generating images with natural faces, thereby reducing the risk of face distortion or distortion in generated videos. This helps improve the realism and audience acceptance of generated content.

[0087] Second, the learning goal of the first stage is to teach the model how to create dynamic scenes based on a single picture and achieve the transition from static to dynamic. The model uses the collaborative work of VAE, CLIP image encoder and denoising U-Net to not only generate video clips consistent with the subject of the conditional picture, but also gradually refine the video content through the denoising process to ensure the clarity and coherence of the generated results.

[0088] The third and second phases focus on video interpolation technology, which enables the model to insert new frames based on existing video sequences, effectively extending the video duration and increasing the richness and smoothness of the action. This process is guided by the use of serialized CLIP embeddings and the splicing of multi-frame VAE encoding to ensure the coherence of the newly added frames with the original video content, thus improving the video viewing experience.

[0089] Fourth, by setting the N_frames parameter to limit the number of frames generated in each round, the technical solution shows sensitive consideration of hardware resources, especially GPU memory, and achieves efficient training under limited resources. At the same time, this mechanism also enhances the model's adaptability to different video lengths and resolution requirements.

[0090] Fifth, it provides an end-to-end solution from a single picture to a complete, high-quality video. Users only need to provide a picture to get a long video that contains rich dynamic changes while maintaining the authenticity of characters and scenes, meeting the needs of a variety of application scenarios from creative preview, content generation to entertainment, advertising, etc.

[0091] A possible implementation of a video generation device based on a diffusion model is provided below, which is used to execute the various execution steps and corresponding technical effects of the video generation method shown in the above embodiment and possible implementation. The device includes:

[0092] A splitting module is used to split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as a conditional image;

[0093] A first encoding module, used for inputting a plurality of video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation;

[0094] A second encoding module, used for encoding the conditional image using a CLIP image encoder to obtain an image embedding representation;

[0095] A splicing module, used for inputting the noisy feature representation and the image embedding representation into a denoising U-Net for splicing to obtain a first video;

[0096] The video optimization module is used to input the first video into the diffusion model to obtain a second video, wherein the second video has a higher video quality than the first video.

[0097] In a possible implementation manner, the first encoding module is further configured to:

[0098] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0099] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0100] In a possible implementation, the step of inputting the first video into the diffusion model to obtain the second video includes:

[0101] Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation;

[0102] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0103] Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features;

[0104] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0105] In one possible implementation, the image embedding representation is input into the denoising U-Net in the form of cross attention.

[0106] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the video generation method provided in the embodiments of the present application. Therefore, the beneficial effects of the video generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0107] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps of any embodiment of the video generation method provided in the embodiment of the present application.

[0108] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0109] Since the instructions stored in the storage medium can execute the steps in any video generation method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any video generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0110] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video generation method based on a diffusion model, characterized in that: The method is applied to a diffusion model, the diffusion model includes a variational autoencoder, a CLIP picture encoder and a denoising U-Net, and the method includes: Split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image; Input multiple video frames into the variational autoencoder and add Gaussian noise to obtain noisy feature representation; Use CLIP image encoder to encode the conditional image to obtain image embedding representation; The noisy feature representation and the image embedding representation are input into the denoising U-Net for concatenation to obtain the first video; The first video is input into a diffusion model to obtain a second video, wherein the second video has a higher video quality than the first video.

2. The video generation method according to claim 1, characterized in that: The step of inputting a plurality of video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation includes: Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation; Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

3. The video generation method according to claim 1, characterized in that: The step of inputting the first video into the diffusion model to obtain the second video includes: Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation; Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net; Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features; After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

4. The video generation method according to claim 1, characterized in that: The image embedding representation is input into the denoising U-Net via cross attention.

5. A video generation device based on a diffusion model, characterized in that: The device comprises: A splitting module is used to split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as a conditional image; A first encoding module, used for inputting a plurality of video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation; A second encoding module, used for encoding the conditional image using a CLIP image encoder to obtain an image embedding representation; A splicing module, used for inputting the noisy feature representation and the image embedding representation into a denoising U-Net for splicing to obtain a first video; The video optimization module is used to input the first video into the diffusion model to obtain a second video, wherein the second video has a higher video quality than the first video.

6. The video generating device according to claim 5, characterized in that: The first encoding module is further used for: Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation; Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

7. The video generating device according to claim 5, characterized in that: The step of inputting the first video into the diffusion model to obtain the second video includes: Input the first video into the variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation; Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net; Input the second noisy feature representation into the denoising U-Net to generate denoised latent space features; After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

8. The video generating device according to claim 5, characterized in that: The image embedding representation is input into the denoising U-Net via cross attention.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the video generation method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the video generation method according to any one of claims 1 to 4.

Images