A video generation method, device, equipment and medium based on diffusion model

Abstract

The embodiments of the present application provide a video generation method, apparatus, device and medium based on a diffusion model, which relate to the field of video generation technology. First, a training video containing a face is split into multiple video frames according to the number of frames, the first video frame containing the face is selected as a conditional image, the multiple video frames are input into a variational autoencoder and Gaussian noise is added to obtain a noisy feature representation, the conditional image is encoded using a CLIP image encoder to obtain an image embedding representation, the noisy feature representation and the image embedding representation are input into a denoising U-Net for splicing to obtain a first video, and the first video is input into a diffusion model to obtain a second video. By inputting the image into the diffusion model, the goal of generating high-quality, long-term videos from a single image is achieved, and the naturalness of face generation and the smoothness of video movements are improved.

Description

Technical Field

[0001] The present application relates to the field of video generation technology, and in particular to a method, apparatus, device and medium for video generation based on a diffusion model. Background Art

[0002] Existing video generation technology is based on machine learning and artificial intelligence, particularly advances in deep learning, which utilize deep neural networks to generate visual content such as faces and human motion. This involves complex models such as generative adversarial networks (GANs) and variational autoencoders (VAEs), which can mimic real-world data distributions to create novel sequences of facial images or full-body motion videos. Multi-view images or depth sensor data are combined to construct three-dimensional models of scenes or people, which are then used to generate videos. This technology helps improve the three-dimensionality and realism of the synthesized videos.

[0003] Despite significant progress, video generation technology still faces a series of challenges and limitations. For example, while algorithms can generate dynamic facial images based on existing images or videos, maintaining the consistency of facial features (such as expression, skin tone, and lighting conditions) between consecutive frames is a difficult task. This can cause facial features in the video to flicker and texture discontinuity, affecting the viewing experience. Furthermore, the naturalness of the movements is insufficient, and it is often difficult to accurately capture and replicate the subtle details and smoothness of human movements, especially complex body movements and micro-expressions. As a result, the generated character movements may appear stiff, mechanical, or unnatural, lacking the soft transitions and natural rhythm of real human movements. Furthermore, when dealing with human motion and changing perspectives, the inaccuracies of 3D reconstruction and pose estimation often lead to distortion or disproportion of the character's body parts, such as abnormal stretching of hands and feet and dislocated joints, which seriously undermines the realism of the video.

[0004] Therefore, how to study a video generation method to solve the above technical problems has become a challenge that those skilled in the art have to face. Summary of the Invention

[0005] The present application provides a method, apparatus, device and medium for video generation based on a diffusion model. By inputting an image into the diffusion model, the goal of generating high-quality, long-duration video from a single image is achieved, and the naturalness of face generation and the smoothness of video movements are improved.

[0006] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a video generation method based on a diffusion model, the method being applied to the diffusion model, the diffusion model including a variational autoencoder, a CLIP image encoder, and a denoising U-Net, the method comprising:

[0008] Split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as the conditional image;

[0009] Input multiple video frames into the variational autoencoder and add Gaussian noise to obtain noisy feature representation;

[0010] Use CLIP image encoder to encode the conditional image to obtain image embedding representation;

[0011] The noisy feature representation and the image embedding representation are input into the denoising U-Net for splicing to obtain the first video;

[0012] The first video is input into a diffusion model to obtain a second video, where the second video has higher video quality than the first video.

[0013] In a possible embodiment, the step of inputting multiple video frames into a variational autoencoder and adding Gaussian noise to obtain a noisy feature representation includes:

[0014] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0015] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0016] In a possible embodiment, the step of inputting the first video into a diffusion model to obtain the second video includes:

[0017] The first video is input into the variational autoencoder for encoding and Gaussian noise is added to obtain a second noisy feature representation;

[0018] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0019] The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features;

[0020] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0021] In one possible embodiment, the image embedding representation is input into the denoising U-Net via cross attention.

[0022] In a second aspect, an embodiment of the present application further proposes a video generation device based on a diffusion model, the device comprising:

[0023] The splitting module is used to split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image;

[0024] The first encoding module is used to input multiple video frames into a variational autoencoder and add Gaussian noise to obtain a noisy feature representation;

[0025] A second encoding module is used to encode the conditional image using the CLIP image encoder to obtain an image embedding representation;

[0026] A splicing module, configured to input the noisy feature representation and the image embedding representation into a denoising U-Net for splicing to obtain a first video;

[0027] The video optimization module is used to input the first video into the diffusion model to obtain a second video, where the second video has higher video quality than the first video.

[0028] In a possible embodiment, the first encoding module is further configured to:

[0029] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0030] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0031] In a possible embodiment, the step of inputting the first video into a diffusion model to obtain the second video includes:

[0032] The first video is input into the variational autoencoder for encoding and Gaussian noise is added to obtain a second noisy feature representation;

[0033] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0034] The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features;

[0035] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0036] In one possible embodiment, the image embedding representation is input into the denoising U-Net via cross attention.

[0037] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the video generation method as described in any one of the first aspects.

[0038] In a fourth aspect, the present application further proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video generation method as described in any one of the first aspects.

[0039] The above-mentioned main scheme of this application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and protected by this application; and in this application, (non-conflicting options) can also be freely combined with each other and with other options. After understanding the scheme of this application, those skilled in the art will understand that there are many combinations based on existing technology and common knowledge, all of which are technical solutions to be protected by this application, and they are not exhaustive here.

[0040] Compared with the existing technology, the embodiments of the present application propose a video generation method, device, equipment and medium based on a diffusion model, which relates to the field of video generation technology. First, a training video containing a face is split into multiple video frames according to the number of frames, and the first video frame containing a face is selected as a conditional image. The multiple video frames are input into a variational autoencoder and Gaussian noise is added to obtain a noisy feature representation. The conditional image is encoded using a CLIP image encoder to obtain an image embedding representation. The noisy feature representation and the image embedding representation are input into a denoising U-Net for splicing to obtain a first video. The first video is input into a diffusion model to obtain a second video. By inputting the image into the diffusion model, the goal of generating high-quality, long-term videos from a single image is achieved, and the naturalness of face generation and the smoothness of video movements are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flow chart of a video generation method based on a diffusion model proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0045] It should be noted that, in the absence of conflict, the features in the embodiments of this application can be combined with each other.

[0046] Variational Auto Encoder (VAE): VAEs learn efficient representations of data (often called latent space representations) to capture key features in an image and reconstruct the image from these features or generate new, similar data. Specifically for the task of generating videos from images, VAEs help models understand and extract information such as the basic structure and texture of people or objects in images.

[0047] CLIP Image Encoder: CLIP (Contrastive Language-Image Pre-training) is a pre-trained model that learns cross-modal representations by comparing image and text data. The CLIP image encoder further enhances the model's understanding of image content, particularly its relevance to context or concepts. This helps generate videos that are not only visually coherent but also somewhat align with underlying semantic information, making the generated actions or scenes more logical and predictable.

[0048] Denoising U-Net: U-Net is a convolutional neural network architecture commonly used for image segmentation and generation tasks, known for its unique encoder-decoder structure and skip connections. This technical solution uses a variant of it, the Denoising U-Net, which primarily addresses potential noise during video generation, improving video clarity and continuity. By gradually adding details and reducing unnecessary interference, the conversion from image to video is smooth and realistic.

[0049] To address the difficulties in achieving facial consistency and integrity in videos generated by existing technologies, as well as the unnatural and distorted movements of people, the present application proposes a method, apparatus, device, and medium for video generation based on a diffusion model. By establishing and training a diffusion model, the model is able to predict and generate a series of continuous frames based on the input image, thereby forming a short but coherent video clip. In this way, the elements in the static image are brought to life, achieving dynamic expression. At the same time, considering hardware resource limitations, such as GPU memory size, the number of generated frames is set to balance computing requirements and generation quality. This is described in detail below.

[0050] Please refer to Figure 1 , Figure 1 A flow chart of a method for video generation based on a diffusion model proposed in an embodiment of the present application is shown. The method is applied to a diffusion model, which includes a variational autoencoder, a CLIP image encoder, and a denoising U-Net. The method includes the following steps:

[0051] Step S1: Split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as the conditional image.

[0052] First, to ensure the model is exposed to diverse data samples, a video is randomly selected from the entire training set for processing at each iteration. Each frame of the selected video is then inspected to see if any faces appear. Because the project objective explicitly focuses on high-quality face generation, if no frames in a video contain faces, the video is not used as a training sample and is discarded.

[0053] Once at least one frame in the video is confirmed to contain a face, a specified number (e.g., N_frames) of video frames are selected according to a specific rule (possibly at random intervals) as training data for the current batch. Among the N_frames frames, the first frame with a face is selected as the conditional image. The model's task is to understand how to use the given conditional frame (i.e., the initial frame containing a face) to reconstruct or generate the remaining video sequence.

[0054] In one possible embodiment, due to GPU memory limitations, a fixed number of frames (e.g., 25 frames) is set as the target length for each generation during actual training. The model only needs to process a fixed number of video frames per round, reducing the computational burden. This enables the denoising U-Net to effectively recover clear information from noisy data while ensuring that the generated video content matches the provided conditional image. Reducing the difference between the output and the original noise-free latent variable enables the model to learn to efficiently remove noise. The embedded representation of the conditional image is used to guide the generation process, ensuring the quality and consistency of the generated video, especially when complex objects such as faces are involved.

[0055] Step S2: Input multiple video frames into a variational autoencoder and add Gaussian noise to obtain a noisy feature representation.

[0056] N_frames video frames are first encoded using a variational autoencoder to obtain a latent space feature representation with lower resolution (downsampled 8 times). Then, a Gaussian noise of random intensity is added to the latent space feature representation to obtain a noisy feature representation, which is used as the input of the denoising U-Net.

[0057] Step S2 includes:

[0058] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0059] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0060] The N_frames video frames extracted from the selected video are first fed into a pre-trained VAE model. The VAE converts these high-resolution raw frames into a low-dimensional (downsampled by a factor of 8) latent space representation. To initiate the denoising learning mechanism during training, Gaussian noise of random intensity is added to the obtained latent variable representation. This noisy latent variable is called the noisy feature representation and becomes the input data for the denoising U-Net network.

[0061] Step S3: Use the CLIP picture encoder to encode the conditional picture to obtain a picture embedding representation.

[0062] The conditional image is converted into an embedding vector through the image encoder in CLIP, which captures the key visual features of the image and is added to the denoising U-Net in the form of cross attention to ensure that the generated video content is consistent with the provided conditional image.

[0063] Step S4: Input the noisy feature representation and the image embedding representation into the denoising U-Net for splicing to obtain the first video.

[0064] In addition to being used to generate embedded representations, the conditional image is also encoded by VAE and its encoding result is replicated N_frames times. These repeated encoding results are then concatenated with the above-mentioned noisy feature representation and used as the input of the denoising U-Net.

[0065] The diffusion model is designed to enable the denoising U-Net to learn how to recover clear information from noisy data. During training, the difference between the denoising U-Net output and the original unnoised latent variable is minimized, allowing the model to remove noise while retaining or enhancing useful signals.

[0066] Previously, a single conditional image was converted into a fixed embedding vector using the CLIP image encoder. After the conditional image is encoded by the VAE, the encoded result is replicated N_frames times and then concatenated with the noisy feature representation to recover the effective information from the noise and generate continuous and high-quality video frames.

[0067] Step S5: Input the first video into the diffusion model to obtain the second video.

[0068] The second video has higher video quality than the first video.

[0069] The first video is short and has a low frame rate, while the second video is long and has a higher resolution. Unlike the previous approach using a single conditional image, step S5 uses a series of conditional images as input. These images are key frames sparsely distributed throughout the target video. Each conditional image is encoded to obtain multiple embedding vectors, forming a sequence.

[0070] VAE encoding is performed on each conditional image separately. Each encoding result is replicated a certain number of times as needed to ensure that the final total number reaches N_frames. This better preserves visual information at different time points or viewpoints. These encoded latent variables are then concatenated with the noisy feature representation and used as the input to the denoising U-Net.

[0071] The goal is to recover a complete and coherent video sequence from sparsely given key frames. The interpolation capability is emphasized, and newly generated transition frames are inserted between some known frames to create smooth and natural dynamic images.

[0072] Step S5 includes:

[0073] The first video is input into the variational autoencoder for encoding and Gaussian noise is added to obtain a second noisy feature representation;

[0074] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0075] The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features;

[0076] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0077] The image embedding representation is input into the denoising U-Net via cross attention.

[0078] In order to recover dense video frames from sparse video frames, achieve the effect of interpolation, and at the same time improve the resolution of the video to generate higher quality videos. The input is a first video, which contains several frames (for example, N_frames frames). Ensure that these frames contain faces or other key features for interpolation and resolution improvement. Then, a series of conditional images are extracted from the input video as key frames. These conditional images are usually frames containing important information or features. VAE encoding operation is performed on each conditional image to obtain a low-resolution latent space feature representation.

[0079] Based on the actual situation, these encoding results are appropriately copied to ensure that the total number is equal to the predetermined N_frames value. For example, if the original conditional image has M frames, but N_frames frames need to be generated, then the encoding of each conditional image may need to be copied N_frames / M times. The sparse video frames are encoded into low-resolution latent space feature representations through VAE, and Gaussian noise of random intensity is added to these latent space feature representations to obtain noisy feature representations. Each conditional image is encoded using the CLIP image encoder to obtain its corresponding second image embedding representation to form a sequence. The VAE encoding of these conditional images is spliced ​​together with the above-mentioned noisy feature representation. The obtained CLIP embedding sequence is integrated into the denoising U-Net through the cross-attention mechanism to ensure that the generated content is consistent and coherent with the provided conditional image.

[0080] The noisy latent space features are input into the denoising U-Net to generate denoised latent space features. The goal of the denoising U-Net is to make the output as close as possible to the latent space representation before noise addition, thereby improving the quality and stability of the generated video. The denoised latent space features are input into the VAE decoder to generate intermediate resolution video frames. The generated intermediate resolution video frames are processed using a super-resolution model (such as ESRGAN, SRGAN, etc.) to further improve the resolution. The super-resolution model can convert low-resolution images into high-resolution images, improving the overall quality of the video. The interpolated and resolution-upgraded video frames are combined into the final video. The generated video has a longer duration and higher resolution.

[0081] During the training process, the parameters are continuously adjusted so that the denoising U-Net can accurately reconstruct the original unnoised video frames from the noisy data, while ensuring that the generated content remains consistent and coherent with the provided conditional images.

[0082] When processing sequential data, the cross-attention mechanism allows a model to focus on relevant information in one sequence (the query) from another sequence (the key and value). In a denoising U-Net, the currently processed data (e.g., noisy feature representations) serves as the query, and the image embedding vectors serve as the key and value. This allows the denoising U-Net to consider key information about the conditional image when processing features at each layer.

[0083] Therefore, in an embodiment of the present application, the user provides a conditional image, which is first converted into a low-dimensional latent variable representation through a VAE encoder. This latent variable representation is copied N_frames times (for example, 25 times) to form an initial noisy feature representation. This noisy feature representation is usually a standard Gaussian noise. The denoising U-Net receives this noisy feature representation and gradually removes the noise. Each forward propagation removes some noise from the noisy feature representation. At the same time, the embedding vector generated by the CLIP image encoder is injected into the denoising U-Net through the cross-attention mechanism to ensure that the generated video content is consistent with the conditional image. After several denoising cycles, a short-duration video with a low frame rate is finally obtained. Although this video already has basic dynamic effects, it has fewer frames and a relatively low resolution.

[0084] The short, low-frame-rate video generated above is used as input. Key frames from this video serve as conditional images in the second stage. Each key frame is encoded using a VAE encoder, resulting in multiple latent variable representations. Each latent variable representation is replicated a certain number of times as needed to ensure a total of N_frames. These latent variable representations are then combined with new standard Gaussian noise to form a new noisy feature representation. The denoising U-Net further processes these noisy feature representations, gradually removing the noise. The multiple embedding vectors generated by the CLIP image encoder are fed into the denoising U-Net using a cross-attention mechanism, ensuring that the generated video content is consistent with the multiple conditional images. After multiple denoising steps, the final result is a longer video with a higher frame rate and resolution. This video not only retains the key visual elements of the original conditional image but also uses interpolation to achieve a smoother and more natural video.

[0085] Compared with the prior art, the video generation method proposed in the embodiment of the present application has the following technical effects:

[0086] First, by excluding video samples without faces during the training data screening phase, we ensure that model training focuses on understanding and generating images containing natural faces, thereby reducing the risk of face distortion or distortion in generated videos. This helps improve the realism and audience acceptance of generated content.

[0087] The second and first phases of learning aim to teach the model how to create dynamic scenes from a single image, achieving the transition from static to dynamic. Leveraging the collaborative work of VAE, CLIP image encoder, and denoising U-Net, the model not only generates video clips that align with the subject matter of the conditional image but also gradually refines the video content through denoising, ensuring clarity and coherence in the generated results.

[0088] The third and second phases focused on video interpolation technology, enabling the model to insert new frames into existing video sequences, effectively extending the video duration and increasing the richness and fluidity of the action. This process was guided by the use of sequenced CLIP embeddings and the splicing of multi-frame VAE encodings, ensuring the coherence of the added frames with the original video content, thus improving the viewing experience.

[0089] Fourth, by setting the N_frames parameter to limit the number of frames generated per round, this technical solution demonstrates sensitive consideration of hardware resources, particularly GPU memory, enabling efficient training with limited resources. This mechanism also enhances the model's adaptability to varying video lengths and resolutions.

[0090] Fifth, it provides an end-to-end solution from a single image to a complete, high-quality video. Users only need to provide a single image to generate a long video that contains rich dynamic changes while maintaining the authenticity of the characters and scenes. This meets the needs of various application scenarios, from creative preview and content generation to entertainment and advertising.

[0091] A possible implementation of a video generation device based on a diffusion model is provided below, which is used to execute the various execution steps and corresponding technical effects of the video generation method shown in the above embodiment and possible implementation. The device includes:

[0092] The splitting module is used to split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image;

[0093] The first encoding module is used to input multiple video frames into a variational autoencoder and add Gaussian noise to obtain a noisy feature representation;

[0094] A second encoding module is used to encode the conditional image using the CLIP image encoder to obtain an image embedding representation;

[0095] A splicing module, configured to input the noisy feature representation and the image embedding representation into a denoising U-Net for splicing to obtain a first video;

[0096] The video optimization module is used to input the first video into the diffusion model to obtain a second video, where the second video has higher video quality than the first video.

[0097] In a possible implementation, the first encoding module is further configured to:

[0098] Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation;

[0099] Adding Gaussian noise to the latent space feature representation obtains the noisy feature representation.

[0100] In one possible implementation, the step of inputting the first video into a diffusion model to obtain the second video includes:

[0101] The first video is input into the variational autoencoder for encoding and Gaussian noise is added to obtain a second noisy feature representation;

[0102] Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net;

[0103] The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features;

[0104] After the denoised latent space features are input into the variational autoencoder, the super-resolution model is used to improve the resolution to obtain the second video.

[0105] In one possible implementation, the image embedding representation is input into the denoising U-Net via cross-attention.

[0106] This preferred embodiment provides a computer device that can implement the steps in any embodiment of the video generation method provided in the embodiments of the present application. Therefore, the beneficial effects of the video generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0107] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, the present invention provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the video generation methods provided in the present invention.

[0108] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0109] Since the instructions stored in the storage medium can execute the steps in any video generation method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any video generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0110] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video generation method based on a diffusion model, characterized in that: The method is applied to a diffusion model, wherein the diffusion model includes a variational autoencoder, a CLIP image encoder, and a denoising U-Net. The method includes: Split the training video containing a face into multiple video frames according to the number of frames, and select the first video frame containing a face as the conditional image; Input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation; Adding Gaussian noise to the latent space feature representation to obtain noisy feature representation; Use CLIP image encoder to encode the conditional image to obtain image embedding representation; The noisy feature representation and the image embedding representation are input into the denoising U-Net for concatenation to obtain the first video. The image embedding representation is input into the denoising U-Net through cross attention. The first video is input into the variational autoencoder for encoding and Gaussian noise is added to obtain a second noisy feature representation; Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net; The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features; After the denoised latent space features are input into a variational autoencoder, a super-resolution model is used to perform resolution enhancement to obtain a second video, which has higher video quality than the first video.

2. A video generation device based on a diffusion model, characterized in that: The device comprises: The splitting module is used to split the training video containing human faces into multiple video frames according to the number of frames, and select the first video frame containing human faces as the conditional image; The first encoding module is used to input multiple video frames into the variational autoencoder for encoding to obtain latent space feature representation; Adding Gaussian noise to the latent space feature representation to obtain noisy feature representation; A second encoding module is used to encode the conditional image using the CLIP image encoder to obtain an image embedding representation; A splicing module is used to input the noisy feature representation and the image embedding representation into the denoising U-Net for splicing to obtain the first video, and the image embedding representation is input into the denoising U-Net in the form of cross attention; a video optimization module, configured to input the first video into a variational autoencoder for encoding and add Gaussian noise to obtain a second noisy feature representation; Use the CLIP image encoder to encode multiple conditional images in the first video to obtain a second image embedding representation, and inject the second image embedding representation into the denoising U-Net; The second noisy feature representation is input into the denoising U-Net to generate denoised latent space features; After the denoised latent space features are input into a variational autoencoder, a super-resolution model is used to perform resolution enhancement to obtain a second video, which has higher video quality than the first video.

3. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the video generation method according to claim 1.

4. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the video generation method according to claim 1.

Images