Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed video, and carrying out the preprocessing of the to-be-processed video, and obtaining a video frame sequence; for each video frame in the video frame sequence, inputting the video frame and random noise into a trained video diffusion model to obtain video latent layer features with noise; inputting the video latent layer features with noise into a trained de-noising VAE decoder for de-noising and decoding to obtain a clear video frame; and generating a clear video based on each clear video frame. According to the technical scheme, the video hidden layer features with noise generated by the diffusion model can be effectively denoised based on the denoising VAE decoder, a clear video is obtained, and the user experience is greatly improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video generation, and in particular to a method, device, electronic device and storage medium for video generation. Background Art

[0002] Video generation has been a hot research topic in the field of artificial intelligence in recent years, and its application areas are very wide, for example: film and television production: automatically generate video content such as movies and TV series to reduce production costs and cycles; virtual reality: create realistic virtual environments and characters to enhance user immersive experience; education and training: produce vivid teaching videos to improve learning efficiency and fun; medical health: generate medical imaging data to assist doctors in diagnosis and treatment, etc.

[0003] At present, the algorithm based on the diffusion model is one of the cutting-edge technologies in the field of video generation. The diffusion model can generate high-quality, high-resolution videos by learning the potential probability distribution in the data, and is used in many application fields.

[0004] Since the generation process of the existing diffusion model relies on the sampling of the latent feature space, and the space itself may contain noise, which in turn affects the clarity of the final generated video, affects user usage, and greatly reduces the user experience. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a method, device, electronic device and storage medium for video generation, which can effectively denoise the noisy latent features of the video generated by the diffusion model based on the denoising VAE decoder to obtain a clear video, thereby greatly improving the user experience.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a video, the method comprising:

[0007] Obtain a video to be processed, and preprocess the video to be processed to obtain a video frame sequence;

[0008] For each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into the trained video diffusion model for forward noise addition to obtain the latent features of the video with noise;

[0009] Input the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain clear video frames;

[0010] A clear video is generated based on the respective clear video frames.

[0011] In a possible implementation, the video frame and random noise are input into a trained video diffusion model to obtain latent features of the video with noise, including:

[0012] Input the video frame into the trained video diffusion model to obtain the video latent features;

[0013] The random noise is input into the trained video diffusion model. The trained video diffusion model will mix the random noise with the video latent features through the forward diffusion process to obtain the video latent features with noise.

[0014] In one possible implementation, the training process of the denoising VAE decoder includes:

[0015] Obtaining a first training video; wherein the first training video is a video with a clear image;

[0016] Input the first training video into the VAE encoder to obtain a feature map;

[0017] Add random noise to the feature map;

[0018] Input the feature map with random noise into the VAE decoder to obtain the reconstructed video;

[0019] By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

[0020] In a possible implementation, the training process of the video diffusion model includes:

[0021] Get the second training video;

[0022] Inputting the second training video into the video diffusion model to be trained to obtain a predicted video;

[0023] The video diffusion model to be trained is trained based on the predicted video and the second training video. If a preset training end condition is met, a trained video diffusion model is obtained.

[0024] In a possible implementation, generating a clear video based on each clear video frame includes:

[0025] The clear video frames are spliced ​​in chronological order to generate a clear video.

[0026] In a second aspect, an embodiment of the present invention provides a device for generating a video, the device comprising:

[0027] An acquisition module is used to acquire a video to be processed and pre-process the video to be processed to obtain a video frame sequence;

[0028] The noise adding module is used to perform the following operations on each video frame in the video frame sequence: inputting the video frame into the trained video diffusion model to obtain the latent features of the video with noise;

[0029] The denoising decoding module is used to input the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain clear video frames;

[0030] The generating module is used to generate a clear video based on each clear video frame.

[0031] In a possible implementation, the noise adding module is further used to input the video frame into a trained video diffusion model to obtain the video latent features;

[0032] The random noise is input into the trained video diffusion model. The trained video diffusion model will mix the random noise with the video latent features through the forward diffusion process to obtain the video latent features with noise.

[0033] In one possible implementation, the training process of the denoising VAE decoder includes:

[0034] Obtaining a first training video; wherein the first training video is a video with a clear image;

[0035] Input the first training video into the VAE encoder to obtain a feature map;

[0036] Add random noise to the feature map;

[0037] Input the feature map with random noise into the VAE decoder to obtain the reconstructed video;

[0038] By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

[0039] In a possible implementation, the training process of the video diffusion model includes:

[0040] Get the second training video;

[0041] Inputting the second training video into the video diffusion model to be trained to obtain a predicted video;

[0042] The video diffusion model to be trained is trained based on the predicted video and the second training video. If a preset training end condition is met, a trained video diffusion model is obtained.

[0043] In a possible implementation, the generation module further includes: splicing the clear video frames in time sequence to generate a clear video.

[0044] In a third aspect, an embodiment of the present invention provides an electronic device, which includes: a processor and a memory, the processor is used to execute a video generation program stored in the memory to implement the above-mentioned video generation method.

[0045] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned video generation method.

[0046] The video generation method, device, electronic device and storage medium provided by the embodiment of the present invention include: obtaining a video to be processed, preprocessing the video to be processed to obtain a video frame sequence; for each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into a trained video diffusion model to obtain a latent feature of the video with noise; inputting the latent feature of the video with noise into a trained denoising VAE decoder for denoising decoding to obtain a clear video frame; and generating a clear video based on each clear video frame. The above technical solution can effectively denoise the latent feature of the video with noise generated by the diffusion model based on the denoising VAE decoder to obtain a clear video, which greatly improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of an embodiment of a method for generating a video provided by an embodiment of the present invention;

[0048] Figure 2 A system diagram of a video generation process provided by an embodiment of the present invention;

[0049] Figure 3 A flowchart of an embodiment of a denoising VAE decoder training process provided by an embodiment of the present invention;

[0050] Figure 4 A block diagram of an embodiment of a video generation device provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0054] The embodiment of the present invention provides a method for generating a video. Figure 1 , Figure 1 A flow chart of an embodiment of a method for generating a video provided by an embodiment of the present invention. Figure 1 The process shown may include the following steps:

[0055] Step 101, obtaining a video to be processed, and preprocessing the video to be processed to obtain a video frame sequence;

[0056] Specifically, the preprocessing is to decompose the video to be processed frame by frame to obtain a video frame sequence with a time sequence. In practical applications, professional video editing software or AI (Artificial Intelligence) technology can be used to decompose the video to be processed frame by frame.

[0057] Among them, use professional video editing software for frame-by-frame decomposition: This method is suitable for precise frame-level processing of the video to be processed. For example, using professional video editing software such as Adobe Premiere Pro, Final Cut Pro or DaVinci Resolve, the video to be processed can be decomposed into frame by frame for processing. These software usually provide a timeline panel that allows users to accurately locate each frame of the video to be processed and perform editing, exporting and other operations.

[0058] Using AI technology to decompose the video to be processed frame by frame: With the development of technology, AI technology can now be used to decompose the video to be processed into frames of images. For example, using deep learning algorithms and computer vision technology, frame-by-frame processing and analysis of the video to be processed can be achieved. This method usually requires the use of specialized software or APIs, such as OpenCV, which can process video files and convert them into video frames.

[0059] Step 102, for each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into a trained video diffusion model to obtain latent features of the video with noise;

[0060] The specific process of obtaining the latent features of the video with noise is as follows: inputting the video frame into the trained video diffusion model to obtain the latent features of the video; inputting the random noise into the trained video diffusion model, and the trained video diffusion model will mix the random noise with the latent features of the video through the forward diffusion process to obtain the latent features of the video with noise.

[0061] The video diffusion model uses the forward process and the reverse process to obtain the latent features of the video. Among them, the forward process (diffusion process): This process is the process of gradually converting the video frame into a Gaussian noise distribution. Through a series of steps of gradually adding noise, the original video frame is converted into noise data. This step can be regarded as a Markov process, in which the video frame at each moment is only related to the video frame at the previous moment. This process simulates the gradual "diffusion" process from data distribution to simple noise distribution.

[0062] Reverse process (denoising process): In contrast to the forward process, the reverse process learns to perform the inverse operation of the forward process, that is, starting from pure noise, gradually "denoising" through a series of inverse steps, and finally generating samples close to the original data distribution. Through this process, the video diffusion model can reconstruct high-quality data samples from the noise, including video frames, thereby extracting the video latent features of the video frames.

[0063] Subsequently, when random noise is input into the video diffusion model, the video diffusion model will first mix the input random noise with the original data distribution through a forward diffusion process to generate a noisy data representation, which is the noisy video latent feature.

[0064] Among them, the video latent features are low-dimensional feature information used to abstractly represent the video content to be processed, which contains key information such as the shape and motion pattern of the object.

[0065] Step 103, input the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain a clear video frame;

[0066] The denoising VAE (Variational Autoencoder) decoder plays a key role in the denoising process by denoising and decoding the latent features of the video to be processed to restore the quality of the original video. The main feature of VAE is that it can learn the latent distribution of data and generate new data through latent space sampling. In the denoising task, VAE can restore the original image by learning the latent representation of the image, thereby achieving the effect of removing noise. Specifically for video denoising, the working principle of the denoising VAE decoder is mainly to denoise and decode the noisy and low-dimensional latent features of the video back to the pixel space to generate denoised video frames. This step utilizes a decoder network that learns how to reconstruct visually clear video frames from vectors in the low-dimensional latent space.

[0067] Step 104: Generate a clear video based on each clear video frame.

[0068] Specifically, the clear video frames obtained from step 102 to step 103 are spliced ​​in time sequence to generate a clear video for the user to watch.

[0069] In order to facilitate the understanding of the above content, Figure 2 A system diagram of a video generation process is shown, such as Figure 2 As shown, the user side inputs the video frames and random noise into the trained video diffusion model, generates the noisy video latent features through the video diffusion model, and inputs the noisy video latent features into the trained denoising VAE decoder. The denoising VAE decoder denoises and decodes the video latent features to obtain clear video frames, and sends the clear video spliced ​​from each clear video frame to the user side.

[0070] The video generation method provided by the embodiment of the present invention includes: obtaining a video to be processed, preprocessing the video to be processed to obtain a video frame sequence; for each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into a trained video diffusion model to obtain a latent feature of the video with noise; inputting the latent feature of the video with noise into a trained denoising VAE decoder for denoising decoding to obtain a clear video frame; and generating a clear video based on each clear video frame. The above technical solution can effectively denoise the latent feature of the video with noise generated by the diffusion model based on the denoising VAE decoder to obtain a clear video, which greatly improves the user experience.

[0071] See also Figure 3 , Figure 3 A flowchart of an embodiment of a denoising VAE decoder training process provided in an embodiment of the present invention. Figure 3 The process shown may include the following steps:

[0072] Step 301, obtaining a first training video; wherein the first training video is a video with a clear image;

[0073] Step 302, input the first training video into the VAE encoder to obtain a feature map;

[0074] The VAE encoder can be understood as the encoder of VAE. VAE is a deep learning model that combines the advantages of autoencoders and generative adversarial networks and can be used for different types of data generation and dimensionality reduction tasks. The core idea of ​​VAE is to approximate the data generation model through a variational distribution to achieve data encoding and decoding. In the structure of VAE, the VAE encoder part plays a key role. It compresses the vector in the input feature space into the hidden feature space to obtain hidden variables. This process can be regarded as a feature extraction, that is, extracting a feature map that can represent the significant features of the video from the first training video to achieve the purpose of dimensionality reduction. The feature map is a low-dimensional representation of the first training video.

[0075] Step 303, adding random noise to the feature map;

[0076] In practical applications, the way of adding random noise to the feature map is different depending on the noise type of random noise. If the noise type of random noise is Gaussian, you can use the NumPy library to generate Gaussian distributed random numbers and add them to the feature map; if the noise type of random noise is pulse, you can generate black and white pixels at random positions and add them to the feature map.

[0077] Step 304, input the feature map with random noise into the VAE decoder to obtain a reconstructed video;

[0078] The VAE decoder can be understood as the decoder of VAE. The VAE decoder reconstructs the video through the feature vectors in the latent space. This process involves restoring the features extracted by the VAE encoder from the original video data into video frames through the decoder. The VAE decoder part is usually composed of multiple fully connected layers or convolutional layers, which are responsible for mapping the feature vectors of the latent space back to the spatial dimensions of the original video. Through the processing of these layers, the VAE decoder can generate new video frames that are visually similar to the original video, thereby obtaining a reconstructed video.

[0079] Step 305, by minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain a denoising VAE decoder.

[0080] Specifically, the loss value is calculated by reconstructing the video and the first training video, and the loss value is as follows:

[0081]

[0082] Among them, x i represents the pixel value of the first training video, Represents the pixel value of the reconstructed video, and N represents the total number of pixels.

[0083] The network parameters of the VAE decoder are determined by minimizing the above loss value Loss, and a trained denoising VAE decoder is obtained.

[0084] The training process of the above-mentioned video diffusion model includes: obtaining a second training video; inputting the second training video into the video diffusion model to be trained to obtain a predicted video; training the video diffusion model to be trained based on the predicted video and the second training video, and obtaining a trained video diffusion model if a preset training end condition is met.

[0085] The preset training end condition may be a preset number of training times or an end condition when the training function value is the minimum, which is not limited here. Since the training process of the video diffusion model is a standard diffusion model training process, it will not be described in detail here.

[0086] See also Figure 4 , is a block diagram of an embodiment of a video generation device provided by an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0087] An acquisition module 401 is used to acquire a video to be processed and pre-process the video to be processed to obtain a video frame sequence;

[0088] Noise adding module 402 is used to perform, for each video frame in the video frame sequence: input the video frame into the trained video diffusion model to obtain the latent features of the video with noise;

[0089] Denoising decoding module 403, used for inputting the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain a clear video frame;

[0090] The generating module 404 is configured to generate a clear video based on each clear video frame.

[0091] The video generation device provided by the embodiment of the present invention includes: obtaining a video to be processed, preprocessing the video to be processed to obtain a video frame sequence; for each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into a trained video diffusion model to obtain a latent feature of the video with noise; inputting the latent feature of the video with noise into a trained denoising VAE decoder for denoising decoding to obtain a clear video frame; and generating a clear video based on each clear video frame. The above technical solution can effectively denoise the latent feature of the video with noise generated by the diffusion model based on the denoising VAE decoder to obtain a clear video, which greatly improves the user experience.

[0092] The noise adding module 402 is also used to input the video frame into the trained video diffusion model to obtain the video latent features;

[0093] The random noise is input into the trained video diffusion model. The trained video diffusion model will mix the random noise with the video latent features through the forward diffusion process to obtain the video latent features with noise.

[0094] In one possible implementation, the training process of the denoising VAE decoder includes:

[0095] Obtaining a first training video; wherein the first training video is a video with a clear image;

[0096] Input the first training video into the VAE encoder to obtain a feature map;

[0097] Add random noise to the feature map;

[0098] Input the feature map with random noise into the VAE decoder to obtain the reconstructed video;

[0099] By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

[0100] In a possible implementation, the training process of the video diffusion model includes:

[0101] Get the second training video;

[0102] Inputting the second training video into the video diffusion model to be trained to obtain a predicted video;

[0103] The video diffusion model to be trained is trained based on the predicted video and the second training video. If a preset training end condition is met, a trained video diffusion model is obtained.

[0104] The generation module 404 further includes: splicing the clear video frames in time sequence to generate a clear video.

[0105] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 5 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 5 Various buses are labeled as bus system 505 .

[0106] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0107] It can be understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0109] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 5022.

[0110] In the embodiment of the present invention, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment, for example, including:

[0111] Obtain a video to be processed, and preprocess the video to be processed to obtain a video frame sequence;

[0112] For each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into the trained video diffusion model for forward noise addition to obtain the latent features of the video with noise;

[0113] Input the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain clear video frames;

[0114] A clear video is generated based on the respective clear video frames.

[0115] In a possible implementation, the video frame and random noise are input into a trained video diffusion model to obtain latent features of the video with noise, including:

[0116] Input the video frame into the trained video diffusion model to obtain the video latent features;

[0117] The random noise is input into the trained video diffusion model. The trained video diffusion model will mix the random noise with the video latent features through the forward diffusion process to obtain the video latent features with noise.

[0118] In one possible implementation, the training process of the denoising VAE decoder includes:

[0119] Obtaining a first training video; wherein the first training video is a video with a clear image;

[0120] Input the first training video into the VAE encoder to obtain a feature map;

[0121] Add random noise to the feature map;

[0122] Input the feature map with random noise into the VAE decoder to obtain the reconstructed video;

[0123] By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

[0124] In a possible implementation, the training process of the video diffusion model includes:

[0125] Get the second training video;

[0126] Inputting the second training video into the video diffusion model to be trained to obtain a predicted video;

[0127] The video diffusion model to be trained is trained based on the predicted video and the second training video. If a preset training end condition is met, a trained video diffusion model is obtained.

[0128] In a possible implementation, generating a clear video based on each clear video frame includes:

[0129] The clear video frames are spliced ​​in chronological order to generate a clear video.

[0130] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0131] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0132] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0133] The electronic device provided in this embodiment may be Figure 5 The electronic device shown in FIG. 1 may perform the following steps: Figure 1 All steps of the video generation method are implemented Figure 1 For details, please refer to the technical effect of the method for generating the video shown in Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0134] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0135] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned video generation method is implemented.

[0136] The processor is used to execute the video generation program stored in the memory to implement the steps of the following video generation method:

[0137] Obtain a video to be processed, and preprocess the video to be processed to obtain a video frame sequence;

[0138] For each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into the trained video diffusion model for forward noise addition to obtain the latent features of the video with noise;

[0139] Input the latent features of the noisy video into the trained denoising VAE decoder for denoising decoding to obtain clear video frames;

[0140] A clear video is generated based on the respective clear video frames.

[0141] In a possible implementation, the video frame and random noise are input into a trained video diffusion model to obtain latent features of the video with noise, including:

[0142] Input the video frame into the trained video diffusion model to obtain the video latent features;

[0143] The random noise is input into the trained video diffusion model. The trained video diffusion model will mix the random noise with the video latent features through the forward diffusion process to obtain the video latent features with noise.

[0144] In one possible implementation, the training process of the denoising VAE decoder includes:

[0145] Obtaining a first training video; wherein the first training video is a video with a clear image;

[0146] Input the first training video into the VAE encoder to obtain a feature map;

[0147] Add random noise to the feature map;

[0148] Input the feature map with random noise into the VAE decoder to obtain the reconstructed video;

[0149] By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

[0150] In a possible implementation, the training process of the video diffusion model includes:

[0151] Get the second training video;

[0152] Inputting the second training video into the video diffusion model to be trained to obtain a predicted video;

[0153] The video diffusion model to be trained is trained based on the predicted video and the second training video. If a preset training end condition is met, a trained video diffusion model is obtained.

[0154] In a possible implementation, generating a clear video based on each clear video frame includes:

[0155] The clear video frames are spliced ​​in chronological order to generate a clear video.

[0156] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0158] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for video generation, characterized in that: The method comprises: Obtaining a video to be processed, and preprocessing the video to be processed to obtain a video frame sequence; For each video frame in the video frame sequence, the following steps are performed: inputting the video frame and random noise into a trained video diffusion model to obtain latent features of the video with noise; Input the latent features of the video with noise into the trained denoising VAE decoder for denoising decoding to obtain a clear video frame; A clear video is generated based on each of the clear video frames.

2. The method according to claim 1, characterized in that The step of inputting the video frame and random noise into a trained video diffusion model to obtain latent features of the video with noise includes: Inputting the video frame into a trained video diffusion model to obtain video latent features; The random noise is input into the trained video diffusion model, and the trained video diffusion model mixes the random noise with the video latent features through a forward diffusion process to obtain the video latent features with noise.

3. The method according to claim 1, characterized in that The training process of the denoising VAE decoder includes: Obtaining a first training video; wherein the first training video is a video with a clear image; Inputting the first training video into the VAE encoder to obtain a feature map; adding random noise to the feature map; Input the feature map with random noise into the VAE decoder to obtain a reconstructed video; By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

4. The method according to claim 1, characterized in that: The training process of the video diffusion model includes: Get the second training video; Inputting the second training video into the video diffusion model to be trained to obtain a predicted video; The video diffusion model to be trained is trained based on the predicted video and the second training video, and a trained video diffusion model is obtained if a preset training end condition is met.

5. The method according to claim 1, characterized in that The generating a clear video based on each of the clear video frames comprises: The clear video frames are spliced ​​in time sequence to generate a clear video.

6. A video generation device, characterized in that: The device comprises: An acquisition module is used to acquire a video to be processed and pre-process the video to be processed to obtain a video frame sequence; A noise adding module is used to perform, for each video frame in the video frame sequence: inputting the video frame and random noise into a trained video diffusion model to obtain a latent feature of the video with noise; A denoising decoding module is used to input the latent features of the video with noise into a trained denoising VAE decoder for denoising decoding to obtain a clear video frame; A generating module is used to generate a clear video based on each of the clear video frames.

7. The device according to claim 5, characterized in that The noise adding module is further used to input the video frame into a trained video diffusion model to obtain video latent features; The random noise is input into the trained video diffusion model, and the trained video diffusion model mixes the random noise with the video latent features through a forward diffusion process to obtain the video latent features with noise.

8. The device according to claim 5, characterized in that The training process of the denoising VAE decoder includes: Obtaining a first training video; wherein the first training video is a video with a clear image; Inputting the first training video into the VAE encoder to obtain a feature map; adding random noise to the feature map; Input the feature map with random noise into the VAE decoder to obtain a reconstructed video; By minimizing the loss value between the reconstructed video and the first training video, the network parameters of the VAE decoder are determined to obtain the denoising VAE decoder.

9. The device according to claim 5, characterized in that The training process of the video diffusion model includes: Get the second training video; Inputting the second training video into the video diffusion model to be trained to obtain a predicted video; The video diffusion model to be trained is trained based on the predicted video and the second training video, and a trained video diffusion model is obtained if a preset training end condition is met.

10. The device according to claim 5, characterized in that The generation module is also used to splice the clear video frames in time sequence to generate a clear video.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a video generation program stored in the memory to implement the video generation method according to any one of claims 1 to 5.

12. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the video generation method according to any one of claims 1 to 5.

Images