Video generation consistency optimization method and system based on diffusion noise iteration

Abstract

The invention discloses a video generation consistency optimization method and system based on diffusion noise iteration, and relates to the field of video generation and video editing. The problem that in the video generation process of an existing video diffusion large model, due to the fact that sampling relevance of all frames is missing and multi-modal guide data is complex and diversified, semantics and time sequences of all frames of the video are inconsistent is solved. According to the method disclosed by the invention, the noise is predicted by combining condition information at each time step according to a preset time step by starting from a pure noise sequence, so that video frames which are clearer and more coherent in content and space-time dimension are gradually generated. And the video diffusion model samples the video tensor obtained in the last time step of the denoising process, and the video tensor is decoded by a decoder to obtain a visual video result. According to the method, the video frame consistency is improved by taking the video tensor obtained in the last time step as a starting point. The method is also suitable for the application fields of video content output and the like realized by using a generated large model diffusion model as a basic framework.

Description

Technical Field

[0001] This invention relates to the field of video generation and video editing technology, specifically to a video generation consistency optimization method and system based on iterative diffusion noise. Background Technology

[0002] Diffusion models, as the most advanced deep generative models, are widely used in the creation of various challenging AIGC (Artificial Intelligence Generated Content) projects. The creation process has evolved from simple text-guided creation to one guided by multimodal data including text, images, and video, becoming a powerful engine for film and television production and personal creative expression. Video content creation based on diffusion models, a core task in this field, enables switching between various roles, scenes, and functions, allowing for a comprehensive and intelligent reconstruction of video content from its underlying semantics to its visual presentation. This brings unprecedented flexibility and creativity to content creation.

[0003] However, while obtaining rich and diverse video content, problems such as video flickering and unevenness are often encountered, especially in conditional video generation. Due to the complexity and diversity of multimodal data (such as text, images, and audio), semantic information recognition errors and inconsistencies between video frames are more pronounced. These challenges not only affect the visual coherence of the generated video but also limit the performance of the model in practical applications. Therefore, in-depth research into methods to improve the semantic and temporal consistency of video frames has become a crucial problem urgently needing to be solved in the current field of video generation.

[0004] The video sampling process based on the diffusion model can gradually complete denoising under the guidance of various content. Since the initial noise during sampling is random noise, and the sampling process for each frame of the video is relatively independent, there is a lack of sampling correlation between frames. Therefore, while the generated video frames have a certain degree of alignment with the guiding content, inconsistencies exist in style, shape, attributes, and motion. To generate clear, smooth, motion-continuous, and semantically consistent video content, it is necessary to enhance the information interaction between video frames, break the original order of the frame sequence, and extract the key features of each frame. Summary of the Invention

[0005] This invention addresses the problems of inconsistent semantics and temporal sequence among video frames caused by the lack of correlation between sampled frames and the complexity and diversity of multimodal guiding data in existing video diffusion models. To this end, this invention proposes a video generation consistency optimization method and system based on iterative diffusion noise.

[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Option 1: This invention proposes a video generation consistency optimization method based on iterative diffusion noise, the method comprising the following steps: Step 1: Obtain the video tensor from the last time step of the video diffusion model sampling denoising process. Through expressions Perform a proximity-based shuffle to obtain the shuffled video tensor. ; Step 2: Obtain the shuffled video tensor from Step 1. Through expression Perform cross-association shuffling to obtain the cross-association shuffled video tensor. ; Step 3: Add random noise to the video tensor obtained in Step 2 using the DDPM Forward noise addition method. In the process, the video noise tensor is obtained. and the video noise tensor and random Gaussian noise Transformed to the frequency domain via Fourier transform; Step 4: For the video noise tensor converted to the frequency domain in Step 3, use the expression... Perform Gaussian low-pass filtering; Step 5: For the random Gaussian noise transformed to the frequency domain in Step 4, use the expression... Perform a Gaussian high-pass filter, corresponding to the low-pass filter; Step 6: Add the results obtained in Steps 4 and 5 element by element, perform an inverse Fourier transform on the sum to convert it to the spatial domain, and obtain the noise tensor. ; with noise tensor The initial noise is sampled for the video diffusion model denoising process, and the next round of sampling denoising process is performed to form the initial noise iteration process; Step 7: After completing the last iteration, use the initial noise obtained in the last iteration as the starting point to perform the last sampling and denoising operation of the video diffusion model, and obtain the final video tensor at the last time step. Step 8: Use the decoder to decode the video tensor obtained in Step 7 to obtain a semantically and temporally consistent visualized video result.

[0007] Furthermore, a preferred embodiment is provided, wherein in step one, the expression is used... Perform a proximity-based shuffle to obtain the shuffled video tensor. The method is as follows: The adjacent association shuffling refers to each adjacent frame starting from the first frame of the video. Frames are grouped together, and then randomly shuffled within each group to obtain the shuffled video tensor. ; expression :

[0008] in, For each consecutive frame starting from the first frame position The index of the frame.

[0009] Furthermore, a preferred embodiment is provided, in which step two uses the shuffled video tensor obtained in step one. Through expression The method for cross-related shuffling is as follows: expression :

[0010] in, For from the first Frame position starts every consecutive The index of the frame.

[0011] Furthermore, a preferred embodiment is provided, wherein the video noise tensor mentioned in step three... and random Gaussian noise Random Gaussian noise tensors with the same number of frames.

[0012] Furthermore, a preferred embodiment is provided, in step four, the video noise tensor converted to the frequency domain in step three is expressed by the expression... The method for performing Gaussian low-pass filtering is as follows: expression :

[0013] in, This represents the total number of frames in the video. Indicates the position of the video frame. For frame position index, These are the fundamental coefficients for low-frequency information in each frame of the video. This is the cutoff frequency of the filter. The distance from the center position, , , For the high frame of the video, The width of the video frame.

[0014] Furthermore, a preferred embodiment is provided, in step five, the random Gaussian noise converted to the frequency domain in step four is expressed by the expression... Gaussian high-pass filtering is performed using the method corresponding to low-pass filtering: expression : .

[0015] Furthermore, a preferred embodiment is provided, in which step six further includes setting the number of initial noise iterations based on the complexity of the generation task and the sampling algorithm used, wherein each iteration includes a sampling denoising process and the initial noise acquisition process of steps one to five.

[0016] Option 2: A video generation consistency optimization system based on iterative diffusion noise, the system comprising: The nearest-neighbor shuffling module is used to process the video tensor obtained at the last time step of the video diffusion model sampling and denoising process. Through expressions Perform a proximity-based shuffle to obtain the shuffled video tensor. ; The cross-association shuffling module is used to obtain the shuffled video tensor from the adjacent association shuffling module. Through expression Perform cross-association shuffling to obtain the cross-association shuffled video tensor. ; The conversion module is used to add random noise to the video tensor obtained in step two using the DDPM Forward noise addition method. In the process, the video noise tensor is obtained. and the video noise tensor and random Gaussian noise Transformed to the frequency domain via Fourier transform; The Gaussian low-pass filter module is used to process the video noise tensor converted to the frequency domain in the conversion module using the expression... Perform Gaussian low-pass filtering; The Gaussian high-pass filter module is used to process random Gaussian noise converted to the frequency domain in the Gaussian low-pass filter module, using the expression... Perform a Gaussian high-pass filter, corresponding to the low-pass filter; The update module is used to add the results from the Gaussian low-pass filter module and the Gaussian high-pass filter module element by element, and then perform an inverse Fourier transform on the sum to convert it to the spatial domain, obtaining the noise tensor. ; with noise tensor The initial noise is sampled for the video diffusion model denoising process, and the next round of sampling denoising process is performed to form the initial noise iteration process; The denoising module is used to perform the final sampling and denoising operation of the video diffusion model after the last iteration, starting from the initial noise obtained in the last iteration, and obtain the final video tensor at the last time step. The visualization module is used to decode the video tensor obtained by the denoising module using the decoder, so as to obtain a visual video result that is consistent in semantics and time sequence.

[0017] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0018] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0019] The advantages of this invention are: The video generation consistency optimization method based on diffusion noise iteration described in this invention is used to enhance the feature interaction between frames in the video sampling process, highlight key frames of the video that are rich in semantic content, thereby improving the semantic alignment between the generated video and various guiding content such as text, images, and videos, while also improving the semantic and temporal consistency of each frame of the generated video.

[0020] The video generation consistency optimization method based on diffusion noise iteration described in this invention ensures that the features of each patch in each frame are affected by the patches in neighboring and intersecting frames, thereby guaranteeing the application of the same style globally. At the same time, this invention highlights the semantically rich video keyframes, which helps to improve the semantic alignment between the generated video and various guiding content such as text, images, and videos.

[0021] This invention is also applicable to applications such as video content output that utilizes Diffusion Models as a basic framework. Attached Figure Description

[0022] Figure 1 This is a flowchart of the video generation consistency optimization method based on diffusion noise iteration as described in Implementation Method 1.

[0023] Figure 2 This is a schematic diagram illustrating the video frame shuffling process of the video generation consistency optimization method based on diffusion noise iteration described in Implementation Method 1.

[0024] Figure 3 This is a schematic diagram illustrating the principle of the multi-scale frequency acquisition process of the video generation consistency optimization method based on diffusion noise iteration described in Implementation Method 1. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0026] Implementation Method 1, Reference Figures 1 to 3 This embodiment proposes a video generation consistency optimization method based on iterative diffusion noise. The sampling process of the video diffusion model starts with a pure noise sequence (called the initial noise). Based on preset time steps, at each time step, conditional information (such as text descriptions) is combined to predict the noise, thereby gradually generating clearer and more coherent video frames in both content and spatiotemporal dimensions. The video tensor obtained at the last time step of the video diffusion model's sampling and denoising process is shown. After decoding, the video result is visualized. This invention primarily uses the video tensor obtained at the last time step. Starting with the above, video frame consistency is improved. A video generation consistency optimization method based on iterative diffusion noise includes the following steps: Step 1: Initialize random noise. The sampling process begins with a random noise tensor that follows a Gaussian distribution. This noise tensor is called the initial noise and is represented as follows: Example of initial noise The shape is (16, 3, 512px, 512px), where: 16 is the number of video frames, 3 is the number of channels (RGB), and the two 512px are the height and width of the video frame, respectively; Step 2: Input guiding conditions, which may include modal data such as text, images, and videos, to form a condition vector that guides the entire denoising process, ensuring that the final generated video content conforms to the guiding conditions. In this example, the guiding condition is text. Step 3: Set the number of denoising steps. The model does not achieve denoising in one step; it usually requires... Step-by-step noise reduction, implementation process .set up This is the index of the current denoising step count. The value ranges from 25 to 1.

[0027] Step 4: Noise prediction, from Beginning, i.e., the first Step, reduce initial noise Step Index Together with the conditional vector, it is input into the noise predictor of the video diffusion model; Step 5: Using the noise predicted in Step 4, and combining it with the selected sampler algorithm (such as DDPM, DDIM, etc.; DDPM is selected in this implementation), calculate the noise tensor after noise reduction. ; Step Six: Step, take the noise tensor obtained in the previous time step. Current step index Together with the conditional vector, it is input into the noise predictor of the video diffusion model; Step 7: Using the noise predicted in Step 6, and combining it with the DDPM sampler algorithm, calculate the noise tensor after noise reduction. ; Step 8: Repeat the noise prediction process in Steps 6 and 7 until Step 1, to obtain the denoised video tensor for this round. .

[0028] Step 9: Process the video tensor Through expressions Perform a proximity-related shuffling, the shuffling process is as follows: Figure 2 As shown in Layer 1, set That is, starting from the first frame of the video, every four adjacent frames are grouped together, and the frames within each group are randomly shuffled to obtain the shuffled video tensor. ; expression :

[0029] Step 10: [The following is a separate, unrelated sentence:] The results obtained in Step 9... Through expressions Perform cross-related shuffling, the shuffling process is as follows: Figure 2 As shown in Layer 2, the last two frames of the first four frames of the video are combined with the first two frames of the next group of four frames to form a new combination. The combinations are then randomly shuffled within each group, and this process is repeated to achieve cross-correlation between groups, resulting in the cross-correlation video tensor. ; expression :

[0030] Step 11: As Figure 1 As shown, random noise is added to the video tensor obtained in step ten using the DDPM Forward noise addition method. In the process, the video noise tensor after adding noise is obtained. ; Step 12: As Figure 1 As shown, the video noise tensor is divided into... and random Gaussian noise Transformed to the frequency domain via Fourier transform (FFT); Step 13: For the video noise tensor converted to the frequency domain, use the expression... Perform Gaussian low-pass filtering, where the filter cutoff frequency is... ; expression :

[0031] Step Fourteen: For the random Gaussian noise transformed to the frequency domain, use the expression... Perform Gaussian high-pass filtering; expression :

[0032] Step 15: Add the elements of the results obtained in Steps 13 and 14 together; Step 16: Perform an inverse Fourier transform on the sum of elements from Step 15 to convert it to the spatial domain, obtaining the video noise tensor. ; Step 17: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The initial noise used in the next sampling denoising process, i.e., the noise from step one. Replace with ; Step 18: With As initial noise, steps one through seventeen are performed to iterate through the video diffusion model sampling and denoising to obtain the initial noise. This example involves a total of 5 iterations. Step 19: After completing the last iteration, using the initial noise obtained in the last iteration as the starting point, perform the last sampling and denoising operation of the video diffusion model, and obtain the final video tensor at the last time step; Step 20: Decode the video tensor obtained in Step 19 using a decoder to obtain a visualized video result with semantic and temporal consistency. The video result becomes clearer and more continuous while meeting the guidance conditions.

[0033] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or technical solutions of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0034] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended technical solutions are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present invention. Clearly, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for video generation consistency optimization based on diffusion noise iteration, characterized in that, The method comprises the following steps: Step one, sampling the video tensor obtained in the last time step of the video diffusion model sampling denoising process , by expression , to obtain a shuffled video tensor ; Step two, shuffle the video tensor obtained in step one By expression Cross-correlation shuffling is performed to obtain a cross-correlation shuffled video tensor ; Step three, adding random noise to the video tensor obtained in step two by DDPM Forward forward noise adding method In the middle, get the video noise tensor And the video noise tensor And the random Gaussian noise Convert to frequency domain by Fourier transform; Step four, the video noise tensor converted to the frequency domain in step three is Gaussian low-pass filtered by the expression ; Step five, the random Gaussian noise converted to the frequency domain in step four is Gaussian high-pass filtered, corresponding to the low-pass filtering Step five, the random Gaussian noise converted to the frequency domain in step four is Gaussian high-pass filtered, corresponding to the low-pass filtering Step six, elementally add the results obtained in step four and step five, inverse Fourier transform the addition result, convert to the spatial domain, and obtain a noise tensor ; use the noise tensor as the initial noise of the sampling denoising process of the video diffusion model to perform the next round of sampling denoising process, and form an initial noise iteration process. Step seven, after completing the last iteration, taking the initial noise obtained in the last iteration as a starting point, performing a sampling denoising operation of the last video diffusion model to obtain a final video tensor at the last time step; Step eight, decoding the video tensor obtained in step seven using a decoder to obtain a visualized video result consistent in semantics and timing.

2. The diffusion noise iteration based video generation consistency optimization method of claim 1, wherein, In step one, the expression The method for performing the adjacent correlation shuffle to obtain the shuffled video tensor is: The adjacent correlation shuffles, that is, each adjacent frame from the first frame of the video becomes a group, and the in-group random shuffle is performed to obtain the shuffled video tensor ;​ Expression : wherein, is the index of the first frame in the sequence of frames. is the index of the first frame in the sequence of frames.

3. The diffusion noise based iterative video generation consistency optimization method of claim 1, wherein, The shuffled video tensor obtained in step one is used in step two By the expression The method for cross-correlation shuffling is: Expression : wherein, is the frame position of the first frame of the sequence. The index of the frame is incremented by one for each consecutive frame. frame of the sequence. The index of the frame is incremented by one for each consecutive frame.

4. The diffusion noise based iterative video generation consistency optimization method of claim 1, wherein, the video noise tensor described in step three and a random Gaussian noise a random Gaussian noise tensor with the same number of frames.

5. The diffusion noise based iterative video generation consistency optimization method of claim 1, wherein, In step four, the video noise tensor converted to the frequency domain in step three is subjected to a Gaussian low-pass filtering by the expression The method for Gaussian low-pass filtering is: Expression : wherein, is the total number of frames of the video, represents the frame position of the video, is the frame position index, is the base coefficient of the low frequency information of each frame of the video, is the cut-off frequency of the filter, is the distance from the center position, , , is the height of the frame of the video, is the width of the frame of the video.

6. The diffusion noise based iterative video generation consistency optimization method of claim 1, wherein, In step five, the random Gaussian noise converted to the frequency domain in step four is Gaussian high-pass filtered by the expression corresponding to the low-pass filtering method Expression : .

7. The diffusion noise based iterative video generation consistency optimization method of claim 1, wherein, In step six, there is also a step of setting the number of iterations of the initial noise according to the complexity of the generation task and the sampling algorithm used, each iteration including a sampling denoising process and the initial noise obtaining process of steps one to six.

8. A video generation consistency optimization system based on diffusion noise iteration, characterized in that, The system comprises: a nearest neighbor shuffle module for shuffling the video tensor resulting from sampling the last time step of the video diffusion model denoising process by the expression to obtain a shuffled video tensor ; cross-correlation shuffling module, for shuffling the video tensor obtained by the adjacent correlation shuffling module by the expression cross-correlation shuffling, to obtain a cross-correlation shuffled video tensor ; a conversion module for adding random noise to the video tensor obtained at step two by DDPM Forward method In this way, a video noise tensor is obtained and the video noise tensor and a random Gaussian noise is converted into the frequency domain by Fourier transform a Gaussian low-pass filter module for applying a Gaussian low-pass filter to the video noise tensor converted to the frequency domain in the conversion module by the expression a Gaussian low-pass filter module for applying a Gaussian low-pass filter to the video noise tensor converted to the frequency domain in the conversion module by the expression a Gaussian high-pass filtering module for filtering the random Gaussian noise converted to the frequency domain in the Gaussian low-pass filtering module by the expression Gaussian high-pass filtering is performed for correspondence with the low-pass filtering; An updating module is configured to perform element addition on the results obtained by the Gaussian low-pass filtering module and the Gaussian high-pass filtering module, perform inverse Fourier transform on the addition result, convert to a spatial domain, and obtain a noise tensor ; the noise tensor is used as initial noise for sampling and denoising process of a video diffusion model to perform a next round of sampling and denoising process, thereby forming an initial noise iteration process . a denoising module for, after completing the last iteration, taking the initial noise obtained in the last iteration as a starting point, performing a sampling denoising operation of the last video diffusion model to obtain a final video tensor at the last time step; a visual display module for decoding the video tensor obtained by the denoising module using a decoder to obtain a visualized video result consistent in semantics and timing.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the method of any one of claims 1-7.

10. A computer device, comprising: comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images