Diffusion model video generation method based on optical flow information

Abstract

The invention discloses a diffusion model video generation method based on optical flow information. Firstly, videos in an existing data set are preprocessed, and reference image information and optical flow information are obtained; then, a standard Stable Diffusion model is used as a backbone network for video generation; two Unets with the same structure as the Unet in the standard SD model are used for extracting reference image features and optical flow features respectively, the reference image features are fused with the self-attention layer of the Unet of the SD model in a self-attention layer, and the optical flow features are fused with the cross attention layer and the time sequence attention layer of the Unet of the SD model in a time sequence attention layer. Meanwhile, in order to better utilize the optical flow information, the invention provides motion loss to further improve the generation effect of the network. In addition, diversified prompt information is adopted, so that the network can obtain a more stable generation effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of AIGC large models, and mainly relates to image-generated videos and the problem of optimizing the temporal coherence of generated videos, and specifically relates to a method for optimizing the temporal coherence of generated videos based on an optical flow information diffusion model. Background Art

[0002] There are usually many ways to study video generation, mainly including methods based on GAN networks and methods based on diffusion models. Diffusion model methods have great advantages in the quality of generated videos compared to GAN-based methods. The generation method based on diffusion models mainly includes the following structure: using the Stable Diffusion (SD) model as the generated backbone network, including VAE, Unet and other models, and using the CLIP model to extract text condition information, or using ControlNet or Unet to extract the corresponding image condition information. In the task of generating videos, a major difficulty lies in how to optimize the temporal coherence of the generated videos. The current mainstream solution is to add a temporal attention module to the Stable Diffusion architecture to constrain the temporal coherence of the generated videos. However, such a solution mainly considers the global feature information and ignores local changes, so the details of the generated video will still be incoherent. If it is on a dataset with more details, this problem will be more obvious. Summary of the invention

[0003] The purpose of the present invention is to provide a method for generating a diffusion model video based on optical flow information in order to overcome the shortcomings of the existing methods. The present invention solves the problem of poor temporal coherence of the existing diffusion model generated video.

[0004] The present invention first obtains the original video in the original data set, then captures the first frame of the video as image condition information, and uses the existing optical flow algorithm to extract the optical flow features between the previous and next frames of the video as condition information for optimizing temporal continuity.

[0005] Then, a Stable Diffusion model is used as the backbone network for video generation, and two Unets with the same structure as the Unet in Stable Diffusion are used to extract reference image information and optical flow information respectively, and the features of the Stable Diffusion model are fused at the self-attention layer and the temporal attention layer respectively, and motion loss is proposed to constrain the temporal coherence of the generated video. In the present invention, by combining optical flow information and motion loss, the temporal coherence of the generated video is further optimized, and the quality of the generated video is improved.

[0006] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0007] Step 1: Preprocess the videos in the existing dataset and select the first frame of the video as the reference image. Use the optical flow algorithm to extract the optical flow information between the previous and next frames of the video. Extract the continuous frames of the video into pictures and store them to obtain a video sequence. Adjust the resolution of all pictures to adapt to subsequent training and reasoning.

[0008] Step 2: Construct video generation network OFDiffusion;

[0009] The standard Stable Diffusion model is used as the backbone network for video generation, and a temporal attention layer is added after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video. Two Unets with the same structure as the Unet in the standard SD model (without the temporal attention layer) are used to extract reference image features and optical flow features respectively. The reference image features are fused with the self-attention layer of the SD model Unet in the self-attention layer, and the optical flow features are fused with the temporal attention layer of the SD model Unet in the cross-attention layer. At the same time, the overall model is called OFDiffusion, and motion loss is proposed to constrain the temporal coherence of the generated video.

[0010] Step 3: Train the video generation network OFDiffusion based on the preprocessed data;

[0011] Combined with the reference image information and optical flow information, the parameters are updated using Diffusion loss and motion loss backpropagation.

[0012] Step 4: Use the trained model for inference to obtain the video generated by the model inference and evaluate the effect of the video.

[0013] Furthermore, the specific method of step 2 is as follows:

[0014] In the video generation network, in order to make the generated video meet the constraints of conditional information (including reference image information and optical flow information), a bypass network is used to extract the features of the conditional information and perform feature fusion with the backbone video generation network. Specifically, the backbone network uses the Stable Diffusion model to generate videos, and adds a temporal attention layer after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video. Stable Diffusion contains a variational encoder (VAE) to map the input image to a latent space to reduce the computational complexity of the network. VAE consists of an encoder ε and a decoder Given an image x, VAE maps it into the latent space using the encoder ε(x) and the decoder Decode the encoded result.

[0015] For the reference image information, a Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Reference Encoder, and the features of each self-attention layer of the Reference Encoder are fused with the features of the self-attention layer of each corresponding position of the Unet in the SD model. Specifically, the features of the self-attention layer input of the Unet in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width and the number of channels respectively, and the input feature of the self-attention layer in ReferenceEncoder is represented as Therefore, the way to fuse the features of Reference Encoder and Unet in the SD model can be expressed as:

[0016]

[0017] Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ];

[0018] where [x sd1 ,x ref ] represents the concatenation of features of SD and ReferenceEncoder, W Q ,W k ,W V Respectively represent the weight matrices of Query, Key, and Value, d represents the number of feature channels, and Self_Attn indicates that self-attention is calculated here. After completing the calculation of self-attention, the first half of the feature is obtained as the output feature of the Unet self-attention layer of the SD model to continue the subsequent network propagation.

[0019] For optical flow information, another Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Flow Controller. The features of each cross attention layer of Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the Unet in the SD model. Specifically, the input features of the Unet temporal attention layer in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width and the number of channels respectively, and the features output by the cross attention layer in FlowController are represented as The way in which the features of FlowController and the features of Unet in the SD model are fused can be expressed as:

[0020]

[0021] Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,s flow ],V=W V [x sd2 ,x flow ];

[0022] where [x sd2 ,x flow ] represents the feature concatenation of SD and FlowController, W Q ,W k ,W V Represent the weight matrices of Query, Key, and Value respectively, d represents the number of feature channels, and Temp_Attn indicates that temporal attention is calculated here. After the temporal attention is calculated, the first half of the features are obtained as the output features of the Unet temporal attention of the SD model to continue the subsequent network propagation. At the same time, Diffusion loss and motion loss are used to constrain the generation effect of the network.

[0023] Furthermore, the specific method of step 3 is as follows:

[0024] Given an image x, SD will learn to denoise the feature z = ε(x). During the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict the range from z0 to z t The noise ∈ added in , uses the mean square error to constrain the predicted noise and the original noise, that is, the Diffusion loss. Specifically:

[0025]

[0026] where ∈ is the noise added according to time step t, ∈ θ represents the Unet with learnable parameters θ (i.e., the Unet in the SD model), and c represents the conditional information (i.e., the reference image information and the optical flow information). In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ added t Remove the noise predicted by the network and record the obtained features as but:

[0027]

[0028] in, is a hyperparameter related to the time step t.

[0029] When the input video sequence has f consecutive frames, a feature sequence is obtained. The feature difference of the previous and next frames is defined as the motion that occurs before and after, and the motion loss can be expressed as:

[0030]

[0031] in represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence. Then the final optimization goal can be expressed as:

[0032]

[0033] Where λ1 and λ2 are custom hyperparameters.

[0034] During the training process, the feature representation of the original image in the latent space is obtained through the SD VAE. Random Gaussian noise ∈ is added to this feature based on time step t as the input of the Unet in the SD model. The reference image information is used as the input of the Reference Encoder. After obtaining the reference image features, the features are fused with the Unet of the SD model. The optical flow information is used as the input of the Flow Controller. After obtaining the optical flow features, the features are fused with the Unet of the SD model. The Unet of the SD model will finally predict the added Gaussian noise. The OFDiffusion model is back-propagated with the predicted noise and the noise ∈ added according to time step t before training combined with the Diffusion loss and motion loss to update the model parameters.

[0035] Furthermore, the specific method of step 4 is as follows:

[0036] In the model inference stage, a random Gaussian noise needs to be provided to the OFDiffusion model. The model will de-noise the features step by step according to the provided condition information and time step, and finally obtain the generated image. Since it is a generated video, a series of random Gaussian noises are provided to the model. The model de-noises each noise separately, and finally obtains an image sequence. Sampled at the set fps, the generated video can be obtained.

[0037] The beneficial effects of the present invention are as follows:

[0038] In order to solve the problem of poor temporal coherence of traditional videos generated based on diffusion models, this paper proposes a method to optimize temporal coherence based on optical flow information and motion loss. First, the optical flow information between adjacent frames is obtained using an optical flow extraction algorithm based on the data set. Then, the Flow Controller is used to extract the optical flow information and perform feature fusion with StableDiffusion. At the same time, in order to better utilize the optical flow information, we proposed motion loss to further improve the generation effect of the network. In addition, we also used a variety of prompt information to enable the network to obtain a more stable generation effect. Our method has achieved excellent results on the TikTok dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the overall network structure adopted by an embodiment of the present invention.

[0040] Figure 2 The generation effect of the present invention is compared with that of the mainstream solution on the TikTok dataset, as well as the effect comparison of the ablation experiment. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0042] The embodiment uses the TikTok dataset for experiments. The main task is to generate a video of the person in the reference picture doing a specified action based on a reference picture of the person and a series of character joint point actions, and add optical flow information to optimize the temporal coherence of the generated video. On the basis of the necessary data mentioned above, this embodiment also needs to obtain additional character joint point sequence information.

[0043] Step 1: Split the videos in the dataset (TikTok) into image sequences according to the video frames. Use the first frame of the video as the reference image information. Use the optical flow extraction algorithm to directly extract the optical flow information of all adjacent frames in the video and save it in the form of a data file. Use joint point extraction networks such as OpenPose or DensePose to extract and model the body movements of the characters in the video, and save them in the form of joint point images, which are called character posture information. Finally, change the resolution of all preprocessed images to 512*512 to facilitate subsequent training.

[0044] Step 2: Select the Stable Diffusion v1.5 model as the backbone model of the video generation network, and add a temporal attention layer after each cross-attention layer in the Unet of the SD model to optimize the temporal coherence of the generated video. Stable Diffusion includes a variational encoder (VAE) and a Unet. The VAE maps the input image to a latent space to reduce the computational complexity of the network. VAE includes an encoder ε and a decoder Given an image x, VAE uses the encoder ε(x) to map it into the latent space and the decoder Decode the encoded result. Unet will receive the features encoded by VAE, and after subsequent training, it can complete the video generation task in this embodiment. The Unet of the standard Stable Diffusion model is mainly composed of many spatial self-attention layers (SpatialAttention Layer) and cross-attention layers (CrossAttention Layer) repeatedly stacked, in which the corresponding downsampling layers and upsampling layers are connected by jump connections to ensure the diversity of features. Since the task in this embodiment involves video generation, we add an additional temporal attention layer (TemporalAttention Layer) after the cross-attention layer in the Unet of the standard SD model, such as Figure 1 shown.

[0045] For the reference image information, a Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Reference Encoder, and the features of each self-attention layer of the Reference Encoder are fused with the features of the self-attention layer of each corresponding position of the Unet in the SD model. Specifically, the features of the self-attention layer input of the Unet in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width and the number of channels respectively, and the input feature of the self-attention layer in ReferenceEncoder is represented as Therefore, the way to fuse the features of Reference Encoder and Unet in the SD model can be expressed as:

[0046]

[0047] Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ];

[0048] where [x sd1 ,x ref ] represents the concatenation of features of SD and ReferenceEncoder, W Q ,W k ,W V Respectively represent the weight matrices of Query, Key, and Value, d represents the number of feature channels, and Self_Attn indicates that self-attention is calculated here. After completing the calculation of self-attention, the first half of the feature is obtained as the output feature of the Unet self-attention layer of the SD model to continue the subsequent network propagation.

[0049] For optical flow information, another Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Flow Controller, and the features of each cross attention layer of the Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the Unet in the SD embodiment. Specifically, we record the input features of the Unet temporal attention layer in the SD model in the embodiment as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width and the number of channels respectively, and the features output by the Flow Controller cross attention layer are expressed as The way in which the optical flow features are fused with the features of Unet in the SD model can be expressed as:

[0050]

[0051] Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,x flow ],V=W V [x sd2 ,x flow ];

[0052] where [x sd2 ,x flow ] represents the feature concatenation of SD and FlowController, W Q ,W k ,W V Respectively represent the weight matrices of Query, Key, and Value, d represents the number of feature channels, and Temp_Attn indicates that temporal attention is calculated here. After completing the calculation of temporal attention, we obtain the first half of the features as the output features of the Unet temporal attention layer of the SD model to continue the subsequent network propagation.

[0053] For character posture information, ControlNet is used for feature extraction. ControlNet is a copy of the downsampling part of Unet in the standard SD model (including model structure and weight parameters). The features of character posture information extracted by ControlNet are fused through the jump connection part of Unet in the SD model.

[0054] Step 3: Train the video generation network based on the preprocessed data, calculate the loss function and perform back propagation.

[0055] Given an image x, SD will learn to denoise the feature z = ε(x). During the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict the range from z0 to z t The noise ∈ added in , uses the mean square error to constrain the predicted noise and the original noise, that is, the Diffusion loss. Specifically:

[0056]

[0057] where ∈ is the noise added according to time step t, ∈ θ represents the Unet with learnable parameters θ (i.e., the Unet in the SD model), and c represents the conditional information (i.e., reference image information, optical flow information, and character pose information). In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ added tRemove the noise predicted by the network and record the obtained features as but:

[0058]

[0059] in, is a hyperparameter related to the time step t.

[0060] When the input video sequence has f consecutive frames, a feature sequence is obtained. The feature difference of the previous and next frames is defined as the motion that occurs before and after, and the motion loss can be expressed as:

[0061]

[0062] in represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence. Then the final optimization goal can be expressed as:

[0063]

[0064] Where λ1 and λ2 are custom hyperparameters. In this embodiment, λ1=1 and λ2=0.5 are specified.

[0065] During the training process, the feature representation of the original image in the latent space is obtained through the SD VAE. Random Gaussian noise is added to this feature based on the time step t as the input of the Unet in the SD model. The reference image information is used as the input of the Reference Encoder. After obtaining the reference image features, the features are fused with the Unet of the SD model. The optical flow information is used as the input of the Flow Controller. After obtaining the optical flow features, the features are fused with the Unet of the SD model. The character posture information is used as the input of the ControlNet. After obtaining the character posture features, the features are fused with the Unet of the SD model. The Unet of the SD model will finally predict the added Gaussian noise, and combine the Diffusion loss and motion loss to perform gradient backpropagation and model parameter update on the OFDiffusion overall model.

[0066] Step 4: Use the trained model for inference to obtain the video generated by the model inference, and evaluate the effect of video generation through indicators such as PSNR, SSIM, LPIPS, L1, and FVD.

[0067] In the model inference stage, the input of the Unet of the SD model is random Gaussian noise, the input of the Reference Encoder is the reference image information, the input of the ControlNet is the character posture information, and the input of the Flow Controller is the optical flow information. After the OFDiffusion model is inferred, we will get a series of pictures generated by the model. For each generated picture, we will calculate the picture-level evaluation indicators (PSNR, SSIM, LPIPS, L1) with the labeled picture. For a series of continuous pictures, we will generate a video at 30fps and calculate the video-level evaluation indicators (FVD) with the labeled video.

[0068] Figure 2 The figure shows the comparison of the video quality generated by our method and other currently effective methods, where Reference Image represents the input reference image information, MagicAnimate is the currently effective method, Ours is our method, and GT represents the true label information of the original video.

[0069] Indicator explanation:

[0070] PSNR: refers to the peak signal-to-noise ratio, which is defined based on MSE (mean square error). The higher the calculated value, the higher the image quality. Specifically, for an m×n original image I and a noisy image K,

[0071]

[0072] Among them, MAX is the maximum pixel value of the image, and the unit of PSNR is dB. The larger the value, the less image distortion.

[0073] SSIM: It is a perception-based model that considers image degradation as a perceptual change in structural information, ranging from 0 to 1. The larger the value, the higher the image quality.

[0074]

[0075] LPIPS: A perceptual loss based on neural network implementation that measures the difference between two images and the perceptual similarity between two images. The smaller the value, the higher the image quality.

[0076] L1: L1 loss, also known as MAE loss, the smaller the value, the better the image quality.

[0077] FVD: is a video quality assessment metric based on FID. It uses Inflated-3D Convnets (I3D) pre-trained on Kinetics to extract features from video clips. The FVD score is then calculated using a combination of the mean and covariance matrices.

[0078] Our experimental results on the TikTok dataset are shown in Table 1. The first column is the current method with relatively good results, and the last row Ours represents our proposed method. Except for the method in the first column, the first four columns are image indicators, and the last column is video indicators. Our video indicator FVD is much lower than the current best method, and the SSIM and L1 image indicators also achieve the best results. The other two image indicators are also basically on par with other sota methods.

[0079]

[0080] Table 1

[0081] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0082] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A diffusion model video generation method based on optical flow information, characterized in that: The steps include: Step 1: Preprocess the videos in the existing dataset and select the first frame of the video as the reference image information; use the optical flow algorithm to extract the optical flow information between the previous and next frames of the video; extract the continuous frames of the video into pictures and store them to obtain the video sequence; adjust the resolution of all pictures to adapt to subsequent training and reasoning; Step 2: Construct video generation network OFDiffusion; The standard Stable Diffusion model is used as the backbone network for video generation, and a temporal attention layer is added after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video. Two Unets with the same structure as the Unet in the standard SD model are used to extract reference image features and optical flow features respectively. The reference image features are fused with the self-attention layer of the SD model Unet in the self-attention layer, and the optical flow features are fused with the temporal attention layer of the SD model Unet in the cross-attention layer. At the same time, the overall model is called OFDiffusion, and motion loss is proposed to constrain the temporal coherence of the generated video. Step 3: Train the video generation network OFDiffusion based on the preprocessed data; Combine the reference image information and optical flow information, and use the Diffusion loss and motion loss back propagation to update the parameters; Step 4: Use the trained model for inference to obtain the video generated by the model inference and evaluate the effect of the video.

2. The method for generating a diffusion model video based on optical flow information according to claim 1, characterized in that: The specific method of step 2 is as follows: In the video generation network, in order to make the generated video meet the constraints of conditional information, the conditional information includes reference image information and optical flow information, a bypass network is used to extract the features of the conditional information and perform feature fusion with the backbone video generation network; specifically, the backbone network uses the Stable Diffusion model to be responsible for video generation, and adds a temporal attention layer after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video; Stable Diffusion contains a variational encoder (VAE) to map the input image to a latent space to reduce the computational complexity of the network; VAE includes an encoder ε and a decoder Given an image x, VAE maps it into the latent space using the encoder ε(x) and the decoder Decode the encoded result; For the reference image information, a Unet with the same structure as the Unet in the standard SD model is used for feature extraction. This network is called Reference Encoder. The features of each self-attention layer of Reference Encoder are fused with the features of the self-attention layer of each corresponding position of Unet in the SD model. Specifically, the features of the self-attention layer input of Unet in the SD model are recorded as Where b, f, h, w, and c represent the batch size, the number of consecutive frames, the feature height, the feature width, and the number of channels, respectively. The input feature of the self-attention layer in the Reference Encoder is represented as Therefore, the way to fuse the features of Reference Encoder and Unet in the SD model can be expressed as: Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ]; where [x sd1 , x ref ] represents the concatenation of SD and Reference Encoder features, W Q , W k , W V Respectively represent the weight matrices of Query, Key, and Value, d represents the number of feature channels, and Self_Attn represents self-attention. After completing the calculation of self-attention, the first half of the feature is obtained as the output feature of the Unet self-attention layer of the SD model to continue the subsequent network propagation. For optical flow information, another Unet with the same structure as the Unet in the standard SD model is used for feature extraction. This network is called Flow Controller. The features of each cross attention layer of Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the Unet in the SD model. Specifically, the input features of the Unet temporal attention layer in the SD model are recorded as Where b, f, h, w, c represent batch size, number of consecutive frames, feature height, feature width and number of channels respectively, and the features output by the cross attention layer in the Flow Controller are represented as The way in which the features of the Flow Controller and the features of the Unet in the SD model are fused can be expressed as: Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,x flow ],V=W V [x sd2 ,x flow ]; where [x sd2 , x flow ] represents the feature concatenation of SD and Flow Controller, W Q , W k , W V Respectively represent the weight matrices of Query, Key, and Value, d represents the number of feature channels, and Temp_Attn represents temporal attention. After completing the calculation of temporal attention, the first half of the features are obtained as the output features of the Unet temporal attention of the SD model to continue the subsequent network propagation. At the same time, Diffusion loss and motion loss are used to constrain the generation effect of the network.

3. The method for generating diffusion model video based on optical flow information according to claim 2, characterized in that: The specific method of step 3 is as follows: Given an image x, SD will learn to denoise the feature z = ε(x); during the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict from z0 to z t The noise ∈ added in , the predicted noise and the original noise are constrained by the mean square error, that is, the Diffusion loss; specifically: where ∈ is the noise added according to time step t, ∈ θ represents the Unet with learnable parameters θ, i.e., the Unet in the SD model, and c represents the conditional information. In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ added t Remove the noise predicted by the network and record the obtained features as but: in, is a hyperparameter related to the time step t; When the input video sequence has f consecutive frames, a feature sequence is obtained. The feature difference of the previous and next frames is defined as the motion that occurs before and after, and the motion loss can be expressed as: in represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence; then the final optimization goal can be expressed as: Where λ1 and λ2 are custom hyperparameters; During the training process, the feature representation of the original image in the latent space is obtained through the SD's VAE, and random Gaussian noise ∈ is added to this feature based on the time step t as the input of the Unet in the SD model; the reference image information is used as the input of the ReferenceEncoder, and after obtaining the reference image features, the features are fused with the Unet of the SD model; the optical flow information is used as the input of the Flow Controller, and after obtaining the optical flow features, the features are fused with the Unet of the SD model; the Unet of the SD model will finally predict the added Gaussian noise, and the OFDiffusion model will be gradient backpropagated and model parameters updated through the predicted noise and the noise ∈ added according to the time step t before training combined with the Diffusion loss and motion loss.

4. The method for generating diffusion model video based on optical flow information according to claim 1, characterized in that: The specific method of step 4 is as follows: During the model inference stage, a random Gaussian noise needs to be provided to the OFDiffusion model. The model will denoise the features step by step according to the provided conditional information and time step, and finally obtain the generated image. Since the video is generated, a series of random Gaussian noises are provided to the model. The model denoises each noise separately, and finally obtains an image sequence. After sampling at the set fps, the generated video can be obtained.

Images