A diffusion model video generation method based on optical flow information

Abstract

The present invention discloses a diffusion model video generation method based on optical flow information. First, the video in the existing data set is preprocessed to obtain reference image information and optical flow information; then the standard Stable Diffusion model is used as the backbone network for video generation; two Unets with the same structure as the Unet in the standard SD model are used to extract reference image features and optical flow features respectively, and the reference image features are fused with the self-attention layer of the SD model Unet in the self-attention layer, and the optical flow features are fused with the temporal attention layer of the SD model Unet in the cross-attention layer. At the same time, in order to better utilize the optical flow information, the present invention proposes motion loss to further improve the generation effect of the network. In addition, the present invention also adopts diversified prompt information to enable the network to obtain a more stable generation effect.

Description

Technical Field

[0001] The present invention belongs to the field of AIGC large model technology, mainly relates to image-generated videos and the problem of optimizing the temporal coherence of generated videos, and specifically relates to a method for optimizing the temporal coherence of generated videos based on an optical flow information diffusion model. Background Art

[0002] Existing research on video generation generally involves numerous approaches, primarily those based on GAN networks and diffusion models. Diffusion models offer significant advantages over GAN-based approaches in terms of generated video quality. Diffusion-based approaches typically employ the following structure: a Stable Diffusion (SD) model, such as a VAE or Unet, is used as the generative backbone network, with a CLIP model extracting textual conditional information, or a ControlNet or Unet extracting corresponding image conditional information. A major challenge in video generation is optimizing the temporal coherence of the generated videos. The current mainstream approach involves adding a temporal attention module to the Stable Diffusion architecture to constrain temporal coherence. However, this approach primarily considers global feature information while ignoring local variations. Consequently, the generated videos still exhibit incoherent details, a problem that is particularly pronounced when working with datasets rich in detail. Summary of the Invention

[0003] The present invention aims to address the shortcomings of existing methods and provide a method for generating diffusion model videos based on optical flow information. The present invention solves the problem of poor temporal coherence of existing diffusion model-generated videos.

[0004] The present invention first obtains the original video in the original data set, then intercepts the first frame of the video as image condition information, and uses the existing optical flow algorithm to extract the optical flow features between the previous and next frames of the video as condition information for optimizing temporal coherence.

[0005] A Stable Diffusion model is then used as the backbone network for video generation. Two Unets with the same structure as the Unet in Stable Diffusion are used to extract reference image information and optical flow information, respectively. These information is then fused with the Stable Diffusion model's features at the self-attention layer and the temporal attention layer, respectively. A motion loss is also proposed to constrain the temporal coherence of the generated video. By combining optical flow information and motion loss, this method further optimizes the temporal coherence of the generated video, improving its quality.

[0006] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0007] Step 1: Preprocess the videos in the existing dataset and select the first frame of the video as the reference image. Use an optical flow algorithm to extract the optical flow information between the previous and next frames. Extract consecutive frames of the video into images and store them to create a video sequence. Adjust the resolution of all images to accommodate subsequent training and inference.

[0008] Step 2: Build the video generation network OFDiffusion;

[0009] The standard Stable Diffusion model is used as the backbone network for video generation. A temporal attention layer is added after each cross-attention layer in the standard SD model's Unet to optimize the temporal coherence of the generated video. Two Unets with the same structure as the Unet in the standard SD model (without the temporal attention layer) are used to extract reference image features and optical flow features respectively. The reference image features are fused with the self-attention layer of the SD model Unet in the self-attention layer, and the optical flow features are fused with the temporal attention layer of the SD model Unet in the cross-attention layer. At the same time, the overall model is called OFDiffusion, and a motion loss is proposed to constrain the temporal coherence of the generated video.

[0010] Step 3: Train the video generation network OFDiffusion based on the preprocessed data;

[0011] Combining the reference image information and optical flow information, the parameters are updated using diffusion loss and motion loss backpropagation.

[0012] Step 4: Use the trained model for inference to obtain the video generated by the model inference and evaluate the effect of the video.

[0013] Furthermore, the specific method of step 2 is as follows:

[0014] In the video generation network, in order to make the generated video meet the constraints of conditional information (including reference image information and optical flow information), a bypass network is used to extract the features of the conditional information and perform feature fusion with the backbone video generation network. Specifically, the backbone network uses the Stable Diffusion model to be responsible for video generation, and adds a temporal attention layer after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video. Stable Diffusion contains a variational encoder (VAE) to map the input image to a latent space to reduce the computational complexity of the network. VAE consists of an encoder ε and a decoder Given an image x, VAE uses the encoder ε(x) to map it into the latent space and the decoder Decode the encoded result.

[0015] For the reference image information, a Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Reference Encoder, and the features of each self-attention layer of the Reference Encoder are fused with the features of the self-attention layer of each corresponding position of the Unet in the SD model. Specifically, the features of the self-attention layer input of the Unet in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width, and the number of channels, respectively. The input features of the self-attention layer in ReferenceEncoder are represented as Therefore, the fusion method of the features of Reference Encoder and Unet in SD model can be expressed as:

[0016]

[0017] Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ];

[0018] where [x sd1 ,x ref ] represents the concatenation of SD and ReferenceEncoder features, W Q ,W k ,W V where d represents the weight matrices for Query, Key, and Value, respectively. d represents the number of feature channels, and Self_Attn indicates that self-attention is being calculated. After the self-attention calculation is completed, the first half of the features are obtained as the output features of the Unet self-attention layer of the SD model to continue the subsequent network propagation.

[0019] For optical flow information, another Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Flow Controller, and the features of each cross-attention layer of Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the Unet in the SD model. Specifically, the input features of the Unet temporal attention layer in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the height of the feature, the width of the feature, and the number of channels, respectively. The output features of the cross attention layer in the FlowController are represented as The way to fuse the features of FlowController and Unet in the SD model can be expressed as:

[0020]

[0021] Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,s flow ],V=W V [x sd2 ,x flow ];

[0022] where [x sd2 ,x flow ] represents the feature concatenation of SD and FlowController, W Q ,W k ,W V where d represents the weight matrices for query, key, and value, respectively. d represents the number of feature channels, and temp_attn indicates that temporal attention is being calculated. After temporal attention is calculated, the first half of the features are obtained as the output features of the SD model's Unet temporal attention for subsequent network propagation. Diffusion loss and motion loss are also used to constrain the network's generation performance.

[0023] Furthermore, the specific method of step 3 is as follows:

[0024] Given an image x, SD will learn to de-noise the feature z = ε(x). During the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict the range from z0 to z t The noise ∈ added in , uses the mean square error to constrain the predicted noise and the original noise, that is, the Diffusion loss. Specifically:

[0025]

[0026] where ∈ is the noise added according to time step t, ∈ θ Represents the Unet with learnable parameters θ (i.e., the Unet in the SD model), and c represents the conditional information (i.e., the reference image information and the optical flow information). In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ added t Remove the noise predicted by the network and record the obtained features as but:

[0027]

[0028] in, is a hyperparameter related to the time step t.

[0029] When the input video sequence has f consecutive frames, a feature sequence is obtained The feature difference between the previous and next frames is defined as the motion occurring before and after, and the motion loss can be expressed as:

[0030]

[0031] in represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence. The final optimization goal can be expressed as:

[0032]

[0033] Where λ1 and λ2 are custom hyperparameters.

[0034] During training, the SD VAE obtains a feature representation of the original image in latent space. Random Gaussian noise ∈ is added to this feature based on time step t as input to the Unet in the SD model. The reference image information is used as input to the Reference Encoder to obtain the reference image features, which are then fused with the features of the SD model's Unet. The optical flow information is used as input to the Flow Controller to obtain the optical flow features, which are then fused with the features of the SD model's Unet. The SD model's Unet finally predicts the added Gaussian noise. This predicted noise, combined with the noise ∈ added before training based on time step t, is used to perform gradient backpropagation and update the OFDiffusion model's model parameters using the diffusion loss and motion loss.

[0035] Furthermore, the specific method of step 4 is as follows:

[0036] During the inference phase, the OFDiffusion model is fed with random Gaussian noise. The model then de-noises the features step by step based on the provided conditional information and time steps, ultimately generating the generated image. Since the model is generating a video, a series of random Gaussian noises are fed to the model, and the model de-noises each noise individually, ultimately generating a sequence of images. This is then sampled at the set fps to generate the generated video.

[0037] The beneficial effects of the present invention are as follows:

[0038] To address the problem of poor temporal coherence in traditional videos generated based on diffusion models, this paper proposes a method to optimize temporal coherence based on optical flow information and motion loss. First, an optical flow extraction algorithm is used based on the dataset to obtain optical flow information between adjacent frames. Subsequently, a Flow Controller is used to extract the optical flow information and perform feature fusion with StableDiffusion. At the same time, in order to better utilize the optical flow information, a motion loss is proposed to further improve the network's generation effect. In addition, a variety of prompt information is also used to enable the network to achieve a more stable generation effect. Our method has achieved excellent results on the TikTok dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the overall network structure adopted by an embodiment of the present invention.

[0040] Figure 2 This is a comparison of the generation effect of the present invention and the mainstream solution on the TikTok dataset, as well as the effect comparison of the ablation experiment. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0042] This example uses the TikTok dataset for experiments. The main task is to generate a video of the person performing a specific action in the reference image based on a reference image of the person and a series of joint movements. The video is then optimized for temporal coherence by adding optical flow information. In addition to the required data mentioned above, this example also requires obtaining sequence information for the person's joints.

[0043] Step 1: Split the videos in the TikTok dataset into image sequences based on the video frames. Use the first frame of the video as the reference image. Use an optical flow extraction algorithm to directly extract optical flow information from all adjacent frames in the video and save it as a data file. Use a joint point extraction network such as OpenPose or DensePose to extract and model the body movements of the characters in the video. Save it as joint point images, which are called character pose information. Finally, change the resolution of all preprocessed images to 512*512 to facilitate subsequent training.

[0044] Step 2: Select the Stable Diffusion v1.5 model as the backbone model of the video generation network, and add a temporal attention layer after each cross-attention layer in the Unet of the SD model to optimize the temporal coherence of the generated video. Stable Diffusion includes a variational encoder (VAE) and a Unet. The VAE maps the input image to a latent space to reduce the computational complexity of the network. The VAE consists of an encoder ε and a decoder Given an image x, VAE uses the encoder ε(x) to map it into the latent space and the decoder Decode the encoded result. Unet will receive the features encoded by VAE, and after subsequent training, it can complete the video generation task in this embodiment. The Unet of the standard Stable Diffusion model is mainly composed of many spatial self-attention layers (SpatialAttention Layer) and cross-attention layers (CrossAttention Layer) repeatedly stacked together, in which the corresponding downsampling layers and upsampling layers are connected by jumps to ensure the diversity of features. Since the task in this embodiment involves video generation, we have added a temporal attention layer (TemporalAttention Layer) after the cross-attention layer in the Unet of the standard SD model, as shown in the figure. Figure 1 shown.

[0045] For the reference image information, a Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Reference Encoder, and the features of each self-attention layer of the Reference Encoder are fused with the features of the self-attention layer of each corresponding position of the Unet in the SD model. Specifically, the features of the self-attention layer input of the Unet in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width, and the number of channels, respectively. The input features of the self-attention layer in ReferenceEncoder are represented as Therefore, the fusion method of the features of Reference Encoder and Unet in SD model can be expressed as:

[0046]

[0047] Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ];

[0048] where [x sd1 ,x ref ] represents the concatenation of SD and ReferenceEncoder features, W Q ,W k ,W V where d represents the weight matrices for Query, Key, and Value, respectively. d represents the number of feature channels, and Self_Attn indicates that self-attention is being calculated. After the self-attention calculation is completed, the first half of the features are obtained as the output features of the Unet self-attention layer of the SD model to continue the subsequent network propagation.

[0049] For optical flow information, another Unet with the same structure as the Unet in the standard SD model (without the temporal attention layer) is used for feature extraction. This network is called Flow Controller. The features of each cross attention layer of Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the SD Unet in the embodiment. Specifically, we record the features of the Unet temporal attention layer input in the SD model in the embodiment as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width and the number of channels respectively, and the features output by the Flow Controller cross attention layer are expressed as The way to fuse the optical flow features and the features of Unet in the SD model can be expressed as:

[0050]

[0051] Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,x flow ],V=W V [x sd2 ,x flow ];

[0052] where [x sd2 ,x flow ] represents the feature concatenation of SD and FlowController, W Q ,W k ,W V where d represents the weight matrices for Query, Key, and Value, respectively. d represents the number of feature channels, and Temp_Attn indicates that temporal attention is being calculated. After calculating temporal attention, we obtain the first half of the features as the output features of the SD model's Unet temporal attention layer for subsequent network propagation.

[0053] For character pose information, ControlNet is used for feature extraction. ControlNet is a copy of the downsampling portion of the UNet in the standard SD model (including the model structure and weight parameters). The character pose features extracted by ControlNet are then fused through the skip connections of the UNet in the SD model.

[0054] Step 3: Train the video generation network based on the preprocessed data, calculate the loss function and perform backpropagation.

[0055] Given an image x, SD will learn to de-noise the feature z = ε(x). During the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict the range from z0 to z t The noise ∈ added in , uses the mean square error to constrain the predicted noise and the original noise, that is, the Diffusion loss. Specifically:

[0056]

[0057] where ∈ is the noise added according to time step t, ∈ θ Represents the Unet with learnable parameters θ (i.e., the Unet in the SD model), and c represents the conditional information (i.e., reference image information, optical flow information, and character pose information). In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ added tRemove the noise predicted by the network and record the obtained features as but:

[0058]

[0059] in, is a hyperparameter related to the time step t.

[0060] When the input video sequence has f consecutive frames, a feature sequence is obtained The feature difference between the previous and next frames is defined as the motion occurring before and after, and the motion loss can be expressed as:

[0061]

[0062] in represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence. The final optimization goal can be expressed as:

[0063]

[0064] Where λ1 and λ2 are custom hyperparameters. In this embodiment, λ1 = 1 and λ2 = 0.5 are specified.

[0065] During training, the SD VAE is used to obtain the feature representation of the original image in the latent space. Random Gaussian noise is added to this feature based on time step t as the input to the Unet in the SD model. The reference image information is used as the input to the Reference Encoder. After obtaining the reference image features, the features are fused with the Unet of the SD model. The optical flow information is used as the input to the Flow Controller. After obtaining the optical flow features, the features are fused with the Unet of the SD model. The character pose information is used as the input to the ControlNet. After obtaining the character pose features, the features are fused with the Unet of the SD model. The Unet of the SD model finally predicts the added Gaussian noise and combines the Diffusion loss and motion loss to perform gradient backpropagation and model parameter updates on the OFDiffusion overall model.

[0066] Step 4: Use the trained model for inference to obtain the video generated by the model inference, and evaluate the effect of the video generation through indicators such as PSNR, SSIM, LPIPS, L1, and FVD.

[0067] During the model inference phase, the SD model's UNet input is random Gaussian noise, the Reference Encoder input is the reference image information, the ControlNet input is the character pose information, and the Flow Controller input is the optical flow information. After inference of the OFDiffusion model, we will obtain a series of model-generated images. For each generated image, image-level evaluation metrics (PSNR, SSIM, LPIPS, L1) are calculated compared with the labeled image. For a series of continuous images, a video is generated at 30fps and video-level evaluation metrics (FVD) are calculated compared with the labeled video.

[0068] Figure 2 The figure shows a comparison of the video quality generated by our method and other currently effective methods. Reference Image represents the input reference image information, MagicAnimate is the currently effective method, Ours is our method, and GT represents the true label information of the original video.

[0069] Indicator explanation:

[0070] PSNR: refers to the peak signal-to-noise ratio, which is defined based on MSE (mean square error). The higher the calculated value, the higher the image quality. Specifically, for an m×n original image I and a noisy image K,

[0071]

[0072] Where MAX is the maximum pixel value of the image, and the unit of PSNR is dB. The larger the value, the less image distortion.

[0073] SSIM: It is a perception-based model that considers image degradation as a perceptual change in structural information. The value ranges from 0 to 1. The larger the value, the higher the image quality.

[0074]

[0075] LPIPS: A perceptual loss implemented based on a neural network that measures the difference between two images and the perceptual similarity between them. The smaller the value, the higher the image quality.

[0076] L1: L1 loss, also known as MAE loss, the smaller the value, the better the image quality.

[0077] FVD: A video quality assessment metric based on FID. It extracts features from video clips using Inflated-3D Convnets (I3D) pre-trained on Kinetics. The FVD score is then calculated using a combination of the mean and covariance matrices.

[0078] Table 1 shows the experimental results of our approach on the TikTok dataset. The first column represents the currently leading methods, and the last row, "Ours," represents our proposed approach. Excluding the methods in the first column, the first four columns represent image metrics, and the last column represents video metrics. Our video metric, FVD, significantly reduces that of the current state-of-the-art methods. We also achieve the best performance on the SSIM and L1 image metrics, and the other two metrics are comparable to those of other SOTA methods.

[0079]

[0080] Table 1

[0081] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0082] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A diffusion model video generation method based on optical flow information, characterized in that: The steps are as follows: Step 1: Preprocess the videos in the existing dataset and select the first frame of the video as the reference image information. At the same time, use the optical flow algorithm to extract the optical flow information between the previous and next frames of the video. Extract the consecutive frames of the video into images and store them to obtain a video sequence. Adjust the resolution of all images to adapt to subsequent training and inference. Step 2: Build the video generation network OFDiffusion; The standard Stable Diffusion model is used as the backbone network for video generation. A temporal attention layer is added after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video. Two Unets with the same structure as the Unet in the standard SD model are used to extract reference image features and optical flow features respectively. The reference image features are fused with the self-attention layer of the SD model Unet in the self-attention layer, and the optical flow features are fused with the temporal attention layer of the SD model Unet in the cross-attention layer. The overall model is called OFDiffusion, and a motion loss is proposed to constrain the temporal coherence of the generated video. In the video generation network, in order to make the generated video meet the constraints of conditional information, the conditional information includes reference image information and optical flow information, a bypass network is used to extract the features of the conditional information and perform feature fusion with the backbone video generation network; specifically, the backbone network uses the Stable Diffusion model to generate videos, and adds a temporal attention layer after each cross-attention layer in the Unet of the standard SD model to optimize the temporal coherence of the generated video; Stable Diffusion contains a variational encoder VAE to map the input image to a latent space to reduce the computational complexity of the network; VAE includes an encoder ε and a decoder Given an image x, VAE uses the encoder ε(x) to map it into the latent space and the decoder Decode the encoded result; For the reference image information, the Unet with the same structure as the Unet in the standard SD model is used for feature extraction. This network is called Reference Encoder. The features of each self-attention layer of the Reference Encoder are fused with the features of the self-attention layer of each corresponding position of the Unet in the SD model. Specifically, the features of the self-attention layer input of the Unet in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the height of the feature, the width of the feature, and the number of channels, respectively. The input features of the self-attention layer in the Reference Encoder are represented as Therefore, the fusion method of the features of Reference Encoder and Unet in SD model can be expressed as: Q=W Q [x sd1 ,x ref ],K=W k [x sd1 ,x ref ],V=W V [x sd1 ,x ref ]; where [x sd1 ,x ref ] represents the feature concatenation of SD and Reference Encoder, W Q ,W k ,W V Represent the weight matrices of Query, Key, and Value respectively, d represents the number of feature channels, and Self_Attn represents self-attention. After completing the calculation of self-attention, the first half of the features are obtained as the output features of the Unet self-attention layer of the SD model to continue subsequent network propagation. Step 3: Train the video generation network OFDiffusion based on the preprocessed data; Combine the reference image information and optical flow information, and use the diffusion loss and motion loss back propagation to update the parameters; Step 4: Use the trained model for inference to obtain the video generated by the model inference and evaluate the effect of the video.

2. The method for generating a diffusion model video based on optical flow information according to claim 1, characterized in that: For optical flow information, another Unet with the same structure as the Unet in the standard SD model is used for feature extraction. This network is called Flow Controller. The features of each cross attention layer of Flow Controller are fused with the features of the temporal attention layer of each corresponding position of the Unet in the SD model. Specifically, the input features of the Unet temporal attention layer in the SD model are recorded as Where b, f, h, w, c represent the batch size, the number of consecutive frames, the feature height, the feature width, and the number of channels, respectively. The output features of the cross attention layer in the Flow Controller are represented as The fusion method of the features of Flow Controller and Unet in SD model can be expressed as follows: Q=W Q [x sd2 ,x flow ],K=W k [x sd2 ,x flow ],V=W V [x sd2 ,x flow ]; where [x sd2 ,x flow ] represents the feature concatenation of SD and FlowController, W Q ,W k ,W V Represent the weight matrices of Query, Key, and Value respectively, d represents the number of feature channels, and Temp_Attn represents temporal attention. After completing the calculation of temporal attention, the first half of the features are obtained as the output features of the Unet temporal attention of the SD model to continue the subsequent network propagation. At the same time, Diffusion loss and motion loss are used to constrain the generation effect of the network.

3. The method for generating a diffusion model video based on optical flow information according to claim 2, characterized in that: The specific method of step 3 is as follows: Given an image x, SD will learn to de-noise the feature z = ε(x); during the training process, the feature z obtained by VAE will add a Gaussian noise ∈ according to the time step t, and the training goal is to predict the range from z0 to z t The noise ∈ added in , the predicted noise and the original noise are constrained by the mean square error, that is, the Diffusion loss; specifically: where ∈ is the noise added according to time step t, ∈ θ Represents the Unet with learnable parameters θ, i.e., the Unet in the SD model, and c represents the conditional information. In addition, in order to more effectively utilize the extracted optical flow information to constrain the temporal coherence of the video, a motion loss is proposed. Specifically, from the feature z based on the time step t with noise ∈ t Remove the noise predicted by the network and record the obtained features as but: in, It is a hyperparameter related to the time step t; When the input video sequence has f consecutive frames, a feature sequence is obtained The feature difference between the previous and next frames is defined as the motion occurring before and after, and the motion loss can be expressed as: in Represents the feature representation of the original image obtained by the encoder ε for the i-th picture in the original video sequence; then the final optimization goal can be expressed as: Where λ1 and λ2 are custom hyperparameters; During the training process, the feature representation of the original image in the latent space is obtained through the SD VAE, and random Gaussian noise ∈ is added to this feature based on the time step t as the input of the Unet in the SD model; the reference image information is used as the input of the ReferenceEncoder, and after obtaining the reference image feature, it is feature fused with the Unet of the SD model; the optical flow information is used as the input of the Flow Controller, and after obtaining the optical flow feature, it is feature fused with the Unet of the SD model; the Unet of the SD model will finally predict the added Gaussian noise, and the OFDiffusion model will be gradient backpropagated and model parameters updated through the predicted noise and the noise ∈ added according to the time step t before training combined with the Diffusion loss and motion loss.

4. The method for generating a diffusion model video based on optical flow information according to claim 1, characterized in that: The specific method of step 4 is as follows: During the model inference phase, a random Gaussian noise needs to be provided to the OFDiffusion model. The model will denoise the features step by step according to the provided conditional information and time step, and finally obtain the generated image. Since the video is generated, a series of random Gaussian noises are provided to the model. The model denoises each noise separately, and finally obtains an image sequence. The generated video can be obtained by sampling at the set fps.

Images