A diffusion model video generation method fusing dynamic perception

Abstract

The application provides a diffusion model video generation method fusing dynamic perception, and relates to the technical field of video generation.The application proposes a global-local coupling module, which comprises a convolution branch and a convolution attention module, wherein the convolution attention module realizes dynamic modulation of convolution features through channel attention weights, thereby effectively fusing local spatial information and global space-time information.The structure effectively overcomes the defects of a large receptive field of a traditional diffusion model, which is prone to overfitting or "memory" of video data, and greatly improves the diversity and generalization ability of generated video content.The application adopts a single-scale training strategy, trains only at one resolution scale, eliminates the detail error accumulation problem that is prone to occur in a traditional multi-scale pyramid structure video generation method, effectively avoids visual artifacts and unnatural detail distortion phenomena caused by step-by-step training of a multi-scale model, and improves the visual realism and quality of the generated video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video generation, in particular to a diffusion model video generation method fusing dynamic perception. BACKGROUND

[0002] The main technologies for video generation are generative adversarial networks (GAN) and diffusion models. Generally, generative adversarial networks use a multi-scale structure, training different resolution generation networks layer by layer, first generating the rough structure of the video, and then gradually improving the detail quality and resolution of the generated video, so as to realize video content generation. When such methods are used for single video scenes, the video is usually split into different scale or resolution segments for training, and the video details and coherence are gradually improved between different scales, and finally the complete video content is synthesized.

[0003] However, the multi-scale structure of the generative adversarial network method has obvious shortcomings, that is, the small errors that occur in the low scale video generation stage are easy to accumulate and amplify in higher scales, resulting in unnatural visual artifacts and detail distortion problems in the generated video, reducing the quality and realism of the generated video. In addition, this method needs to train multiple generators of different scales step by step, resulting in long training time, large resource consumption, and difficulty in adapting to the needs of fast video generation and editing.

[0004] In recent years, video generation techniques based on diffusion models have emerged. Diffusion model technology uses diffusion and inverse diffusion processes to gradually master the distribution of video data by adding noise to the data during training and training the network to remove these noises. For example, SinFusion realizes high-quality generation of images or videos through a diffusion framework. This method can effectively improve the visual quality and diversity of the generated video and reduce visual artifacts.

[0005] However, traditional diffusion model video generation techniques usually rely on large-scale video datasets for pre-training and are difficult to use only a single video as a training sample. At the same time, its network structure usually has a large receptive field, which can easily lead to overfitting or memorization of training data, resulting in video content that is too similar to the input video, lacking sufficient diversity and generalization ability.

[0006] Because video generation techniques usually need to be fast and efficient with low resource consumption, a method must be proposed that can avoid multi-scale error accumulation and achieve high-quality, diversified video generation using only a single video dataset. This requires the proposed method to have a limited receptive field to avoid overfitting to all the details and features of the original video, while also learning local features and global spatio-temporal features to meet the practical application needs of efficient generation of diversified video content under single video conditions.

[0007] The first type of single video generation method based on a generative adversarial network (GAN) currently generally adopts a multi-scale pyramid architecture, that is, different generator networks are trained at different scales to restore video details step by step. The main defect of this method is that the multi-scale step-by-step training process is prone to cumulative error effect of details. Specifically, even if there is only a slight error or artifact in the low-scale generation stage, it will be gradually magnified at a higher scale, resulting in obvious visual artifacts and unnatural detail distortion in the final video sequence, which seriously reduces the visual quality and authenticity of the generated video. In addition, the multi-scale step-by-step training architecture itself requires training a large number of models, resulting in large computational overhead and high resource consumption, which is not suitable for scenarios with limited generation efficiency and resource usage.

[0008] Although the second type of video generation technology based on diffusion models can significantly improve the diversity and visual quality of the generated video, it generally relies on large-scale video datasets for pre-training, making it difficult to achieve high-quality generation and editing using only a single video. In addition, the network structure of traditional diffusion models generally has a large receptive field, making the network more likely to learn global information of the complete video content, resulting in overfitting and even "memorizing" the input video data. This "overfitting" problem limits the diversity and generalization ability of the generated video, making the model unable to generalize well to new video content generation scenarios. SUMMARY

[0009] In view of the deficiencies of the prior art, the purpose of the present application is to provide a diffusion model video generation method fusing dynamic perception, comprising:

[0010] Step 1: Obtain a reference video, decode the reference video at a preset frame rate to obtain a sequence of continuous video frames wherein N represents the total number of frames of the sequence of continuous video frames, represents the Nth image;

[0011] Step 2: crop and add noise to each image in the sequence of continuous video frames to obtain a noisy image , and further obtain a sequence of noisy images wherein represents the Nth image corresponding to the noisy image;

[0012] Step 3: input the sequence of noisy images into a single-scale diffusion model to obtain a sequence of predicted feature maps wherein denotes the Nth prediction feature map, and denotes a parameter vector of the single-scale diffusion model, the single-scale diffusion model comprising a plurality of global-local coupled modules, the global-local coupled modules comprising a ResNet convolution branch and a convolution attention module branch;

[0013] Step 4: based on the sequence of continuous video frames and the sequence of prediction feature maps , a joint loss function value is calculated;

[0014] Step 5: based on the joint loss function value , the weight matrix of the ResNet convolution branch and the weight matrix of the convolution attention module branch in the single-scale diffusion model are updated by back propagation, and the single-scale diffusion model is continuously updated for multiple times to obtain a trained single-scale diffusion model;

[0015] Step 6: the to-be-generated video is decoded, cropped and added with noise, and input into the trained single-scale diffusion model to obtain a sequence of prediction feature maps of the to-be-generated video;

[0016] Step 7: based on the trained single-scale diffusion model, video generation and inference are performed on the sequence of prediction feature maps to obtain a sequence of continuous prediction frames, a sequence of optimized frames and a sequence of smooth transition frames.

[0017] Optionally, step 2 specifically comprises:

[0018] Step 2.1: for each image in the sequence of continuous video frames , random large-scale cropping is performed on the image according to a current resolution ratio of the image and an area ratio in a preset range to obtain a cropped image ;

[0019] Step 2.2: Gaussian noise is added to the cropped image to obtain a noisy image , which is specifically realized by the following formula:

[0020] ;

[0021] wherein, t denotes a diffusion time step, is a noise control coefficient in the diffusion process, denotes noise randomly sampled from a standard normal distribution;

[0022] All images of the noisy image comprise a sequence of noisy images .

[0023] Optionally, step 3 specifically includes:

[0024] Step 3.1: For noisy image sequences For each noisy image in , the noisy image is input into the ResNet convolution branch in the global-local coupling module to obtain the first feature map;

[0025] Step 3.2: The first feature map is subjected to global average pooling to obtain the feature vector , which is specifically achieved through the following formula:

[0026] ;

[0027] Among them, W represents the total number of width indexes, H represents the total number of height indexes, and C represents the total number of channels. Indicates that the first feature map is in Channel, i Width index, Height-indexed pixel feature values;

[0028] Step 3.3: Transform the feature vector Input the convolutional attention module branch to obtain the intermediate feature map, and then obtain the intermediate feature map corresponding to each noisy image;

[0029] Step 3.4: Input the intermediate feature map into the next global-local coupling module and repeat the above operation until the last global-local coupling module outputs the predicted feature map. The predicted feature maps corresponding to all noisy images constitute the predicted feature map sequence. .

[0030] Optionally, step 3.3 specifically includes:

[0031] According to the feature vector , calculate the attention weight of each channel , which is specifically achieved through the following formula:

[0032] ;

[0033] in, is the Sigmoid function, is the neighborhood set centered on channel c, is the kth weight matrix in the weight matrix Q, is the eigenvector The kth eigenvector in ;

[0034] The attention weight and eigenvectors Multiply them together to get the intermediate feature map.

[0035] Optionally, step 4 specifically comprises:

[0036] calculating the mathematical expectation between each frame of image and its corresponding predicted feature map and summing up to obtain the L2 loss function value , which is specifically realized by the following formula:

[0037] ;

[0038] based on each frame of image and its corresponding predicted feature map , calculating the CLIP semantic consistency loss function value ;

[0039] and further calculating the joint loss function value , which is specifically realized by the following formula:

[0040] ;

[0041] wherein, is a weight coefficient.

[0042] Optionally, step 7 specifically comprises:

[0043] Step 7.1: improving the trained single-scale diffusion model to obtain a first improved model and a second improved model;

[0044] Step 7.2: inputting the predicted feature map sequence of the to-be-generated video into the first improved model to obtain a continuous predicted frame sequence;

[0045] Step 7.3: inputting the continuous predicted frame sequence into the second improved model to obtain an optimized frame sequence;

[0046] Step 7.4: in the continuous video frame sequence, obtaining the middle frame of each two frames of image in the optimized frame sequence, and further obtaining a new video frame sequence, inputting the new video frame sequence into the second improved model to obtain a smooth transition frame sequence.

[0047] Optionally, step 7.1 specifically comprises:

[0048] For each global-local coupling module in the trained single-scale diffusion model, a first time processing is added before the convolution attention module branch after the global average pooling, specifically, a diffusion time embedding function and an inter-frame difference embedding function , to obtain a first improved model; a second time processing is added before the convolution attention module branch after the global average pooling, specifically, a diffusion time embedding function is added , to obtain a second improved model.

[0049] Optionally, step 7.2 specifically comprises:

[0050] In the global-local coupling module of the first improved model, the predicted feature map sequence of the video to be generated is sequentially subjected to the ResNet convolution branch, the global average pooling, the diffusion time embedding function and the inter-frame difference embedding function , the convolution attention module branch, and the output feature map is input into the next global-local coupling module of the first improved model, until the last global-local coupling module of the first improved model outputs a continuous predicted frame sequence.

[0051] Optionally, step 7.3 specifically comprises:

[0052] In the global-local coupling module of the second improved model, the continuous predicted frame sequence is sequentially subjected to the ResNet convolution branch, the diffusion time embedding function , the convolution attention module branch, and the output feature map is input into the next global-local coupling module of the second improved model, and the last global-local coupling module of the second improved model outputs an optimized frame sequence.

[0053] The beneficial effects produced by the above technical solution are:

[0054] The application proposes a global-local coupling module, which comprises a convolution branch and a convolution attention module. The convolution attention module dynamically modulates the convolution features through channel attention weights, thereby effectively fusing local spatial information and global spatio-temporal information. This structure effectively overcomes the defects of the traditional diffusion model, such as large receptive field, easy overfitting, and easy "memory" of video data, greatly improving the diversity and generalization ability of the generated video content. The video generation network of the application adopts a single-scale training strategy, which only trains at one resolution scale, eliminating the problem of cumulative error of details in the traditional multi-scale pyramid structure video generation method, effectively avoiding the visual artifacts and unnatural detail distortion phenomena caused by multi-scale model training, and improving the visual realism and quality of the generated video. The application effectively prevents the network from over-memorizing the global information of the video by using a random large-size cropping training strategy, while preserving the overall structure of the video and greatly improving the diversity of the generated content, meeting the actual application requirements of single-video-based fast and diversified video generation. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a diffusion model video generation method fusing dynamic perception is shown in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0057] To solve the problems in the prior art, the present application provides a diffusion model video generation method fusing dynamic perception, which overcomes the defects of detail error accumulation, large resource consumption, strong data dependency and easy overfitting in the prior art. The present application only trains based on a single video, proposes a global-local coupling module based on dynamic perception, combines a ResNet convolution branch and a convolution attention module structure, effectively limits the receptive field range of model learning video data on the network structure, thereby avoiding over-memorizing the global content of the video by the model and ensuring the diversity of the generated video. At the same time, the single-scale training architecture eliminates the problem of multi-scale error accumulation, greatly reduces the resource usage and computational burden, and improves the applicability and efficiency of the video generation method.

[0058] Specifically Figure 1 The diffusion model video generation method fusing dynamic perception provided by the present application can include the following steps:

[0059] Step 1: Obtain a reference video, decode the reference video by a preset frame rate to obtain a continuous video frame sequence wherein N represents the total number of frames of the continuous video frame sequence, represents the Nth image;

[0060] In the specific implementation process, the preset frame rate can be 25-30 frames per second.

[0061] Step 2: Crop and add noise to each image in the continuous video frame sequence to obtain a noisy image , and further obtain a noisy image sequence wherein represents the Nth image corresponding to the noisy image;

[0062] Step 2.1: Randomly crop the image at a large scale according to the current resolution ratio of the image and the area ratio of the preset range to obtain a cropped image

[0063] In the specific implementation process, the area ratio of the preset range is ​​, the center position of the cropped window obeys uniform distribution in horizontal and vertical directions.

[0064] wherein the random large-scale cropping can manufacture visual field drift, reduce the risk of model overfitting, retain the main objects and background structures of the original frame, ensure the temporal consistency, and reduce the number of single-frame pixels to improve the training efficiency. The cropped image can be further scaled and data augmented to serve as a network input sample.

[0065] Step 2.2: performing scaling and data augmentation on the cropped image adding Gaussian noise to obtain a noisy image , which is achieved by the following formula:

[0066] ;

[0067] wherein t represents a diffusion time step, is a noise control coefficient in the diffusion process, decreases from 1 to 0 with the increase of the diffusion time step t, represents noise randomly sampled from a standard normal distribution;

[0068] all images are noisy images to form a noisy image sequence .

[0069] Step 3: inputting the noisy image sequence into a single-scale diffusion model to obtain a predicted feature map sequence wherein represents the Nth predicted feature map, and θ represents a parameter vector of the single-scale diffusion model, the single-scale diffusion model comprising a plurality of global-local coupling modules, the global-local coupling modules comprising a ResNet convolution branch and a convolution attention module branch;

[0070] The single-scale diffusion model has several advantages over the multi-scale method when it comes to image or video generation tasks. First, the single-scale method avoids the problem of error accumulation in traditional multi-scale methods, reducing image or video artifacts that occur when transferring information between different scales. Multi-scale generation methods typically use a layer-by-layer enhancement strategy, with each layer refining the image based on the previous one. However, this also makes it easy to accumulate errors and produce inconsistent details in the generated results. In contrast, the single-scale model optimizes at a single scale, avoiding this problem and generating more consistent and detailed outputs. In addition, the single-scale method greatly reduces the computational burden and model complexity, making the training process more efficient, especially when generating high-resolution images. The single-scale design still maintains relatively low computational resource consumption. This method balances the generation quality and computational cost, making it suitable for large-scale generation tasks.

[0071] Step 3.1: For each noisy image in the noisy image sequence , input the noisy image into the ResNet convolution branch in the global-local coupling module to obtain the first feature map; wherein the first feature map .

[0072] Step 3.2: Perform global average pooling on the first feature map to obtain the feature vector , which is achieved by the following formula:

[0073] ;

[0074] where W represents the total number of width indices, H represents the total number of height indices, C represents the total number of channels, represents the pixel feature value of the first feature map at the th channel, the i th width index, and the th height index;

[0075] Step 3.3: Input the feature vector into the convolution attention module branch to obtain the intermediate feature map, and then obtain the intermediate feature map corresponding to each noisy image;

[0076] Specifically, according to the feature vector , the attention weight of each channel is calculated, which is achieved by the following formula:

[0077] ;

[0078] where is the Sigmoid function, is the neighborhood set centered on channel c, is the kth weight matrix in the weight matrix Q. The weight matrix can be obtained through learning. is the eigenvector The kth eigenvector in ;

[0079] The attention weight and eigenvectors Multiply them together to get the intermediate feature map.

[0080] Step 3.4: Input the intermediate feature map into the next global-local coupling module and repeat the above operation until the last global-local coupling module outputs the predicted feature map. The predicted feature maps corresponding to all noisy images constitute the predicted feature map sequence. .

[0081] Step 4: Based on a sequence of continuous video frames and predicted feature map sequence , calculate the joint loss function value ;

[0082] Specifically, calculate each frame of image And its corresponding predicted feature map The mathematical expectation between them and summation is used to obtain the L2 loss function value. , which is specifically achieved through the following formula:

[0083] ;

[0084] Based on each frame And its corresponding predicted feature map , calculate the CLIP semantic consistency loss function value ;

[0085] Then calculate the joint loss function value , which is specifically achieved through the following formula:

[0086] ;

[0087] in, is the weight coefficient.

[0088] In the comprehensive loss design of the present application, the introduced λ coefficient assumes the role of a "trade-off valve" for dynamically balancing the relative importance between pixel-level fidelity (L2 loss) and semantic consistency constraint (CLIP semantic loss). When λ is small, the network focuses more on pixel-by-pixel denoising and texture restoration, and the generated sequence has higher fidelity in terms of details, but the cross-frame or text consistency may be slightly insufficient. Conversely, when λ is large, the model significantly enhances semantic coherence and text controllability, although it may slightly weaken local high-frequency details, it can effectively alleviate the problems of temporal jitter and semantic drift. In practice, λ is usually initialized between 0.1-1.0 to obtain the optimal pixel-semantic balance in different scenarios.

[0089] Step 5: updating the weight matrix of the ResNet convolution branch and the weight matrix of the convolution attention module branch of the single-scale diffusion model by backpropagation based on the joint loss function value, and then continuing to update the single-scale diffusion model multiple times to obtain a trained single-scale diffusion model;

[0090] Step 6: decoding, cropping, and adding noise to the video to be generated, and inputting the trained single-scale diffusion model to obtain a predicted feature map sequence of the video to be generated;

[0091] Step 7: based on the trained single-scale diffusion model, performing video generation and inference on the predicted feature map sequence to obtain a continuous predicted frame sequence, an optimized frame sequence, and a smooth transition frame sequence.

[0092] Step 7.1: improving the trained single-scale diffusion model to obtain a first improved model and a second improved model;

[0093] For each global-local coupling module in the trained single-scale diffusion model, a first temporal processing is added before the convolution attention module branch after global average pooling, specifically, a diffusion time embedding function and an inter-frame difference embedding function are added to obtain the first improved model; a second temporal processing is added before the convolution attention module branch after global average pooling, specifically, a diffusion time embedding function is added to obtain the second improved model.

[0094] Step 7.2: inputting the predicted feature map sequence of the video to be generated into the first improved model to obtain a continuous predicted frame sequence;

[0095] In the global-local coupling module of the first improved model, the predicted feature map sequence of the video to be generated is sequentially passed through the ResNet convolution branch, global average pooling, diffusion time embedding function ​and an inter-frame difference embedding function , a convolutional attention module branch, and the output feature map is input into the next global-local coupling module of the first improved model until the last global-local coupling module of the first improved model outputs a continuous predicted frame sequence.

[0096] That is, after obtaining the predicted feature map sequence of the video to be generated, the predicted feature map sequence of the video to be generated is input into the first improved model, and sequentially passes through a plurality of improved global-local coupling modules. In the first improved global-local coupling module, the predicted feature map sequence of the video to be generated sequentially passes through a ResNet convolution branch, a global average pooling, a diffusion time embedding function and an inter-frame difference embedding function , a convolutional attention module branch, and the output feature map is input into the next global-local coupling module of the first improved model until the last global-local coupling module of the first improved model outputs a continuous predicted frame sequence.

[0097] Among them, step 7.2 can be understood as a prediction branch. In the traditional method, video generation usually depends on a simple time series prediction model, but when generating multiple frames, due to insufficient processing of time correlation, it is easy to produce incoherent motion. The technical solution of step 7.2 of the present application ensures the macro motion coherence in the generation process. This branch avoids motion inconsistency in the prediction process through accurate time series modeling, can effectively capture the dependence relationship in the time dimension during the generation process, and significantly improves the continuity and smoothness of the video.

[0098] Step 7.3: inputting the continuous predicted frame sequence into the second improved model to obtain an optimized frame sequence;

[0099] In the global-local coupling module of the second improved model, the continuous predicted frame sequence sequentially passes through a ResNet convolution branch, a diffusion time embedding function , a convolutional attention module branch, and the output feature map is input into the next global-local coupling module of the second improved model, and the last global-local coupling module of the second improved model outputs an optimized frame sequence.

[0100] Similarly, the continuous predicted frame sequence is input into the second improved model and sequentially passes through a plurality of improved global-local coupling modules. In the first improved global-local coupling module, the continuous predicted frame sequence sequentially passes through a ResNet convolution branch, a diffusion time embedding function , the convolution attention module branch, input the feature map sequence into the next improved global-local coupling module, sequentially serial processing, and output the optimized frame sequence from the last improved global-local coupling module.

[0101] Wherein, due to the different functions added by the first improved model and the second improved model, the two improved global-local coupling modules are also different.

[0102] Wherein, step 7.3 can be understood as a correction branch, which effectively removes artifacts and improves the accuracy of inter-frame details by correcting the details of the predicted frame. Traditional video generation methods usually have rough details when predicting frames, resulting in artifacts or distortion in the generated video at local details. Step 7.3 can realize the correction of details, which not only can repair the details error, but also can maintain the consistency of the global video content, avoiding the quality loss caused by multiple corrections in traditional methods, ensuring the high fidelity of video quality.

[0103] Step 7.4: In the continuous video frame sequence, obtain the intermediate frame image of every two frames of the optimized frame sequence, and then obtain the new video frame sequence; input the new video frame sequence into the second improved model to obtain the smooth transition frame sequence, specifically, input the new video frame sequence into the global-local coupling module of the second improved model, and sequentially pass the new video frame sequence through the ResNet convolution branch, diffusion time embedding function , the convolution attention module branch, and input the output feature map into the next global-local coupling module of the second improved model, and output the smooth transition frame sequence from the last global-local coupling module of the second improved model.

[0104] Wherein, step 7.4 can be understood as an interpolation branch, which significantly improves the frame rate and visual smoothness of the video by generating intermediate frames. Traditional video generation methods often rely on simple interpolation algorithms when dealing with frame rate improvement, lacking attention to timing details, resulting in insufficient smoothness or naturalness between frames. However, the scheme of the present application generates intermediate frames through interpolation algorithms, which not only improves the video frame rate, but also makes the visual transition of the video smoother, enhancing the viewing experience. The design of the interpolation branch effectively makes up for the lack of smooth transition in traditional methods during dynamic generation, making the video present a more delicate and realistic dynamic effect.

[0105] The three branches maintain parameter uniformity and style consistency due to sharing a single-scale diffusion model backbone with global-local coupling modules. At the same time, through differentiated embedding conditions and input forms, the "prediction-correction-interpolation" closed loop is realized, solving the problems of motion fragmentation, detail artifacts, and insufficient frame rate, while significantly improving the quality and efficiency of video generation.

[0106] Compared with the traditional video inference method, the three-branch collaborative inference module has significant innovation points in the video generation task. Specifically, the design of each branch improves the quality and efficiency of video generation in different ways, solving the problems of error accumulation and detail distortion commonly found in previous methods.

[0107] The application proposes a single-scale diffusion model-based video generation method, and the key technical improvements are as follows:

[0108] (1) A global-local coupling module is proposed, which includes a convolution branch and a convolution attention module. The convolution attention module dynamically modulates the convolution features through channel attention weights, effectively fusing local spatial information and global spatio-temporal information. This structure effectively overcomes the defects of traditional diffusion models, such as large receptive field, easy overfitting or "memory" of video data, greatly improving the diversity and generalization ability of generated video content.

[0109] (2) The video generation network of the application adopts a single-scale training strategy, which only trains at one resolution scale, eliminating the problem of detail error accumulation in traditional multi-scale pyramid structure video generation methods, effectively avoiding visual artifacts and unnatural detail distortion caused by multi-scale model training, and improving the visual realism and quality of generated video.

[0110] (3) The application adopts a random large-size cropping training strategy, effectively preventing the network from over-memorizing global information of video frames, preserving the overall structure of the video while greatly improving the diversity of generated content, meeting the practical application requirements of single video-based fast and diversified video generation.

[0111] (4) A three-branch collaborative inference framework is proposed, which effectively decouples video frame prediction, artifact correction, and interpolation tasks, and independently trains them before working in series during inference, effectively improving the flexibility and stability of the video generation process and expanding the application capabilities of a single video in multiple scenarios such as video prediction, video interpolation, and video editing.

[0112] Compared with the prior art, the single-scale diffusion video generation method based on dynamic perception proposed by the application has significant advantages in video quality, diversity, computational resource consumption, and efficiency, and the specific performance is as follows:

[0113] (1) The dynamic perception global-local coupling module structure proposed in the application significantly improves the visual quality and diversity of the generated video. In the experiment, compared with the traditional multi-scale GAN method (such as SinGAN-GIF) and the existing diffusion model (such as SinFusion), in the single video generation task of the standard video generation evaluation dataset HP-VAE-GAN and SinGAN-GIF, the SVFID index is reduced from 0.0119 of the traditional method to 0.0071, the performance is improved by more than 40%, and the visual quality and authenticity of the video are effectively improved.

[0114] (2) The single-scale network structure adopted in the application effectively avoids the problem of cumulative error of details generated by the multi-scale model during step-by-step training, and the generated video sequence no longer has obvious artifacts or detail distortion in vision, significantly improving the overall visual coherence and realism of the video sequence.

[0115] (3) In terms of resource occupation and computing efficiency, the application is significantly better than the existing method. The single-scale fusion dynamic perception diffusion model structure significantly reduces the number of model parameters and the computational complexity, reduces the required training resources by more than 50%, and significantly improves the inference generation speed. In actual testing, it only takes 47 seconds to generate a 144x256 resolution video frame, while the existing multi-scale video diffusion model takes 1.12 minutes, and the inference speed is improved by more than 30%, which is more suitable for resource-constrained environments and real-time generation scenarios.

[0116] (4) The three-branch video inference framework proposed in the application significantly enhances the flexibility of the video generation task. In the experiment, the prediction branch realizes accurate video frame prediction, the correction branch effectively eliminates local detail artifacts in the video, and the interpolation branch improves the frame rate and smoothness of the video in actual application, effectively meeting the practical application requirements of video editing, video interpolation and video extrapolation.

[0117] In summary, the video generation method proposed in the application has made significant improvements and optimizations in the visual quality, diversity, generalization ability and resource use efficiency of the generated video, and can effectively overcome the many defects of traditional technologies, and has good technical advancement and practical application prospect.

[0118] The above description is only the preferred embodiments of the disclosure and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the application involved in the embodiments of the disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features and the technical features disclosed in the embodiments of the disclosure (but not limited to) having similar functions.

Claims

1. A method for generating video using a diffusion model integrating dynamic perception, characterized in that: include: Step 1: Obtain a reference video and decode it at a preset frame rate to obtain a continuous video frame sequence. , where N represents the total number of frames in a continuous video frame sequence, Indicates the Nth frame image; Step 2: For each frame in the continuous video frame sequence Cropping and noise addition are performed to obtain a noisy image , and then obtain the noisy image sequence ,in, Represents the noisy image corresponding to the Nth frame image; Step 3: Convert the noisy image sequence Input into the single-scale diffusion model to obtain the predicted feature map sequence ,in, represents the Nth predicted feature map, θ represents a parameter vector of a single-scale diffusion model, the single-scale diffusion model includes a plurality of global-local coupling modules, and the global-local coupling module includes a ResNet convolution branch and a convolutional attention module branch; Step 4: Based on a sequence of continuous video frames and predicted feature map sequence , calculate the joint loss function value ; Step 5: Based on the joint loss function value , the weight matrix of the ResNet convolution branch and the weight matrix of the convolutional attention module branch in the single-scale diffusion model are updated through back propagation, and then the single-scale diffusion model is updated multiple times to obtain the trained single-scale diffusion model; Step 6: Decode, crop, and add noise to the video to be generated, and input it into the trained single-scale diffusion model to obtain a sequence of predicted feature maps of the video to be generated; Step 7: Based on the trained single-scale diffusion model, perform video generation and inference on the predicted feature map sequence to obtain a continuous predicted frame sequence, an optimized frame sequence, and a smooth transition frame sequence; Step 7.1: Improve the trained single-scale diffusion model to obtain a first improved model and a second improved model; Specifically, for each global-local coupling module in the trained single-scale diffusion model, the first temporal processing is added after the global average pooling and before the convolution attention module branch. Specifically, the diffusion time embedding function is added. and inter-frame difference embedding function , get the first improved model; add the second temporal processing after the global average pooling and before the convolution attention module branch, specifically, add the diffusion time embedding function , and obtain the second improved model; Step 7.2: Input the predicted feature map sequence of the video to be generated into the first improved model to obtain a continuous predicted frame sequence; Specifically, in the global-local coupling module of the first improved model, the predicted feature map sequence of the video to be generated is sequentially passed through the ResNet convolution branch, global average pooling, and diffusion time embedding function. and inter-frame difference embedding function , the convolutional attention module branch, and inputs the output feature map into the next global-local coupling module of the first improved model until the last global-local coupling module of the first improved model outputs a continuous sequence of predicted frames; Step 7.3: Input the continuous predicted frame sequence into the second improved model to obtain the optimized frame sequence; Specifically, in the global-local coupling module of the second improved model, the continuous prediction frame sequence is sequentially passed through the ResNet convolution branch, the diffusion time embedding function , the convolutional attention module branch, and inputs the output feature map into the next global-local coupling module of the second improved model, and the last global-local coupling module of the second improved model outputs the optimized frame sequence; Step 7.4: In the continuous video frame sequence, obtain the intermediate frame image between every two frames in the optimized frame sequence to obtain a new video frame sequence, and input the new video frame sequence into the second improved model to obtain a smooth transition frame sequence.

2. The method for generating a video using a diffusion model integrating dynamic perception according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: For each frame in the continuous video frame sequence , according to the current resolution ratio of the image and the area ratio of the preset range, the image is randomly cropped on a large scale to obtain the cropped image ; Step 2.2: Cropped image Add Gaussian noise to get a noisy image , which is specifically achieved through the following formula: ； in, t represents the diffusion time step, is the noise control coefficient in the diffusion process, represents noise randomly sampled from a standard normal distribution; All images Noisy image Composing a noisy image sequence .

3. The method for generating a video using a diffusion model integrating dynamic perception according to claim 1, wherein: Step 3 specifically includes: Step 3.1: For noisy image sequences For each noisy image in , the noisy image is input into the ResNet convolution branch in the global-local coupling module to obtain the first feature map; Step 3.2: The first feature map is subjected to global average pooling to obtain the feature vector , which is specifically achieved through the following formula: ； Among them, W represents the total number of width indexes, H represents the total number of height indexes, and C represents the total number of channels. Indicates that the first feature map is in Channel, i Width index, Height-indexed pixel feature values; Step 3.3: Transform the feature vector Input the convolutional attention module branch to obtain the intermediate feature map, and then obtain the intermediate feature map corresponding to each noisy image; Step 3.4: Input the intermediate feature map into the next global-local coupling module and repeat the above operation until the last global-local coupling module outputs the predicted feature map. The predicted feature maps corresponding to all noisy images constitute the predicted feature map sequence. .

4. The method for generating a video using a diffusion model integrating dynamic perception according to claim 3, wherein: Step 3.3 specifically includes: According to the feature vector , calculate the attention weight of each channel , which is specifically achieved through the following formula: ； in, is the Sigmoid function, is the neighborhood set centered on channel c, is the kth weight matrix in the weight matrix Q, is the eigenvector The kth eigenvector in ; The attention weight and eigenvectors Multiply them together to get the intermediate feature map.

5. The method for generating a video using a diffusion model integrating dynamic perception according to claim 1, wherein: Step 4 specifically includes: Calculate each frame of image And its corresponding predicted feature map The mathematical expectation between them and summation is used to obtain the L2 loss function value. , which is specifically achieved through the following formula: ； Based on each frame And its corresponding predicted feature map , calculate the CLIP semantic consistency loss function value ; Then calculate the joint loss function value , which is specifically achieved through the following formula: ； in, is the weight coefficient.

Images