A video frame interpolation method based on image diffusion model

Abstract

The application relates to a video frame filling method based on an image diffusion model, which comprises the following steps: S1, acquiring a low-frame-rate video, carrying out denoising diffusion implicit model inversion, obtaining first noise hidden encoding, and carrying out linear interpolation to obtain second noise hidden encoding; S2, inputting the second noise hidden encoding into a diffusion model, calculating a nearest neighbor domain through nearest neighbor search, and then executing S3 and S4 in each time step in the order of T to 1; S3, inputting the first noise hidden encoding of the time step t into an extended attention module to extract key frame features; S4, propagating the key frame features in the second noise hidden encoding based on interframe feature correlation, inputting the second noise hidden encoding of the time step t into the diffusion model, and carrying out denoising by taking the propagation result as a guide condition; and S5, after executing S3 and S4 in each time step, outputting a video after frame filling. Compared with the prior art, the application has the advantages of improving the fluency of the frame-filled video and the like.

Description

Technical Field

[0001] This invention relates to the technical field of video frame interpolation, and in particular to a video frame interpolation method based on an image diffusion model. Background Technology

[0002] With the widespread use of mobile devices, users are capturing increasingly more low-quality, low-frame-rate video data. Meanwhile, the image sensors in existing outdoor computer vision systems often struggle to support high-frame-rate video capture, necessitating algorithmic frame interpolation for these low-frame-rate videos. Low-frame-rate videos result in choppy motion during playback. To improve viewing experience and enhance the performance of computer vision systems, certain processing solutions are needed to interpolate frames from low-frame-rate videos. The goal of video interpolation is to add new composite frames between existing video frames, making playback smoother. Current video interpolation techniques are primarily based on motion assessment and motion compensation; that is, first, motion information between consecutive frames is evaluated, and then frames are interpolated based on the motion vectors. However, these methods are not ideal for low-quality, low-frame-rate videos, easily causing blurring and artifacts.

[0003] On the other hand, in recent years, there has been an increasing number of studies exploring the use of deep learning for video frame interpolation, such as content-aware frame interpolation and using generative adversarial networks to synthesize new frames. However, many of these methods require a lot of data and time for network training, and the generated videos suffer from stuttering and jitter due to frame discontinuities. They are also susceptible to complex dynamics, have low robustness, and cannot quickly generate high-quality, smooth, and natural interpolated videos. Summary of the Invention

[0004] The purpose of this invention is to provide a video frame interpolation method based on an image diffusion model to improve the smoothness of interpolated video and reduce stuttering and jitter.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A video frame interpolation method based on an image diffusion model, the method comprising:

[0007] S1. Obtain the original low frame rate video, perform noise reduction and diffusion implicit model inversion on each frame of the low frame rate video to obtain the first noise implicit code corresponding to each frame, record the frame corresponding to the first noise implicit code as the key frame, and perform linear interpolation on the first noise implicit code to obtain the second noise implicit code.

[0008] S2. Input the latent variables of all frames at each time step in the second noise latent coding into the diffusion model. The self-attention module of the diffusion model extracts the features of each frame at each time step of the second noise latent coding. The nearest neighbor is calculated through nearest neighbor search. Then, S3 and S4 are executed in each time step in the order from T to 1.

[0009] S3. Input the first noise latent code at time step t into the extended attention module to extract keyframe features;

[0010] S4. Calculate the inter-frame feature correlation based on the nearest neighbor, propagate the key frame features in the second noise latent code based on the inter-frame feature correlation, input the second noise latent code at time step t into the diffusion model, and use the propagation result as a guiding condition for denoising.

[0011] S5. After executing S3 and S4 at each time step, output the video with interpolated frames.

[0012] Furthermore, the steps of S1 are as follows:

[0013] Obtain the original low frame rate video I = [I 1 ,…,I n ], where I represents a frame in a low frame rate video, let For the latent variable of the i-th frame at time step t, perform noise-diffusion implicit model inversion on each frame of the low frame rate video to obtain the first noise latent code. The frame corresponding to the first noise hidden code is recorded as the keyframe. Linear interpolation is performed on the first noise hidden code to obtain the second noise hidden code. Where N is the target frame number, and the second noise hidden code is denoted as X, and the second noise hidden code contains...

[0014]

[0015] Let it be X t ,

[0016] The second noise hidden coding

[0017]

[0018] Let it be X i .

[0019] Furthermore, after performing the noise reduction and diffusion implicit model inversion, the latent variables of the i-th frame at time step t of the first noise implicit coding are:

[0020]

[0021] Wherein, the initial latent variable of the i-th frame in the first noise latent coding is the i-th original frame, i.e. Gaussian noise ∈ ~N(0,I), hyperparameter It is the variance of a series of Gaussian noises, and T is the total number of time steps.

[0022] Furthermore, the latent variable in the j-th frame at time step t in the second noise latent coding is:

[0023]

[0024] Where j∈(i,i+1) is the floating-point index, and λ=j-1 is the interpolation coefficient. It is the second noise hidden code after interpolation.

[0025] Furthermore, the steps of S2 are as follows:

[0026] The latent variables of all frames at each time step in the second noisy latent coding are input into the diffusion model. The self-attention module of the diffusion model extracts the set of features of all frames at each time step of the second noisy latent coding. Let φ(λ) be the feature of the i-th frame at each time step t in φ(X). remember The features corresponding to two adjacent frames are and i+ represents the nearest next frame index, and i- represents the nearest previous frame index. The nearest neighbor γ is calculated based on the nearest neighbor search. i+ and γ i- .

[0027] Furthermore, the nearest neighbor γ i+ and γ i- for:

[0028] γ i± [p] = arg q minD(φ(x i )[p],φ(x i± )[q])

[0029] Where p and q are the spatial positions in the feature map, and D is the cosine distance.

[0030] Furthermore, the steps in S3 are as follows:

[0031] Select keyframe indices from the floating-point indices to form a keyframe index set K, denoted as K. To select a tensor composed of the elements with frame indices in K from the second noisy latent coding at time step t, which is the first noisy latent coding at time step t, the first noisy latent coding at time step t is input into the extended attention module to extract keyframe features φ(X). K ).

[0032] Furthermore, the specific steps for calculating the inter-frame feature correlation based on the nearest neighbor are as follows:

[0033] Linear combination φ(X) K The features corresponding to each spatial location p and frame number i are used to obtain the inter-frame feature correlation F. γ .

[0034] Furthermore, the inter-frame feature correlation is as follows:

[0035] F γ (φ(X K ),i,p)=w i ·φ(J i+ )[γ i+ [p]]+(1-w i )·φ(J i- )[γ i- [p]

[0036] Wherein, φ(J i± )∈φ(X K ),w i ∈(0,1) is a scalar that is proportional to the distance between the i-th frame and its adjacent keyframes.

[0037] Furthermore, the video after frame interpolation is as follows: in,

[0038]

[0039] in, The diffusion model is represented by TokenFlow, which describes the process of keyframe features propagating in the second noise latent coding. t-1 This represents the image at time step t-1 obtained by denoising the second noise implicit coding at time step t. After frame interpolation, the video is output by the diffusion model based on the inter-frame feature correlation.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) This invention performs noise-diffusion implicit model inversion on input low frame rate videos to obtain the noise-coded latent image corresponding to each frame and interpolates to fill in the frames. The noise-diffusion implicit model inversion encodes the original video image from the pixel space to the latent variable space, while gradually adding Gaussian noise until the image latent code completely conforms to the standard Gaussian distribution. Then, linear interpolation is performed on the noise latent code to fill in the desired number of frames, and the original noise latent code is recorded as the keyframe. Compared with existing video editing methods that usually perform inter-frame processing directly in the pixel space, manipulating the noise image in the latent code space greatly reduces the computational complexity, improves the speed of generating frame-filled videos, and the explicit operation on the latent variable features helps to maintain the consistency of the generated results.

[0042] (2) Based on the pre-calculated inter-frame feature correlation, i.e., the nearest neighbor region, this invention propagates keyframe features in the latent space, ensuring redundancy and consistency of features across all frames. Explicit feature propagation significantly improves the consistency of the interpolated video, forcing these features to inherit the same inter-frame correlation and information redundancy as the original video, thereby maintaining the temporal coherence and inter-frame consistency of the interpolated video. Traditional generative model-based video editing methods struggle to explicitly encourage temporal consistency in generated results, easily leading to stuttering and jitter. This invention, by propagating the features of noisy latent variable images in the latent space, fully utilizes redundant information in adjacent frames, significantly improving consistency and reducing stuttering and jitter.

[0043] (3) This invention encodes video images from pixel space to latent space, greatly reducing computational complexity. Existing video editing methods typically perform inter-frame processing directly in pixel space, which often consumes a lot of computational resources when faced with high-resolution image input. Furthermore, due to the redundancy of inter-frame information in video, directly calculating pixel features can easily lead to a significant waste of computational power. This invention adopts a scheme of manipulating image features in latent space, avoiding direct calculation in pixel space, thereby reducing the requirements for hardware computing power and the consumption of computation time. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention;

[0045] Figure 2 This is a schematic diagram of the inversion of the noise reduction diffusion implicit model of the present invention;

[0046] Figure 3 This is a schematic diagram of the attention module;

[0047] Figure 4 This is a flowchart illustrating the application of the present invention in an autonomous driving scenario;

[0048] Figure 5 This is a schematic diagram of the diffusion model calculation process. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0050] To address the challenges of low frame rate video interpolation, this invention proposes a video interpolation method based on a diffusion model and latent space token flow. This method utilizes a pre-trained, powerful image generation model for low frame rate video interpolation, fully leveraging the prior knowledge acquired by the model on the pre-training dataset. This avoids the need for extensive new data annotation and model training, while also generating visually superior interpolated video results. First, each frame of the input low frame rate video is inverted using a denoising diffusion implicit model (DDIM) to obtain a corresponding noisy latent image. Next, the noisy latent images are marked as keyframes, and linear interpolation is used to interpolate them to the desired number of frames. Then, the noisy latent images are input into the diffusion model, and at each time step, features (tokens) are extracted from each network layer of the self-attention modules. The nearest neighbor search (NN Search) algorithm is used to calculate the correlation between the inter-frame features. Next, the following operations are repeated at each time step to denoise and restore the noisy image step by step. First, the noisy latent encoding of keyframes is input into the Extended Attention module to extract its features. Next, a crucial step propagates these keyframe features in the latent space based on the previously calculated inter-frame feature correlations, forcing these features to carry the same inter-frame correlations and information redundancy as the features in the original video, thereby maintaining the temporal coherence and inter-frame consistency of the padded video. Finally, the padded noisy latent encoding image is input into the propagation model, using the propagation results as conditions to guide the network to output the denoised image, i.e., generating the padded video.

[0051] The flowchart of the method of the present invention is as follows Figure 1 As shown. The method of the present invention specifically includes:

[0052] S1. Obtain the original low frame rate video, perform noise reduction and diffusion implicit model inversion on each frame of the low frame rate video to obtain the first noise implicit code corresponding to each frame, record the frame corresponding to the first noise implicit code as the key frame, and perform linear interpolation on the first noise implicit code to obtain the second noise implicit code.

[0053] S2. Input the latent variables of all frames at each time step in the second noise latent coding into the diffusion model. The self-attention module of the diffusion model extracts the features of each frame at each time step of the second noise latent coding. The nearest neighbor is calculated through nearest neighbor search. Then, S3 and S4 are executed in each time step in the order from T to 1.

[0054] S3. Input the first noise latent code at time step t into the extended attention module to extract keyframe features;

[0055] S4. Calculate the inter-frame feature correlation based on the nearest neighbor, propagate the key frame features in the second noise latent code based on the inter-frame feature correlation, input the second noise latent code at time step t into the diffusion model, and use the propagation result as a guiding condition for denoising.

[0056] S5. After executing S3 and S4 at each time step, output the video with interpolated frames.

[0057] In S1, a noise-diffusion implicit model inversion is performed on the input low frame rate video to obtain the noise-coded latent image for each frame and then interpolate to fill in the frames. The noise-diffusion implicit model inversion encodes the original video image from the pixel space to the latent variable space, while progressively adding Gaussian noise until the image latent code completely conforms to a standard Gaussian distribution. Then, linear interpolation is performed on the noise-coded latent image to fill in the desired number of frames; the original noise-coded latent image is recorded as a keyframe. Compared to existing video editing methods that typically perform inter-frame processing directly in the pixel space, manipulating the noise image in the latent code space significantly reduces computational complexity, and explicit manipulation of latent variable features helps maintain the consistency of the generated results.

[0058] Noise reduction diffusion implicit model inversion such as Figure 2 As shown. Given the original video I = [I] with n frames. 1 ,…,I n ],set up Let be the latent variable of the i-th frame at time step t. Then, the process of adding noise during the noise reduction and diffusion implicit model inversion is as follows:

[0059]

[0060] The initial latent variable is the original frame, i.e. Gaussian noise ∈ ~N(0,I), hyperparameter It is the variance of a series of Gaussian noises, and T is the total number of time steps.

[0061] Obtain the noise hidden code corresponding to each frame of the original video. Perform linear interpolation on it:

[0062]

[0063] Where j∈(i,i+1) is the floating-point index, and λ=j-1 is the interpolation coefficient. This is the interpolated encoding. The original natural number indices and floating-point indices are remapped to natural number indices in ascending order. The mapping of the original natural number indices is labeled as the keyframe index set K, resulting in the interpolated noise latent code. Where N is the target number of frames.

[0064] For ease of expression, let's denote the noise implicit coding tensor as X, and we have:

[0065]

[0066]

[0067]

[0068] In S2, the nearest neighbor of the noisy latent-coded image in the feature space of the diffusion model is calculated. Specifically, the noisy image is input into the diffusion model, and at each time step, the features of each layer of the self-attention module in the diffusion model are output. Then, the similarity between features is calculated to obtain the matching relationship between frames. The purpose of this step is to provide a basis for subsequent feature propagation. Natural videos contain a lot of redundant information, and there are corresponding relationships between different frames. The model features also retain this redundancy, so feature-level matching can be established. Most methods do not explicitly model the redundancy of the video in the feature space. The technology proposed in this patent makes full use of the redundancy of the video feature space, which makes it possible to globally propagate keyframe features in the latent space, thereby achieving the consistency of the video after frame interpolation.

[0069] For each time step of the interpolated noise implicit coding X t The latent variables of all frames i∈[N] Input the diffusion model and extract features from its self-attention module. The specific process is as follows:

[0070] φ(X t )=A·V

[0071] Where A = Attention(Q; K), the query vector Q, key vector K, and value vector V are all results of linear transformations of the input vectors. The attention function is:

[0072]

[0073] The maximum flexibility function is defined as:

[0074]

[0075] Where z is a vector, z = [z1, z2, ..., z2] K ].

[0076] After interpolation, the noise implicitly encodes the features of all frames at each time step. It is denoted as φ(X).

[0077] Next, φ(X) is used to calculate the feature correlation of the video after frame interpolation. At each time step t, the feature of the noise latent coding corresponding to the i-th frame is: Let the features corresponding to two adjacent frames be denoted as and Where i+ represents the nearest next frame index and i- represents the nearest previous frame index, the nearest neighbor γ can be calculated through nearest neighbor search. i+ and γ i- :

[0078] γ i± [p] = arg q minD(φ(x i )[p],φ(x i± )[q])

[0079] Where p and q are the spatial positions in the feature map, and D is the cosine distance.

[0080] Attention module such as Figure 3 As shown.

[0081] In S3, feature extraction is performed on the noisy latent-coded keyframe image. Similar to the previous step, the noisy latent-coded keyframe image is input into a diffusion model with an extended attention module. At each time step, the features of each layer of the extended attention module in the diffusion model are output. The extended attention module is a component that extends self-attention to the temporal dimension, providing a simple and effective way to convey global information between keyframes. Here, the extended attention module is used to maintain a globally consistent appearance of keyframe features. The purpose of this step is to ensure consistency between different keyframes.

[0082] K is the set of keyframe indices, denoted as K. To select a tensor composed of elements with frame indices in K from the noise implicit coding after frame interpolation, each frame of the original video is treated as a keyframe. The result of the noise-diffusion implicit model inversion is then input into a diffusion model with an extended attention module, and its features are extracted from the extended attention module.

[0083]

[0084] in Query vector key vector value vector Both are the result of linear transformations of the input vectors. The extended attention function is:

[0085]

[0086] The definition of the flexible maximum value function has been given in S2.

[0087] In S4, based on the pre-calculated nearest neighbor, the inter-frame feature correlation is calculated, and keyframe features are propagated in the latent space to maintain feature redundancy and consistency across all frames. This is the core innovation of this invention. Explicit feature propagation can significantly improve the consistency of the patched video, forcing these features to have the same inter-frame correlation and information redundancy as the features in the original video, thereby maintaining the temporal coherence and inter-frame consistency of the patched video.

[0088] The nearest neighbor γ calculated in B i± The keyframe features φ(X) calculated in C are... K The propagation occurs in the interpolated video through a linear combination of φ(X). K Features corresponding to each spatial location p and frame number i in the diagram:

[0089] F γ (φ(X K ),i,p)=w i ·φ(J i+ )[γ i+ [p]]+(1-w i )·φ(J i- )[γ i- [p]

[0090] Where φ(J) i± )∈φ(X K ),w i ∈(0,1) is a scalar that is proportional to the distance between the i-th frame and its adjacent keyframes.

[0091] The noisy latent-coded image after frame interpolation is then input into the diffusion model. The propagation result is used as a condition to guide the model to output a denoised image, restoring the interpolated video. The denoising process replaces the features generated by the model, ensuring consistency between the generated and propagated features. This step maps from the coding space back to the pixel space, restoring the final video result. Some methods require additional post-processing to enhance consistency, while this method effectively maintains the temporal coherence and inter-frame consistency of the interpolated video by propagating features in the latent space based on the correlation of latent coded features.

[0092] The noise implicit code is input into the diffusion model for denoising. The feature propagation process in D is executed in each self-attention module of each layer of the diffusion model. The specific process is as follows:

[0093]

[0094] Obtain the output video after frame interpolation

[0095] The pseudocode for the method is shown in Table 1.

[0096] Table 1. Video interpolation methods based on image diffusion model and latent space feature flow.

[0097]

[0098]

[0099] The beneficial effects of the method of the present invention are as follows:

[0100] 1. No Model Training Required: Utilizing a pre-trained image diffusion model avoids extensive data annotation and model tuning. This invention is built upon a readily available and stable pre-trained image generation model, eliminating the need for additional fine-tuning and significantly reducing computational and storage costs. Compared to training a video generation model from scratch, it also avoids the need for extensive video data annotation and the time cost of model training.

[0101] 2. High Inter-Frame Consistency: By propagating keyframe features in the latent space, the redundancy between video frames is explicitly modeled and maintained, effectively eliminating inconsistencies in the generated results. Traditional generative model-based video editing methods struggle to explicitly encourage temporal consistency in the generated results, easily leading to stuttering and jitter. This invention, however, by propagating the features of noisy latent variable images in the latent space, fully utilizes redundant information in adjacent frames, significantly improving consistency.

[0102] 3. Low computational complexity: Encoding video images from pixel space to latent space significantly reduces computational complexity. Existing video editing methods typically perform inter-frame processing directly in pixel space. When faced with high-resolution image input, this often consumes substantial computational resources. Furthermore, due to the redundancy of inter-frame information in video, directly calculating pixel features can lead to significant computational waste. This invention employs a scheme that manipulates image features in latent space, avoiding direct computation in pixel space, thereby reducing the hardware requirements and computation time consumption.

[0103] 4. High Parallelism: The modules in this solution are highly independent, and each step can perform parallel computations on different batches of input data. Therefore, pipeline technology can be used to improve its throughput, resulting in good performance when dealing with large amounts of input data in practical applications. Furthermore, the algorithm's computational resource consumption is primarily concentrated on tensor computations, making it well-suited for hardware accelerators such as GPUs and TPUs to further enhance parallelism and improve computational speed in real-world applications.

[0104] 5. Realistic Image Synthesis: Based on a powerful prior image generation model, this invention achieves high-quality video frame interpolation with detailed images. Compared to traditional frame interpolation algorithms that only utilize motion information, this invention relies on the rich prior knowledge and powerful generation capabilities of a pre-trained image model to synthesize higher-quality, more logically consistent frame details, effectively avoiding problems such as blurring and artifacts.

[0105] 6. High versatility: This invention provides a simple and effective video editing framework that can be widely applied to various diffusion-based image editing techniques. The process is simple and can be combined with various diffusion-based image editing methods, making it possible to apply these methods to video tasks.

[0106] The method of this invention can be applied to the technical field of image acquisition equipment in autonomous driving systems. This invention relies on a pre-trained image diffusion model, and it is necessary to select a suitable diffusion model and its pre-training dataset in advance based on the equipment's memory and computing power, actual application scenarios, and other conditions. First, the data collected by the image acquisition equipment is processed by a continuous frame editing module to edit the video into several continuous frame videos. The purpose is to (1) ensure that the variation between input video frames is not too large, and (2) ensure that the number of input video frames is not too large. Next, based on the target frame rate and the number of frames in each edited video segment, the expected number of frames in the video after frame interpolation is calculated. Each frame number in the array corresponds to the input video with the same index number. The input video and the frame number array are input into the algorithm module in batches to obtain the video after frame interpolation. This algorithm has good parallelism and can use pipeline technology to greatly improve its throughput. Therefore, when inputting data in batches, it is not necessary to wait for the previous batch of data to produce results, so as to improve the overall performance of the module. The flowchart of the application in the autonomous driving scenario is shown below. Figure 4 As shown.

[0107] In this invention, the diffusion model calculation process is as follows: Figure 5 As shown. The inter-frame feature correlation Fy is used as a guiding condition, encoded, and then fed into the denoising step of each time step of the model. The encoding embedding methods include two types: (1) concatenating with the noisy latent code image along the channel dimension; (2) using a cross-attention module, with the correlation encoding as its key vector K and value vector V. The noisy latent code corresponding to each frame at time step T is input into the diffusion model, undergoing a total of T denoising steps, each step being a UNet containing several attention modules and residual connections. After denoising is completed, the result is input into the decoder to obtain the pixel space image of the corresponding frame.

[0108] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A video frame interpolation method based on an image diffusion model, characterized in that, The methods include: S1. Obtain the original low frame rate video, perform noise reduction and diffusion implicit model inversion on each frame of the low frame rate video to obtain the first noise implicit code corresponding to each frame, record the frame corresponding to the first noise implicit code as the key frame, and perform linear interpolation on the first noise implicit code to obtain the second noise implicit code. S2. Input the latent variables of all frames at each time step in the second noise latent coding into the diffusion model. The self-attention module of the diffusion model extracts the features of each frame at each time step of the second noise latent coding. The nearest neighbor is calculated through nearest neighbor search. Then, S3 and S4 are executed in each time step in the order from T to 1. S3. Input the first noise latent code at time step t into the extended attention module to extract keyframe features; S4. Calculate the inter-frame feature correlation based on the nearest neighbor, propagate the key frame features in the second noise latent code based on the inter-frame feature correlation, input the second noise latent code at time step t into the diffusion model, and use the propagation result as a guiding condition for denoising. S5. After executing S3 and S4 at each time step, output the video with interpolated frames; the steps of S2 are as follows: The latent variables of all frames at each time step in the second noisy latent coding are input into the diffusion model. The self-attention module of the diffusion model extracts the set of features of all frames at each time step of the second noisy latent coding. , recorded as , In each time step t, the feature of the i-th frame is ,remember The features corresponding to two adjacent frames are and i+ represents the nearest next frame index, and i- represents the nearest previous frame index. The nearest neighborhood is calculated based on nearest neighbor search. and ; The steps for S3 are as follows: Select keyframe indices from the floating-point indices to form a keyframe index set K, denoted as K. To select a tensor composed of the elements with frame indices in K from the second noisy latent coding at time step t, which is the first noisy latent coding at time step t, the first noisy latent coding at time step t is input into the extended attention module to extract keyframe features. ; The specific steps for calculating the inter-frame feature correlation based on the nearest neighbor are as follows: Linear combination For each spatial location p and frame number i in the corresponding feature map, the inter-frame feature correlation is obtained. ; Inter-frame feature correlation is as follows: in, It is a scalar that is proportional to the distance between the i-th frame and its adjacent keyframes.

2. The video frame interpolation method based on an image diffusion model according to claim 1, characterized in that, The steps for S1 are as follows: Acquire raw low frame rate video , Let a frame in a low frame rate video be represented by a symbol. For time step No. The latent variables of the frames are obtained by performing a noise-diffusion implicit model inversion on each frame of the low frame rate video to obtain the first noise latent code. The frame corresponding to the first noise hidden code is recorded as the keyframe. Linear interpolation is performed on the first noise hidden code to obtain the second noise hidden code. ,in The target frame number is denoted as the second noise hidden code. The second noise hidden coding Recorded as , The second noise hidden coding Recorded as .

3. The video frame interpolation method based on an image diffusion model according to claim 2, characterized in that, After performing the noise reduction and diffusion implicit model inversion, the time step of the first noise implicit coding No. The latent variables of the frame are: Among them, the first noise hidden code's first The initial latent variable of the frame is the first... The original frame, i.e. Gaussian noise hyperparameters It is the variance of a series of Gaussian noises. It is the total number of time steps.

4. The video frame interpolation method based on an image diffusion model according to claim 3, characterized in that, Second noise hidden coding time step No. The latent variables of the frame are: in, For floating-point index, These are interpolation coefficients. It is the second noise hidden code after interpolation.

5. The video frame interpolation method based on an image diffusion model according to claim 1, characterized in that, Nearest Neighbor and for: in p , q For spatial location in the feature map, D It is the cosine distance.

6. The video frame interpolation method based on an image diffusion model according to claim 1, characterized in that, The video after frame interpolation is as follows: ,in, in, Represents a diffusion model. This represents the process by which keyframe features propagate through the second noise latent coding. This represents the image at time step t-1 obtained by denoising the second noise implicit coding at time step t. After frame interpolation, the video is output by the diffusion model based on the inter-frame feature correlation.

Images