Online adaptive video frame insertion method and device based on inertial motion propagation

Abstract

The invention belongs to the technical field of video processing, and discloses an online self-adaptive video frame insertion method and device based on inertial motion propagation, and the method comprises the steps: obtaining plane features of a current frame and a next frame, and obtaining prior motion information which comprises an intermediate frame optical flow from the previous frame to the current frame and propagation features; obtaining motion features from the current frame to the next frame according to the propagation features and the plane features of the next frame; estimating an intermediate frame optical flow of the current frame and the next frame according to the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame and the propagation characteristics; and carrying out frame insertion on the current frame and the next frame according to the intermediate frame optical flow of the current frame and the next frame. According to the invention, accurate, reliable and efficient video frame insertion can be adaptively realized for videos with different motion amplitudes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video processing, and particularly relates to an online adaptive video interpolation method and device based on inertial motion propagation. BACKGROUND

[0002] Video interpolation aims to generate intermediate frames from two edge frames, and has a very broad application space. In recent years, thanks to the rapid development of deep learning, video interpolation technology has also made great progress. Among them, the video interpolation algorithm based on optical flow estimation aims to estimate the optical flow from the edge frame to the intermediate frame to realize the generation of the intermediate frame, which has higher efficiency than the algorithm that directly generates the intermediate frame, and has higher flexibility when inserting multiple frames.

[0003] Due to the uncertainty of inter-frame motion, existing video interpolation algorithms based on optical flow estimation often use various assumptions (such as linear, quadratic or cubic assumptions) to model inter-frame motion, but in complex scenes, the generated intermediate frame optical flow is often not accurate enough, especially when dealing with large-scale motion or high-resolution videos, the generated intermediate frame may have obvious distortion or artifacts. This problem is particularly evident in video moving objects, and the distortion of the static part is lighter, while the dynamic part is usually severely affected. For example, Jiang et al. introduced a local linear assumption in the document "Super slomo: High quality estimation of multiple intermediate frames for video interpolation," to estimate the optical flow of the intermediate frame motion by simply interpolating the inter-frame motion. Xu et al. proposed in the document "Quadratic video interpolation, Advances in Neural Information Processing Systems" to extract high-order motion information from multiple frames to model the motion of the intermediate frame. Although this method further improves the interpolation effect, it still faces obvious degradation in large-scale motion and high-resolution video interpolation.

[0004] Therefore, how to adaptively realize accurate, reliable and efficient video interpolation for videos with different motion amplitudes is still a technical problem at present. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the present application provides an online adaptive video interpolation method and device based on inertial motion propagation.

[0006] The technical problem to be solved by the present application is realized by the following technical scheme: An online adaptive video frame interpolation method based on inertial motion propagation, comprising: Obtaining plane features of the current frame and the next frame, and obtaining prior motion information; the prior motion information includes: intermediate frame optical flow from the previous frame to the current frame and propagation features; the propagation features are obtained by cross attention on the plane features of the current frame and the previous frame; According to the propagation features and the plane features of the next frame, the motion features from the current frame to the next frame are obtained; According to the motion features from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame and the propagation features, the intermediate frame optical flow between the current frame and the next frame is estimated; According to the intermediate frame optical flow between the current frame and the next frame, the current frame and the next frame are interpolated.

[0007] Optionally, according to the propagation features and the plane features of the next frame, the motion features from the current frame to the next frame are obtained, comprising: The propagation features and the plane features of the next frame are spliced in the channel dimension, and the spliced features are compressed in the channel by convolution operation, to obtain attention features with the same number of channels as the plane features; The attention features and the plane features of the next frame are cross-attentive to obtain an attention map; The attention map is converted into the motion features from the current frame to the next frame through a learnable linear transformation.

[0008] Optionally, according to the motion features from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame and the propagation features, the intermediate frame optical flow between the current frame and the next frame is estimated, comprising: The plane features of the next frame, the motion features from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame and the propagation features are input into a pre-trained optical flow estimator, and the intermediate frame optical flow between the current frame and the next frame is output by the optical flow estimator; wherein the optical flow estimator is constructed based on convolution layers.

[0009] Optionally, according to the intermediate frame optical flow between the current frame and the next frame, the current frame and the next frame are interpolated, comprising: According to the plane features of the current frame and the next frame, and the intermediate frame optical flow between the current frame and the next frame, the intermediate frame residual between the current frame and the next frame is determined; According to the intermediate frame optical flow between the current frame and the next frame and the intermediate frame residual, the intermediate frame between the current frame and the next frame is generated.

[0010] Optionally, the optical flow estimator is also used to output a mask for processing occlusions; According to the intermediate frame optical flow between the current frame and the next frame and the intermediate frame residual, the intermediate frame between the current frame and the next frame is generated, comprising: ; in, Indicates the current frame. Indicates the next frame. This represents the optical flow between the next frame and the current frame. This represents the optical flow between the current frame and the next frame. This refers to the mask. This represents the intermediate frame residual between the current frame and the next frame. This represents a warp operation that performs a geometric transformation on an image. This indicates the frame between the current frame and the next frame.

[0011] Optionally, based on the planar features of the current frame and the next frame, and the intermediate frame optical flow between the current frame and the next frame, the intermediate frame residual between the current frame and the next frame is determined, including: The planar features of the current frame and the next frame, as well as the optical flow of the intermediate frame between the current frame and the next frame, are input into a preset correction network, which is then used to output the intermediate frame residual between the current frame and the next frame.

[0012] Optionally, the method further includes: Each time, three consecutive frames are extracted from the video stream, and the first and third frames of the three consecutive frames are interpolated to obtain the reconstructed second frame; The optical flow estimator is subjected to online self-supervised learning based on the second frame of the three consecutive frames and the reconstructed second frame.

[0013] Optionally, the step of performing online self-supervised learning on the optical flow estimator based on the second frame of the three consecutive frames and the reconstructed second frame includes: Based on the second frame of the three consecutive frames and the reconstructed second frame, the loss is calculated using a preset loss function; Online self-supervised learning is performed on the linear transformation and the optical flow estimator based on the loss; The loss function is: ; in, This represents the second frame of the reconstruction. This refers to the second frame in the three consecutive frames. The intermediate frame is obtained by optical flow warping of the first and third frames out of the three consecutive frames. Represents the Laplace loss function. l It indicates a loss.

[0014] Optionally, In the generation The following are generated during the process: , ; in, This refers to the first frame. This refers to the third frame. This represents the optical flow of the intermediate frames between the 3rd frame and the 1st frame. This represents the optical flow of the intermediate frames between the first and third frames. This refers to the mask. This represents the intermediate frame residual between the third frame and the first frame.

[0015] The present invention also provides an online adaptive video frame interpolation device based on inertial motion propagation, comprising: The acquisition module is used to acquire the planar features of the current frame and the next frame, and to acquire prior motion information; the prior motion information includes: the optical flow and propagation features of the intermediate frames from the previous frame to the current frame; the propagation features are obtained by performing cross-attention on the planar features of the current frame and the previous frame; The motion feature generation module is used to obtain the motion features from the current frame to the next frame based on the propagation features and the planar features of the next frame; The optical flow estimation module is used to estimate the intermediate frame optical flow between the current frame and the next frame based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics. The frame interpolation module is used to interpolate the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame.

[0016] The online adaptive video frame interpolation method based on inertial motion propagation provided by this invention utilizes the characteristics of motion inertia to propagate previous motion as the motion prior of the current two frames to help estimate the optical flow of the intermediate frame. At the same time, it uses the attention map of the propagation features and the current frame features to perform implicit motion propagation to help the current frame features perform motion modeling.

[0017] Furthermore, when this invention is actually deployed on video streams, it utilizes the intermediate frame generation process to perform self-supervised learning for video frame interpolation, ensuring the robustness of the video frame interpolation algorithm to motion when deployed on different video streams.

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0019] Figure 1 This is a flowchart of an online adaptive video frame interpolation method based on inertial motion propagation provided in an embodiment of the present invention; Figure 2The article compares the computational complexity of existing video frame interpolation algorithms with that of the frame interpolation method of this invention in terms of feature extraction. Figure 3 The paper compares the differences between existing video frame interpolation algorithms and the frame interpolation method of this invention in actual testing. Figure 4 The paper compares the differences in motion modeling between existing video frame interpolation algorithms and the frame interpolation method of this invention; Figure 5 The paper presents a qualitative comparison of the present invention with several existing frame interpolation algorithms on the Vimeo90k dataset; Figure 6 The image shows a visual comparison of the frame interpolation results of the present invention with those of several existing frame interpolation algorithms. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0021] To achieve accurate, reliable, and efficient video frame interpolation adaptively for videos with varying motion amplitudes, this invention provides an online adaptive video frame interpolation method based on inertial motion propagation. (See [link to relevant documentation]). Figure 1 As shown, the method includes the following steps: S10. Obtain the planar features of the current frame and the next frame, and obtain prior motion information; the prior motion information includes: the optical flow and propagation features of the intermediate frames from the previous frame to the current frame; the propagation features are obtained by performing cross-attention on the planar features of the current frame and the previous frame.

[0022] Here, the current frame can be extracted using a pre-defined video stream feature extraction module. and the next frame planar features ( , ).

[0023] For example, the feature extraction module proposed in the paper "G. Zhang, Y. Zhu, H. Wang, Y. Chen, G. Wu, and L. Wang, “Extracting motion and appearance via inter-frame attention for efficient video frame interpolation,” in 2023 IEEE / CVF Conference on ComputerVision and Pattern Recognition (CVPR), 2023, pp. 5682–5692” can be used. As a video stream feature extraction module in this invention, it performs feature extraction on video frames in the video stream to obtain their planar features. If all video frames in the video stream are represented as... Then, the corresponding planar features obtained by feature extraction on them can be represented as .

[0024] In practical applications, planar features of all frames in a video stream can be extracted at once, or planar features can be extracted separately during frame-by-frame processing; both are possible.

[0025] In step S10, if the current frame For the first frame of the video stream, the prior motion information (optical flow of the intermediate frames from the previous frame to the current frame) is then used. and characteristics of transmission This can be obtained through zero initialization; if the current frame If it is the second frame or a later frame in the video stream, then the prior motion information is based on the previous frame. and the current frame This is generated during the frame interpolation process, and this part will be explained in detail in subsequent steps.

[0026] S20. Based on the propagation characteristics and the planar characteristics of the next frame, obtain the motion characteristics from the current frame to the next frame.

[0027] Specifically, regarding the characteristics of transmission and planar features of the next frame The concatenation is performed along the channel dimension, and the concatenated features are then compressed through convolution to obtain the number of channels and the planar features. The same attention features; then, the same attention features and the planar features of the next frame. Obtain an attention map by performing cross-attention. Next, the attention map is transformed using a learnable linear transformation. Convert to motion features from the current frame to the next frame .

[0028] Here, the learnable linear transformation refers to a parameterized linear mapping f(x) = Wx + b (or without bias b), where the matrix W (and vector b) are learned through training. x is the attention map to be linearly transformed, and f(x) is the current frame obtained after the linear transformation. To the next frame Motion characteristics .

[0029] S30. Estimate the intermediate frame optical flow between the current frame and the next frame based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics.

[0030] Here, the current frame With the next frame Intermediate frame optical flow Includes: the current frame To the next frame Intermediate frame optical flow and the next frame Up to the current frame Intermediate frame optical flow .

[0031] In step S30, based on the current frame To the next frame Motion characteristics Optical flow between the previous frame and the current frame and characteristics of transmission Estimate the current frame With the next frame Intermediate frame optical flow This can include: the planar features of the next frame. Motion features from the current frame to the next frame Optical flow between the previous frame and the current frame and characteristics of transmission The input is fed into a pre-trained optical flow estimator, which then outputs the optical flow of the intermediate frame between the current frame and the next frame. The optical flow estimator can be constructed based on convolutional layers.

[0032] For example, in one implementation, the optical flow estimator used in this invention can be represented as: ; in, This represents a two-layer convolution operation, where the input feature channel dimension consists of four features. The sum of the channel dimensions, It is a mask output by the optical flow estimator to handle occlusions.

[0033] S40. Interpolate the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame.

[0034] Specifically, step S40 includes: (1) Based on the planar features of the current frame and the next frame And the optical flow between the current frame and the next frame. Determine the intermediate frame residual between the current frame and the next frame. .

[0035] Specifically, the planar features of the current frame and the next frame are... And the optical flow between the current frame and the next frame. The input is fed into a preset correction network, which then outputs the intermediate frame residual between the current frame and the next frame. .

[0036] The correction network can be directly adopted from the paper "G. Zhang, Y. Zhu, H. Wang, Y. Chen, G. Wu, and L. Wang, “Extracting motion and appearance via inter-frame attention for efficient video frame interpolation,” in 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5682–5692." Of course, it is not limited to this.

[0037] (2) Generate the intermediate frame between the current frame and the next frame based on the optical flow and the intermediate frame residual between the current frame and the next frame.

[0038] Specifically, based on the optical flow and residual between the intermediate frames of the current frame and the next frame, an intermediate frame between the current frame and the next frame is generated, including: ; In the formula, Indicates the current frame. Indicates the next frame. This represents the optical flow between the next frame and the current frame. This represents the optical flow between the current frame and the next frame. This represents the mask output by the optical flow estimator. This represents the intermediate frame residual between the current frame and the next frame. This represents a warp operation that performs a geometric transformation on an image. This indicates the intermediate frame between the current frame and the next frame, that is, the interpolation between the current frame and the next frame.

[0039] The above completes the frame interpolation for the current frame and the next frame. Then, the remaining frames in the video stream can continue to be interpolated using the methods shown in steps S10-S40. It is understood that when continuing with the next frame... and the next frame During frame interpolation, the intermediate frame optical flow estimated in step S30 can be used as the intermediate frame optical flow in the prior motion information, and the propagation features in the prior motion information will also be updated accordingly. and The cross-attention of planar features. Therefore, the frame interpolation method provided by this invention can be intuitively represented by the following formula: ; in, Indicates to and Prior motion information acquired during frame interpolation This indicates that it will continue to spread to the next person. and Prior motion information imported during frame interpolation.

[0040] The online adaptive video frame interpolation method based on inertial motion propagation provided by this invention utilizes the characteristics of motion inertia to propagate previous motion as the motion prior of the current two frames to help estimate the optical flow of the intermediate frames. At the same time, it uses the attention map of the propagation features and the current frame features to perform implicit motion propagation to help the current frame features perform motion modeling.

[0041] In one embodiment, the online adaptive video frame interpolation method based on inertial motion propagation provided by the present invention may further include the following steps: (i) Extract three consecutive frames from the video stream each time, and interpolate the first frame and the third frame of the three consecutive frames to obtain the reconstructed second frame; Here, the frame interpolation method used when interpolating the first and third frames in three consecutive frames can refer to the method shown in steps S10 to S40.

[0042] (ii) The optical flow estimator is subjected to online self-supervised learning based on the second frame of the three consecutive frames and the reconstructed second frame.

[0043] Specifically, based on the second frame out of three consecutive frames and the reconstructed second frame, a pre-defined loss function is used to calculate the loss. Then, online self-supervised learning is performed on the optical flow estimator based on this loss, that is, the learnable parameters of the optical flow estimator are adjusted according to the loss, so that the parameters are adjusted and optimized in the direction of loss convergence. After multiple training sessions, the performance of the optical flow estimator can be optimized to adapt it to the current video stream. Similarly, the learnable linear transformation matrix W (and vector b) in step S20 can also be adjusted and optimized based on this loss.

[0044] The loss function is as follows: ; in, This represents the calculated loss. This represents the second frame of the reconstruction. This indicates the second frame out of three consecutive frames. By analyzing the first frame out of three consecutive frames and the 3rd frame The intermediate frame obtained by optical flow warping. Denotes the Laplace loss function, for calculating For example, the expression for the Laplace loss function is: ; in, This represents the average pooling operation, where the number of average pooling operations is . As you can see, the smaller the resolution, the greater the weight assigned. This helps maintain the consistency of motion, resulting in a smooth and natural video.

[0045] Preferably, in one implementation, It can be generated This is generated during the process. Specifically, step S40 shows the formula for generating intermediate frames. Based on this formula, the process of interpolating the first and third frames of three consecutive frames extracted from the video stream can be expressed as: ; in, This represents the first frame out of three consecutive frames extracted from the video stream. This represents the third frame out of three consecutive frames extracted from the video stream. This represents the optical flow between the 3rd frame and the 1st frame. This indicates the optical flow between the first and third frames. This indicates the mask output by the optical flow estimator during the frame interpolation process of frames 1 to 3 using steps S10 to S40. This represents the intermediate frame residual between the 3rd frame and the 1st frame, which is also generated during the interpolation process.

[0046] and That is, the first term on the right side of the equals sign in the above expression, that is: .

[0047] In the above embodiments incorporating online self-supervised learning, video frames in the video stream are reconstructed by generating intermediate frames, and self-supervised learning for video frame interpolation is performed based on this reconstruction. This ensures the robustness of the video frame interpolation algorithm to motion across different video streams. In comparison, existing methods that use hybrid expert models to improve the robustness of video frame interpolation significantly increase the computational cost. The online self-supervised learning approach of this invention eliminates the need to construct a hybrid expert model, enabling the video frame interpolation method to handle videos with varying motion amplitudes and allowing the algorithm to adapt to the motion amplitude of the video stream online.

[0048] In practical applications, when the method of this invention is actually deployed to a video stream, the self-supervised learning of video frame interpolation can be performed using the methods shown in steps (i) to (ii), thereby ensuring the robustness of the video frame interpolation algorithm to motion when deployed to different video streams. For example, the self-supervised learning of video frame interpolation can be performed first using the methods shown in steps (i) to (ii), so that the parameters of the optical flow estimator adapt to the current input video stream, and then the actual frame interpolation of the video stream can be performed according to the process shown in steps S10 to S40. During the self-supervised learning stage, video frames can be extracted from the video stream multiple times, with each extraction consisting of three consecutive frames, and the process shown in steps (i) to (ii) is executed each time. Furthermore, even after entering the actual frame interpolation stage, the self-supervised learning process using the methods shown in steps (i) to (ii) can be performed concurrently; both are feasible and can improve the frame interpolation effect.

[0049] Figure 2 The paper compares the computational complexity of feature extraction between existing video frame interpolation algorithms and the frame interpolation method of this invention. In (a), the yellow portion represents the existing video frame interpolation method performing feature extraction twice per frame when processing the video stream, resulting in redundant extraction and increased computational redundancy. Figure 2 (b) in the figure represents the method of the present invention, which uses a structure similar to a cyclic network to process video streams and does not have computational redundancy.

[0050] Figure 3 The article compares the differences in the amount of data processed between existing video frame interpolation algorithms and the frame interpolation method of the present invention. Here, (c) indicates that the existing method improves the frame interpolation performance by augmenting data. This approach leads to a significant increase in the computational complexity of the algorithm and ignores the efficiency required for the actual deployment of the algorithm to the video stream. Figure 3 In the diagram, (d) represents the testing process of the present invention in the experiment, and this testing process is consistent with the actual deployment of the model.

[0051] Figure 4This paper compares the differences in motion modeling between existing video frame interpolation algorithms and the frame interpolation method of this invention. In existing video frame interpolation frameworks, mainstream methods use predefined assumptions to model inter-frame motion. These incorrect predefined motion pattern assumptions lead to inconsistent deformation between consecutive frames. This makes it difficult for the deformation features of new frames to align, resulting in distortion and blurring, especially when dealing with large-scale and complex motion common in high-resolution videos (such as 4K). Figure 4 As shown in (e), existing models estimate inter-frame motion linearly during the inference phase. The estimated position, located midway between two frames, is inaccurate, leading to incorrect object position estimations due to linear motion modeling. Furthermore, video frame interpolation algorithms often lack robustness across different motion amplitudes, failing to maintain good performance simultaneously. To address this, existing algorithms employ hybrid expert models, training different experts for different motion amplitudes and ultimately fusing their outputs to ensure robustness. However, this approach also increases the number of parameters and computational cost. (See also...) Figure 4 As shown in (f), the method of the present invention incorporates previous motions when performing motion estimation, and uses the motion inertia to perform physical estimation, which is more accurate.

[0052] Based on Figures 2-4 Analysis shows that the present invention has the following advantages compared with existing methods: First, compared with existing video frame interpolation methods, this invention uses the characteristics of motion inertia to estimate the optical flow of intermediate frames, providing a more reliable solution for different motion amplitudes.

[0053] Secondly, compared to existing video interpolation models, the proposed video interpolation algorithm exhibits stronger generalization performance and can handle more complex interpolation scenarios, such as high-resolution or high-motion videos. This is because, on the one hand, the invention incorporates motion information during optical flow estimation, enabling the interpolation process to adapt to videos with varying motion amplitudes and demonstrating strong generalization ability; on the other hand, the use of online self-supervised learning allows the interpolation effect to adaptively optimize with the input video stream, further improving generalization performance. Furthermore, due to its recurrent network-like structure, the invention offers faster inference speeds and lower computational costs when applied to video streams. Simultaneously, the method can adaptively handle videos with varying motion amplitudes, ensuring the algorithm's robustness.

[0054] The following simulation experiments will further illustrate the beneficial effects of the present invention.

[0055] Simulation conditions: On a system with an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz CPU, an NVIDIA RTX2080Ti graphics card, and an Ubuntu 16.04 operating system, Python was used as the programming language, PyTorch was used as the deep learning network framework, and Vimeo90K was used as the training dataset. This dataset provides diverse challenges for evaluating different video interpolation methods, ranging from simple to extremely complex motion scenes. The dataset contains approximately 90,000 video clips. Each video clip has 7 frames, all from different dynamic scenes, with a resolution of 720p, suitable for video interpolation tasks. This training dataset provides rich motion patterns and image features, helping the model effectively reduce motion distortion and blur when generating intermediate frames.

[0056] During training, the Vimeo 90K dataset was used, with video sequences of frames 1, 3, 5, and 7 (i.e., frame pairs) selected as input and frames 2, 4, and 6 as output. Each video frame was cropped into 256×256 image patches, and data augmentation was performed through random flipping, time reversal, and rotation. During optimization, the AdamW optimizer was used, with parameters set as follows: and use weight decay During training, a 2000-step warm-up of the learning rate is performed first, increasing the learning rate from 0 to... Then, a cosine annealing learning rate scheduling method is used to gradually increase the learning rate over the next 300 epochs. Down to During training, observe the trend of training loss. When the training loss tends to stabilize, or when the number of iterations of the model reaches the preset iteration limit, stop training and save the network structure and corresponding model parameters.

[0057] After training the model, the test sets used covered different resolutions from 2K to 4K. These included: GOPRO, Vimeo90K, Xiph, YouTube, and Davis. GOPRO, Vimeo90K, and YouTube are commonly used video frame interpolation evaluation datasets for evaluating frame interpolation in general scenes, while Davis and Xiph are datasets specifically designed for evaluating frame interpolation in high-motion and high-resolution videos. The references for these datasets are as follows: “M. Choi, H. Kim, B. Han, N. Xu, and K. M. Lee, “Channel Attention IsAll You Need for Video Frame Interpolation,” Proceedings of the AAAIConference on Artificial Intelligence, vol. 34, no. 07, pp. 10 663–10 671,Apr. 2020, number: 07. [Online]. Available: https: / / ojs.aaai.org / index.php / AAAI / article / view / 6693”; “W. Bao, W.-S. Lai, X. Zhang, Z. Gao, and M.-H. Yang, “Memc-net:Motion estimation and motion compensation driven neural network for videointerpolation and enhancement,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. 43, no. 3, pp. 933–948, 2021.”; “H. Sim, J. Oh, and M. Kim, “Xvfi: extreme video frameinterpolation,” in 2021 IEEE / CVF International Conference on Computer Vision(ICCV), 2021, pp. 14 469–14 478.”; “C. Liu, G. Zhang, R. Zhao, and L. Wang, “Sparse global matching for video frame interpolation with large motion,” in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp.19 125–19 134.

[0058] The comparison methods used include: UPR-Net (CVPR 2023), EMA-VFI (CVPR 2023), AMT (CVPR2023), VFIMamba (Nips 2024), EMA-VFI[D], EMA-VFI[D,R] (ECCV 2024), and BIM-VFI (CVPR2025), all of which are the most advanced methods in the past two years.

[0059] Experimental results: Two consecutive frames are selected from the test set and input into the trained video frame interpolation network. The output of the model is the intermediate frame.

[0060] Table 1 shows the comparison results of the frame interpolation performance of each model in general video stream scenarios: Table 1

[0061] Among them, the objective evaluation performance indicators for frame interpolation performance include PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and LPIPS (Visual Perception Indicator).

[0062] Table 2 shows the comparison results of each model in terms of multi-frame interpolation performance (PSNR / SSIM): Table 2

[0063] Table 3 shows the comparison results of the frame interpolation performance (PSNR / SSIM) of each model in high-resolution scenes: Table 3

[0064] Table 4 shows the comparison results of various computational load indicators between the method of the present invention and existing algorithms on 720P video: Table 4

[0065] As can be seen from Tables 1 to 4, this invention not only has better PSNR and SSIM in regular video streams, high-motion video streams, and high-resolution video streams, but also has much better visual perception than existing algorithms. In addition, our algorithm has the second fewest FLOPS and the fastest inference speed.

[0066] In addition, the simulation also compared the qualitative performance of each model on the Vimeo90k dataset, such as... Figure 5 As shown, the left side shows the actual intermediate frame and its local magnified area, and the right side shows the interpolation results of each model in turn. In order to more clearly present the difference between the prediction and the actual value, an error heatmap is also displayed below the inserted intermediate frame, where the bright area represents a large deviation and the black area represents a perfect match. Figure 5 In the figure, (a) is the frame interpolation result of UPRNet-LARGE, (b) is the frame interpolation result of AMT-G, (c) is the frame interpolation result of EMA-VFI, (d) is the frame interpolation result of EMA-VFI[D], (e) is the frame interpolation result of EMA-VFI[D,R], (f) is the frame interpolation result of VFIMamba, (g) is the frame interpolation result of BiM-VFI

[39] , and (h) is the frame interpolation result of the present invention. Figure 6 A visual comparison of the frame interpolation results from various frame interpolation algorithms is shown. From Figure 5 and Figure 6 As can be seen, compared with existing frame interpolation algorithms, the present invention achieves better frame interpolation results.

[0067] The frame interpolation method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0068] Based on the same inventive concept, embodiments of the present invention also provide an online adaptive video frame interpolation device based on inertial motion propagation. This device can be a software product for implementing online video frame interpolation. The device includes: an acquisition module, a motion feature generation module, an optical flow estimation module, and a frame interpolation module.

[0069] The acquisition module is used to acquire the planar features of the current frame and the next frame, and to acquire prior motion information. The prior motion information includes: the optical flow and propagation features of the intermediate frames from the previous frame to the current frame. The propagation features are obtained by performing cross-attention on the planar features of the current frame and the previous frame. The motion feature generation module is used to obtain the motion features from the current frame to the next frame based on the propagation features and the planar features of the next frame; The optical flow estimation module is used to estimate the intermediate frame optical flow between the current frame and the next frame based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics. The frame interpolation module is used to interpolate the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame.

[0070] Optionally, the motion feature generation module is specifically used for: The propagation feature and the planar feature of the next frame are concatenated along the channel dimension, and the concatenated feature is compressed by channel through convolution operation to obtain an attention feature with the same number of channels as the planar feature. An attention map is obtained by performing cross-attention on the attention features and the planar features of the next frame; The attention map is transformed into motion features from the current frame to the next frame through a learnable linear transformation.

[0071] Optionally, the optical flow estimation module is specifically used for: The planar features of the next frame, the motion features from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation features are input into a pre-trained optical flow estimator, which is then used to output the intermediate frame optical flow between the current frame and the next frame; wherein, the optical flow estimator is constructed based on convolutional layers.

[0072] Optionally, the frame interpolation module is specifically used for: Based on the planar features of the current frame and the next frame, and the intermediate frame optical flow between the current frame and the next frame, determine the intermediate frame residual between the current frame and the next frame. The intermediate frames between the current frame and the next frame are generated based on the optical flow and residual of the intermediate frames between the current frame and the next frame.

[0073] Optionally, the optical flow estimator is further used to output a mask for processing occlusions; the frame interpolation module generates an intermediate frame between the current frame and the next frame based on the optical flow and residual of the intermediate frame between the current frame and the next frame, including: ; in, Indicates the current frame. Indicates the next frame. This represents the optical flow between the next frame and the current frame. This represents the optical flow between the current frame and the next frame. This refers to the mask. This represents the intermediate frame residual between the current frame and the next frame. This represents a distortion operation that performs a geometric transformation on an image. This indicates the frame between the current frame and the next frame.

[0074] Optionally, the frame interpolation module determines the intermediate frame residual between the current frame and the next frame based on the planar features of the current frame and the next frame, as well as the intermediate frame optical flow between the current frame and the next frame, including: The planar features of the current frame and the next frame, as well as the optical flow of the intermediate frame between the current frame and the next frame, are input into a preset correction network, which is then used to output the intermediate frame residual between the current frame and the next frame.

[0075] Optionally, the device further includes: an online self-supervised learning module; The online self-supervised learning module is used to extract three consecutive frames from the video stream each time, interpolate the first and third frames of the three consecutive frames to obtain the reconstructed second frame, and perform online self-supervised learning on the optical flow estimator based on the second frame of the three consecutive frames and the reconstructed second frame.

[0076] Optionally, the online self-supervised learning module performs online self-supervised learning on the optical flow estimator based on the second frame of the three consecutive frames and the reconstructed second frame, including: calculating the loss using a preset loss function based on the second frame of the three consecutive frames and the reconstructed second frame; and performing online self-supervised learning on the linear transformation and the optical flow estimator based on the loss. The loss function is: ; in, This represents the second frame of the reconstruction. This refers to the second frame in the three consecutive frames. The intermediate frame is obtained by optical flow warping of the first and third frames out of the three consecutive frames. Represents the Laplace loss function. l It indicates a loss.

[0077] Optionally, In the generation The following are generated during the process: , ; in, This refers to the first frame. This refers to the third frame. This represents the optical flow of the intermediate frames between the 3rd frame and the 1st frame. This represents the optical flow of the intermediate frames between the first and third frames. This refers to the mask. This represents the intermediate frame residual between the third frame and the first frame.

[0078] It should be noted that, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiment.

[0079] In this invention, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0080] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0081] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (devices), or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An online adaptive video frame interpolation method based on inertial motion propagation, characterized in that, include: Obtain planar features of the current frame and the next frame, and obtain prior motion information; The prior motion information includes: the optical flow and propagation features of the intermediate frames from the previous frame to the current frame; the propagation features are obtained by performing cross-attention on the planar features of the current frame and the previous frame; Based on the propagation characteristics and the planar characteristics of the next frame, the motion characteristics from the current frame to the next frame are obtained; Based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics, estimate the intermediate frame optical flow between the current frame and the next frame; Frame interpolation is performed between the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame.

2. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 1, characterized in that, Based on the propagation characteristics and the planar characteristics of the next frame, the motion characteristics from the current frame to the next frame are obtained, including: The propagation feature and the planar feature of the next frame are concatenated along the channel dimension, and the concatenated feature is compressed by channel through convolution operation to obtain an attention feature with the same number of channels as the planar feature. An attention map is obtained by performing cross-attention on the attention features and the planar features of the next frame; The attention map is transformed into motion features from the current frame to the next frame through a learnable linear transformation.

3. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 1, characterized in that, Based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics, the intermediate frame optical flow between the current frame and the next frame is estimated, including: The planar features of the next frame, the motion features from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation features are input into a pre-trained optical flow estimator, which is then used to output the intermediate frame optical flow between the current frame and the next frame; wherein, the optical flow estimator is constructed based on convolutional layers.

4. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 3, characterized in that, Frame interpolation is performed between the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame, including: Based on the planar features of the current frame and the next frame, and the intermediate frame optical flow between the current frame and the next frame, determine the intermediate frame residual between the current frame and the next frame. The intermediate frames between the current frame and the next frame are generated based on the optical flow and residual of the intermediate frames between the current frame and the next frame.

5. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 4, characterized in that, The optical flow estimator is also used to output a mask for processing obstructions; The step of generating an intermediate frame between the current frame and the next frame based on the intermediate frame optical flow and intermediate frame residual between the current frame and the next frame includes: ； in, Indicates the current frame. Indicates the next frame. This represents the optical flow between the next frame and the current frame. This represents the optical flow between the current frame and the next frame. This refers to the mask. This represents the intermediate frame residual between the current frame and the next frame. This represents a distortion operation that performs a geometric transformation on an image. This indicates the frame between the current frame and the next frame.

6. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 4, characterized in that, Based on the planar features of the current frame and the next frame, and the intermediate frame optical flow between the current frame and the next frame, determine the intermediate frame residual between the current frame and the next frame, including: The planar features of the current frame and the next frame, as well as the optical flow of the intermediate frame between the current frame and the next frame, are input into a preset correction network, which is then used to output the intermediate frame residual between the current frame and the next frame.

7. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 5, characterized in that, The method further includes: Each time, three consecutive frames are extracted from the video stream, and the first and third frames of the three consecutive frames are interpolated to obtain the reconstructed second frame; The optical flow estimator is subjected to online self-supervised learning based on the second frame of the three consecutive frames and the reconstructed second frame.

8. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 7, characterized in that, The step of performing online self-supervised learning on the optical flow estimator based on the second frame of the three consecutive frames and the reconstructed second frame includes: Based on the second frame of the three consecutive frames and the reconstructed second frame, the loss is calculated using a preset loss function; The optical flow estimator is subjected to online self-supervised learning based on the loss; The loss function is: ； in, This represents the second frame of the reconstruction. This refers to the second frame in the three consecutive frames. The intermediate frame is obtained by optical flow warping of the first and third frames out of the three consecutive frames. Represents the Laplace loss function. l It indicates a loss.

9. The online adaptive video frame interpolation method based on inertial motion propagation according to claim 8, characterized in that, In the generation The following are generated during the process: ， ； in, This refers to the first frame. This refers to the third frame. This represents the optical flow of the intermediate frames between the 3rd frame and the 1st frame. This represents the optical flow of the intermediate frames between the first and third frames. This refers to the mask. This represents the intermediate frame residual between the third frame and the first frame.

10. An online adaptive video frame interpolation device based on inertial motion propagation, characterized in that, include: The acquisition module is used to acquire planar features of the current frame and the next frame, and to acquire prior motion information; The prior motion information includes: the optical flow and propagation features of the intermediate frames from the previous frame to the current frame; the propagation features are obtained by performing cross-attention on the planar features of the current frame and the previous frame; The motion feature generation module is used to obtain the motion features from the current frame to the next frame based on the propagation features and the planar features of the next frame; The optical flow estimation module is used to estimate the intermediate frame optical flow between the current frame and the next frame based on the motion characteristics from the current frame to the next frame, the intermediate frame optical flow from the previous frame to the current frame, and the propagation characteristics. The frame interpolation module is used to interpolate the current frame and the next frame based on the optical flow of the intermediate frame between the current frame and the next frame.

Images