Video frame insertion method and device based on plane feature-guided intermediate frame optical flow

Abstract

The present invention discloses a video interpolation method and device based on plane feature-guided intermediate frame optical flow. The method comprises: extracting plane features, layered features, and first motion information from the first and second frames of a video to be inserted as intermediate frames; predicting a motion offset and motion coefficient based on the plane features; using the motion offset and motion coefficient to model the inter-frame motion; determining the second motion information based on the first motion information, the motion offset, and the motion coefficient; generating an intermediate frame optical flow based on the second motion information; determining an intermediate frame residual based on the layered features and the intermediate frame optical flow; and generating an intermediate frame based on the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual. The present invention can improve the temporal resolution of videos, especially in scenes with large-scale motion or complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and image processing, and in particular relates to a video frame insertion method and device based on plane feature-guided intermediate frame optical flow. Background Art

[0002] Video interpolation aims to generate intermediate frames from two edge frames and has a broad application space. In recent years, thanks to the rapid development of deep learning, video interpolation technology has made significant strides. Among them, video interpolation algorithms based on optical flow estimation aim to generate intermediate frames by estimating the optical flow from edge frames to intermediate frames. Compared to algorithms that directly generate intermediate frames, these algorithms are more efficient and offer greater flexibility when interpolating multiple frames.

[0003] Existing video interpolation methods often use linear assumptions to model inter-frame motion. However, in complex scenes, the generated intermediate frames are often inaccurate. Especially when dealing with large-scale motion or complex scenes, the generated intermediate frames may exhibit noticeable distortion or artifacts. This problem is particularly pronounced in dynamic video content, where static parts are less distorted, while dynamic parts are often more severely affected. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a video frame insertion method and device based on plane feature-guided intermediate frame optical flow.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] A video frame interpolation method based on plane feature-guided intermediate frame optical flow, comprising:

[0007] Extracting planar features, hierarchical features, and first motion information from a first frame and a second frame in a video where intermediate frames are to be inserted; the first motion information is inter-frame motion information between the first frame and the second frame;

[0008] Predicting a motion offset and a motion coefficient based on the planar features; the motion offset and the motion coefficient are used to model inter-frame motion;

[0009] Determining second motion information according to the first motion information, the motion offset, and the motion coefficient; the second motion information includes inter-frame motion information between the first frame and the intermediate frame, and inter-frame motion information between the second frame and the intermediate frame;

[0010] generating an intermediate frame optical flow according to the second motion information;

[0011] determining an intermediate frame residual according to the hierarchical features and the intermediate frame optical flow;

[0012] An intermediate frame is generated according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual.

[0013] Optionally, predicting the motion offset and the motion coefficient according to the planar feature includes:

[0014] According to the planar features, a pre-trained trajectory estimator is used to predict motion bias and motion coefficients; wherein the trajectory estimator is constructed based on a convolutional layer.

[0015] Optionally, determining the second motion information according to the first motion information, the motion offset, and the motion coefficient includes:

[0016]

[0017] Wherein, W represents the motion coefficient, B represents the motion offset, and t represents the interpolation position of the intermediate frame. represents inter-frame motion information from the first frame to the second frame, Indicates inter-frame motion information from the second frame to the first frame, the first motion information includes and represents inter-frame motion information from the first frame to the intermediate frame, represents the inter-frame motion information from the second frame to the intermediate frame, wherein the second motion information includes and

[0018] Optionally, generating an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual includes:

[0019] I t =M×warp(I0,F t→0 )+(1-M)×warp(I1,F t→1 )+ΔI t ;

[0020] Where M represents the mask used to process the occluder, I0 represents the first frame, I1 represents the second frame, and F t→0 represents the optical flow from the intermediate frame to the first frame, F t→1 represents the optical flow from the intermediate frame to the second frame, and the intermediate frame optical flow includes F t→0 and F t→1 , ΔI t represents the intermediate frame residual, warp(·) represents the operation of warping by optical flow, I t represents the intermediate frame.

[0021] Optionally, the plane features are obtained by extracting plane features from the first frame and the second frame using a pre-trained first feature extractor;

[0022] The hierarchical features are obtained by performing hierarchical feature extraction on the first frame and the second frame using a pre-trained second feature extractor;

[0023] The first motion information is obtained by extracting inter-frame motion information from the first frame and the second frame using a pre-trained motion information extractor;

[0024] The motion bias and motion coefficient are obtained by predicting the motion bias and motion coefficient using a pre-trained trajectory estimator;

[0025] The intermediate frame optical flow is obtained by performing optical flow estimation using a pre-trained optical flow estimator;

[0026] The intermediate frame residual is the residual of the intermediate frame predicted by the pre-trained correction network;

[0027] Among them, the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network are all neural networks and are obtained through joint training.

[0028] Optionally, the process of jointly training the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator, and the correction network includes:

[0029] Constructing a video frame sample set; each training sample in the video frame sample set includes a first sample frame, a second sample frame, and a sample intermediate frame;

[0030] Extracting planar features of the first sample frame and the second sample frame using the first feature extractor in training, extracting hierarchical features of the first sample frame and the second sample frame using the second feature extractor in training, and extracting first motion information using the motion information extractor in training;

[0031] Input the plane features of the first sample frame and the second sample frame into the trajectory estimator being trained so that it predicts the motion bias and motion coefficient;

[0032] Determining second motion information based on the first motion information extracted by the motion information extractor under training and the motion bias and motion coefficient predicted by the trajectory estimator under training;

[0033] Inputting the determined second motion information into an optical flow estimator in training, so that the optical flow estimator estimates and outputs an intermediate frame optical flow between the first sample frame and the second sample frame;

[0034] Inputting the hierarchical features of the first sample frame and the second sample frame, and the intermediate frame optical flow between the first sample frame and the second sample frame into the correction network being trained, so that the correction network predicts and outputs the intermediate frame residual between the first sample frame and the second sample frame;

[0035] Generate an intermediate frame between the first sample frame and the second sample frame according to the optical flow of the first sample frame, the second sample frame and the intermediate frame residual;

[0036] Calculating a loss value based on the difference between the sample intermediate frame and the intermediate frame generated during training;

[0037] The parameters of the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network in training are adjusted according to the loss value. After adjusting the parameters, the training is continued using the next training sample until the training is completed.

[0038] Optionally, the loss value is calculated using the following loss function:

[0039] l=loss(I gt ,I t )+0.5×loss(I gt ,I warp );

[0040] Among them, I t represents the intermediate frames generated during the training process, I gt Indicates the intermediate frames contained in the training sample, I warp represents the intermediate frame obtained by optical flow distortion, loss(·) is the Laplace loss function, and l represents the loss value.

[0041] The present invention also provides a video frame insertion device for guiding the optical flow of intermediate frames based on plane features, comprising:

[0042] An extraction module is configured to extract planar features, hierarchical features, and first motion information from a first frame and a second frame in a video where intermediate frames are to be inserted; the first motion information is inter-frame motion information between the first frame and the second frame;

[0043] A prediction module, configured to predict a motion offset and a motion coefficient based on the planar features; the motion offset and the motion coefficient are used to model inter-frame motion;

[0044] a first determining module, configured to determine second motion information based on the first motion information, the motion offset, and the motion coefficient; the second motion information including inter-frame motion information between the first frame and the intermediate frame, and inter-frame motion information between the second frame and the intermediate frame;

[0045] A first generating module, configured to generate an intermediate frame optical flow according to the second motion information;

[0046] A second determining module is used to determine an intermediate frame residual according to the hierarchical features and the intermediate frame optical flow;

[0047] The second generation module is configured to generate an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual.

[0048] Optionally, the prediction module is specifically configured to: predict motion bias and motion coefficients using a pre-trained trajectory estimator based on the plane features; wherein the trajectory estimator is constructed based on a convolutional layer.

[0049] Optionally, the first determining module is specifically configured to perform the following operations:

[0050]

[0051] Wherein, W represents the motion coefficient, B represents the motion offset, and t represents the interpolation position of the intermediate frame. represents inter-frame motion information from the first frame to the second frame, Indicates inter-frame motion information from the second frame to the first frame, the first motion information includes and represents inter-frame motion information from the first frame to the intermediate frame, represents the inter-frame motion information from the second frame to the intermediate frame, wherein the second motion information includes and

[0052] The video interpolation method based on plane feature-guided intermediate frame optical flow provided by the present invention utilizes plane features as a reliable source of motion prior information, predicts motion bias and motion coefficient through plane features, realizes accurate modeling of inter-frame motion through plane features, and then determines inter-frame motion information (second motion information) and optical flow (intermediate frame optical flow) based on the motion bias and motion coefficient, realizes more accurate intermediate frame optical flow estimation, thereby generating more accurate intermediate frames and improving the video resolution after interpolation.

[0053] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a video frame insertion method based on plane feature-guided intermediate frame optical flow provided by an embodiment of the present invention;

[0055] Figure 2 The visual evaluation effects of the present invention and existing models in general interpolation scenarios are shown;

[0056] Figure 3 The visual evaluation effects of the present invention and existing models in extreme interpolation scenarios are shown;

[0057] Figure 4 The visual evaluation effects of the present invention and existing models on a test set specifically used for evaluating large motion interpolation frames are shown. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0059] In existing video interpolation frameworks, mainstream methods use predefined assumptions to model inter-frame motion. This predefined motion pattern assumption leads to inconsistent deformation between two consecutive frames. As a result, the deformation features of the new frame are often unable to align, resulting in distortion and blurring, especially when processing large-scale and complex motion common in high-resolution videos (such as 4K). If these assumptions are not used, using networks or special techniques such as model distillation to achieve implicit inter-frame motion modeling requires additional training costs or deeper networks to fit this distribution. This will increase the computational complexity and inference latency of the model, seriously hindering the practical application of the model.

[0060] To address the above issues, in order to improve the temporal resolution of videos, especially in large-scale motion or complex scenes, an embodiment of the present invention provides a video interpolation method based on plane feature-guided intermediate frame optical flow, such as Figure 1 As shown, the method includes the following steps:

[0061] S10 , extracting planar features, hierarchical features, and first motion information from the first frame and the second frame in the video to be inserted as intermediate frames; the first motion information is inter-frame motion information between the first frame and the second frame.

[0062] It can be understood that the first frame is the front frame, the second frame is the back frame, and the middle frame is a video frame to be inserted between the front frame and the back frame.

[0063] A planar feature refers to a two-dimensional projection of the image content in a frame, describing visual characteristics such as the shape, texture, and color of objects in the image. In embodiments of the present invention, a pre-trained first feature extractor can be used to extract planar features from the first and second frames. Specifically, the first and second frames are input to the pre-trained first feature extractor, which extracts planar features from the first and second frames, thereby outputting the planar features of the first and second frames.

[0064] Hierarchical features refer to the representation of image content at different scales or levels. They describe visual information at different levels of the image, from global to local. In embodiments of the present invention, a pre-trained second feature extractor can be used to perform hierarchical feature extraction on the first and second frames. Specifically, the first and second frames of the video are input to the pre-trained second feature extractor, which performs hierarchical feature extraction on the first and second frames, thereby outputting hierarchical features of the first and second frames.

[0065] The first motion information is the inter-frame motion information between the first frame and the second frame. It can be understood that the motion information between two frames refers to the displacement information of pixels or feature points in the image between adjacent frames, which describes the motion pattern of the image content over time. In the embodiment of the present invention, the first motion information includes and Indicates the inter-frame motion information from the first frame to the second frame, represents the inter-frame motion information from the second frame to the first frame, where i and j are the row and column indices of the pixels in the image (e.g., feature map), respectively. In an embodiment of the present invention, a pre-trained motion information extractor can be used to extract inter-frame motion information from the first and second frames. Specifically, the first and second frames of the video to be inserted as intermediate frames are input into the pre-trained motion information extractor, which extracts inter-frame motion information from the first and second frames, thereby outputting first motion information.

[0066] The first feature extractor, the second feature extractor and the motion information extractor are all neural networks, and their specific network structures can be composed of various types.

[0067] For example, the neural network mentioned in section 3.1 of the paper “G. Zhang, Y. Zhu, H. Wang, Y. Chen, G. Wu, and L. Wang, “Extracting motion and appearance via inter-frame attention for efficient video frame interpolation,” in 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5682–5692.” can be used to simultaneously extract the first motion information and the plane features. In this paper, the Transformer is used to extract the plane features, and the motion information is calculated by repeatedly using the attention map in the Transformer module. After obtaining the inter-frame attention map, the plane features are extracted. and The approximate position of each pixel in the next frame is then estimated by comparing its similarity with the pixel in the next frame. The inter-frame motion information is then extracted by calculating the displacement between the pixel in the original frame and the estimated position. The specific calculation process is as follows:

[0068]

[0069] Among them, B represents the coordinate graph, and its element B i,j The value of is (i,j), For B i,j neighborhood.

[0070] In addition, the low-level feature extractor mentioned in Section 3.1 of the above document can also be used to extract hierarchical features. It first expands the number of channels of the image through a convolutional layer, and then extracts features of different scales through multiple downsampling layers and convolutional layers to obtain hierarchical features.

[0071] S20. Predicting a motion offset and a motion coefficient based on the plane features; the motion offset and the motion coefficient are used to model inter-frame motion.

[0072] In an embodiment of the present invention, the motion offset and motion coefficient can be predicted using a pre-trained trajectory estimator. Specifically, the plane features are input to the pre-trained trajectory estimator, which predicts and outputs the motion offset and motion coefficient based on the plane features. This process is expressed as:

[0073]

[0074] in, represents the plane features of the second frame, represents the planar features of the first frame, W represents the motion coefficient, B represents the motion offset, and t represents the position of the intermediate frame to be inserted. Its value range is (0, 1). A value of 0.5 indicates that the time interval between the intermediate frame, the first frame, and the second frame is equal. Of course, the insertion position is not limited to 0.5; in practice, frames can be inserted at any time between the first and second frames.

[0075] In the embodiment of the present invention, the trajectory estimator can be constructed based on a convolutional layer. For example, two convolutional layers can be used as the network architecture of the trajectory estimator, but the present invention is not limited to this.

[0076] In step S20, the motion bias and motion coefficient are accurately estimated through a pre-trained trajectory estimator, thereby achieving precise modeling of inter-frame motion through planar features. Moreover, the trajectory estimator only uses two convolutional layers to generate the bias and coefficient, without increasing the computational complexity and latency during model inference, which is more conducive to the deployment of the model on mobile devices and real-time application.

[0077] S30 . Determine second motion information according to the first motion information, the motion offset, and the motion coefficient; the second motion information includes inter-frame motion information between the first frame and the intermediate frame, and inter-frame motion information between the second frame and the intermediate frame.

[0078] Here, the second motion information includes and Indicates the inter-frame motion information from the first frame to the intermediate frame, Indicates the inter-frame motion information from the second frame to the middle frame.

[0079] Specifically, in this step, determining the second motion information according to the first motion information, the motion offset, and the motion coefficient includes:

[0080]

[0081] S40: Generate an intermediate frame optical flow according to the second motion information.

[0082] Specifically, based on the second motion information, motion-aware methods are used to generate intermediate frame optical flows. Motion-aware methods are algorithms and techniques that consider motion information in video processing tasks and are commonly used in video frame insertion.

[0083] Optionally, in one implementation, the intermediate frame optical flow can be estimated using a pre-trained optical flow estimator. Specifically, the second motion information is input into the pre-trained optical flow estimator, which predicts and outputs the intermediate frame optical flow based on the second motion information. Here, the optical flow estimator can be constructed based on convolutional layers. For example, a three-layer convolutional layer can be used as the network architecture of the optical flow estimator.

[0084] S50: Determine the intermediate frame residual according to the hierarchical features and the intermediate frame optical flow.

[0085] Here, the intermediate frame residual is the residual of the intermediate frame estimated based on the hierarchical features and the intermediate frame optical flow. This intermediate frame residual can be predicted using a pre-trained correction network. This correction network is also a neural network. Preferably, RefineNet shown in Figure 9 of the aforementioned document can be used as the correction network in the embodiment of the present invention.

[0086] S60: Generate an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual.

[0087] Specifically, generating an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual includes:

[0088] I t =M×warp(I0,F t→0 )+(1-M)×warp(I1,F t→1 )+ΔI t ;

[0089] Among them, M represents the mask used to process the occlusion, which can be generated together with the intermediate frame optical flow. That is, after the second motion information is input into the pre-trained optical flow estimator, the optical flow estimator can output the mask M at the same time as outputting the intermediate frame optical flow. I0 represents the first frame, I1 represents the second frame, and F t→0 represents the optical flow from the intermediate frame to the first frame, F t→1 Indicates the optical flow from the intermediate frame to the second frame. The intermediate frame optical flow includes F t→0 and F t→1 , ΔI t represents the intermediate frame residual, warp(·) represents the operation of warping by optical flow, I t Indicates an intermediate frame.

[0090] In an embodiment of the present invention, the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network are all neural networks. These neural networks as a whole can constitute a video enhancement model, and these neural networks can be jointly trained.

[0091] Specifically, the process of jointly training the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator, and the correction network includes:

[0092] (1) Construct a video frame sample set; each training sample in the video frame sample set includes a first sample frame, a second sample frame, and a sample middle frame.

[0093] It can be understood that the front frame in the first sample frame video, the back frame in the second sample frame video, and the sample middle frame are actual middle frames between the first sample frame and the second sample frame.

[0094] For example, the Vimeo 90K dataset can be used as a sample set of video frames. This dataset contains approximately 90,000 pairs of video frames, divided into preceding and following frames and intermediate frames. Each pair of frames comes from a different dynamic scene and has a resolution of 720p, making it suitable for video interpolation tasks. This dataset provides rich motion patterns and image features, helping the model effectively reduce motion distortion and blur when generating intermediate frames. Furthermore, each video frame can be cropped into a 256×256 image patch and data augmented by random flipping, time reversal, and rotation.

[0095] (2) Using the first feature extractor in training to extract the planar features of the first sample frame and the second sample frame, using the second feature extractor in training to extract the hierarchical features of the first sample frame and the second sample frame, and using the motion information extractor in training to extract the first motion information.

[0096] The specific implementation of this step is basically similar to the above step S10 and will not be repeated here.

[0097] (3) The planar features of the first sample frame and the second sample frame are input into the trajectory estimator being trained so that it predicts the motion bias and motion coefficient.

[0098] The specific implementation method of this step is basically similar to the above step S20 and will not be repeated here.

[0099] (4) The second motion information is determined based on the first motion information extracted by the motion information extractor in training and the motion bias and motion coefficient predicted by the trajectory estimator in training.

[0100] The specific implementation of this step is basically similar to the above step S30 and will not be repeated here.

[0101] (5) The determined second motion information is input into the optical flow estimator in training, so that it estimates and outputs the intermediate frame optical flow between the first sample frame and the second sample frame.

[0102] The specific implementation of this step is basically similar to the above step S40 and will not be repeated here.

[0103] (6) The hierarchical features of the first sample frame and the second sample frame, as well as the intermediate frame optical flow between the first sample frame and the second sample frame, are input into the correction network being trained so that it predicts and outputs the intermediate frame residual between the first sample frame and the second sample frame.

[0104] The specific implementation of this step is basically similar to the above step S50 and will not be repeated here.

[0105] (7) Generate an intermediate frame between the first sample frame and the second sample frame according to the optical flow of the first sample frame and the second sample frame and the intermediate frame residual.

[0106] The specific implementation of this step is basically similar to the above step S60 and will not be repeated here.

[0107] (8) Calculate the loss value based on the difference between the sample intermediate frame and the intermediate frame generated during training. Here, the loss value is calculated using the following loss function:

[0108] l=loss(I gt ,I t )+0.5×loss(I gt ,I warp );

[0109] Among them, I t represents the intermediate frames generated during the training process, I gt Indicates the sample middle frame, I warp represents the intermediate frame obtained by optical flow distortion, loss(·) is the Laplace loss function, and l represents the loss value.

[0110] The expression of the above Laplace loss function is:

[0111]

[0112] Among them, L i (·) represents the average pooling operation, i represents the number of average pooling operations, and i max is the upper limit corresponding to i, for example, i max =5;loss(I,I gt ) represents I and I gt Laplace loss, I specifically calculate loss(I gt ,I t ) and loss(I gt ,I warp ) are substituted into I t and I warp From the expression of the Laplace loss function, we can see that L i The smaller the resolution of (I), the more weight is given. i-1 The larger the value, the more consistent the motion will be, making the generated video smooth and natural.

[0113] (9) Adjust the parameters of the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator, and the correction network during training according to the loss value. After adjusting the parameters, continue training with the next training sample until the training is completed.

[0114] Specifically, in the optimization process of adjusting parameters, AdamW optimizer is used, parameters are set to β1 = 0.9, β2 = 0.999, and weight decay is used to 1e -4During the training process, we first warmed up the learning rate for 2000 steps and increased the learning rate from 0 to 1e -4 Then, the cosine annealing learning rate scheduling method is used to increase the learning rate from 1e to -4 Down to 1e -5 During training, observe the trend of the loss value. When the loss value tends to be stable or the number of iterations reaches the pre-set iteration limit, stop training, save the network structure and corresponding model parameters, and obtain the trained model.

[0115] As described above, the video interpolation method provided by the embodiment of the present invention uses plane features as a reliable source of motion prior information, predicts motion bias and motion coefficients through plane features, and realizes accurate modeling of inter-frame motion through plane features, and then determines inter-frame motion information (second motion information) and optical flow (intermediate frame optical flow) based on the motion bias and motion coefficient, thereby realizing more accurate intermediate frame optical flow estimation, thereby generating more accurate intermediate frames and improving the video resolution after interpolation. Compared with the existing video interpolation method, the video interpolation method proposed in the embodiment of the present invention has stronger generalization performance and can solve more complex interpolation scenarios, such as high-resolution video or large motion video.

[0116] The beneficial effects of the embodiments of the present invention can be further illustrated by the following simulation experiments:

[0117] 1. Simulation conditions

[0118] This dataset, developed on an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, an NVIDIA RTX2080Ti, and an Ubuntu 16.04 operating system, uses Python as the programming language, Pytorch as the deep learning framework, and Vimeo90K as the training dataset. This dataset provides diverse challenges for evaluating different video interpolation methods, ranging from simple to extremely complex motion scenes. The test datasets used cover resolutions from 2K to 4K. These datasets include: UCF101, Xiph, SNU_FILM, HD, X4K1000FPS, SNU_FILM_L, and Xiph_L. UCF101, Xiph, SNU_FILM, HD, and X4K1000FPS are commonly used datasets for evaluating video interpolation in general scenes or high-resolution video interpolation. SNU_FILM_L and Xiph_L are specifically designed for evaluating large-motion video interpolation. Dataset references include:

[0119] “K.Soomro,A.R.Zamir,and M.Shah,“UCF101:ADataset of 101Human ActionsClasses From Videos in The Wild,”Dec.2012,arXiv:1212.0402[cs].[Online]”;

[0120] “S.Niklaus and F.Liu,“Softmax splatting for video frameinterpolation,”in 2020IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR),2020,pp.5436-5445”;

[0121] “M.Choi,H.Kim,B.Han,N.Xu,and K.M.Lee,“Channel Attention Is All YouNeed for Video Frame Interpolation,”Proceedings of the AAAI Conference onArtificial Intelligence,vol.34,no.07,pp.10 663-10 671,Apr.2020,number:07.[Online];

[0122] “W.Bao,W.-S.Lai,X.Zhang,Z.Gao,and M.-H.Yang,“Memc-net:Motionestimation and motion compensation driven neural network for videointerpolation and enhancement,”IEEE Transactions on Pattern Analysis andMachine Intelligence,vol.43,no.3,pp.933-948,2021.”;

[0123] "H.Sim, J.Oh, and M.Kim, "Xvfi: extreme video frame interpolation," in 2021IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp.14469-14 478.";

[0124] "C. Liu, G. Zhang, R. Zhao, and L. Wang, "Sparse global matching for videoframe interpolation with large motion," in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp.19125-19 134".

[0125] The existing comparison methods used include: RIFE (ECCV 2022), IFRNet (CVPR 2022), EBME-H (WACV 2023), CURE (ECCV 2022), VFIFormer (CVPR 2022), DQBC (IJCAI 2023), UPR-Net (CVPR 2023), EMA-VFI (CVPR 2023), MA-CSPA (CVPR 2023), DBVI (ECCV 2022), M2M (CVPR2022), STMFnet (CVPR 2022), AMT (CVPR 2023), VFIMamba (Nips 2024), SGVFI (CVPR 2024), all of which are the most advanced methods in the past two years.

[0126] 2. Simulation content

[0127] Table 1 Comparison of fixed-position interpolation performance of various models in general scenes

[0128]

[0129] In Table 1, all tests were performed on a V100. The experimental results are from their original paper or the paperwithcode website. OOM indicates that the video memory exceeds the limit.

[0130] Table 2 Comparison of multi-frame interpolation performance of each model

[0131]

[0132] Table 3 Comparison of frame insertion performance of each model in large motion scenes

[0133]

[0134] In Table 3, the V in the Training Dataset column represents Vimeo90K, and V+X represents Vimeo90k and LAVIB.

[0135] Table 4 Performance comparison between the present invention and the model requiring four frames for interpolation

[0136]

[0137] In Tables 1 to 4 above, interpolation performance is measured in terms of PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). As can be seen in Table 1, the present invention significantly outperforms existing methods in high-resolution video interpolation and more difficult video interpolation scenarios. Furthermore, as the difficulty of interpolation increases, the gap between the present invention and existing algorithms gradually widens, a trend shown in Table 3. In particular, in high-resolution, high-motion scenarios, such as the Xiph_L test set, the present invention achieves a PSNR performance that is over 2dB higher than other existing methods.

[0138] Figure 2 The visual evaluation effects of the present invention and existing models in general interpolation scenarios are shown; Figure 3 The visual evaluation effects of the present invention and existing models in extreme interpolation scenarios are shown; Figure 4 The figure shows the visual evaluation performance of the proposed method compared to existing models on a test set specifically designed for evaluating large-motion interpolation. The visual results in the figure demonstrate that the proposed method can maintain the approximate shape of objects in complex scenes, while existing methods directly lead to severe image distortion and severe destruction of the image's semantic structure.

[0139] In summary, the present invention can improve the temporal resolution of videos, especially the temporal resolution in large-scale motion or complex scenes, and can be widely used in entertainment, digital media, and social public security, such as animation generation, intelligent monitoring, and other scenes with video interpolation requirements.

[0140] The method provided in the embodiments of the present invention can be applied to electronic devices. Specifically, the electronic devices can be desktop computers, portable computers, smart mobile terminals, servers, etc. This is not limited here; any electronic device that can implement the present invention falls within the scope of protection of the present invention.

[0141] Based on the same inventive concept, an embodiment of the present invention further provides a video frame insertion device for guiding the optical flow of intermediate frames based on plane features, comprising:

[0142] An extraction module, configured to extract plane features, hierarchical features, and first motion information from a first frame and a second frame in a video to be inserted as intermediate frames;

[0143] A prediction module is used to predict motion offsets and motion coefficients based on plane features; the motion offsets and motion coefficients are used to model inter-frame motion;

[0144] A first determining module, configured to determine second motion information based on the first motion information, the motion offset, and the motion coefficient;

[0145] A first generating module, configured to generate an intermediate frame optical flow according to the second motion information;

[0146] A second determination module is used to determine the intermediate frame residual according to the hierarchical features and the intermediate frame optical flow;

[0147] The second generation module is used to generate an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow and the intermediate frame residual.

[0148] Optionally, the prediction module is specifically configured to: predict the motion bias and motion coefficient using a pre-trained trajectory estimator according to the plane features; wherein the trajectory estimator is constructed based on a convolutional layer.

[0149] Optionally, the first determining module is specifically configured to perform the following operations:

[0150]

[0151] Wherein, W represents the motion coefficient, B represents the motion offset, and t represents the interpolation position of the intermediate frame. represents inter-frame motion information from the first frame to the second frame, Indicates inter-frame motion information from the second frame to the first frame, the first motion information includes and represents inter-frame motion information from the first frame to the intermediate frame, represents the inter-frame motion information from the second frame to the intermediate frame, wherein the second motion information includes and

[0152] Optionally, the second generating module is specifically configured to perform the following operations:

[0153] I t =M×warp(I0,F t→0 )+(1-M)×warp(I1,F t→1 )+ΔI t ;

[0154] Where M represents the mask used to process the occluder, I0 represents the first frame, I1 represents the second frame, and F t→0 represents the optical flow from the intermediate frame to the first frame, F t→1 represents the optical flow from the intermediate frame to the second frame, and the intermediate frame optical flow includes F t→0 and F t→1 , ΔI t represents the intermediate frame residual, warp(·) represents the operation of warping by optical flow, I t represents the intermediate frame.

[0155] Optionally, the plane features are obtained by extracting plane features from the first frame and the second frame using a pre-trained first feature extractor;

[0156] The hierarchical features are obtained by performing hierarchical feature extraction on the first frame and the second frame using a pre-trained second feature extractor;

[0157] The first motion information is obtained by extracting motion information from the first frame and the second frame using a pre-trained motion information extractor;

[0158] The motion bias and motion coefficient are obtained by predicting the motion bias and motion coefficient using a pre-trained trajectory estimator;

[0159] The intermediate frame optical flow is obtained by performing optical flow estimation using a pre-trained optical flow estimator;

[0160] The intermediate frame residual is the residual of the intermediate frame predicted by the pre-trained correction network;

[0161] Among them, the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network are all neural networks and are obtained through joint training.

[0162] Optionally, the process of jointly training the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the trajectory estimator, the optical flow estimator, and the correction network includes:

[0163] Constructing a video frame sample set; each training sample in the video frame sample set includes a first sample frame, a second sample frame, and a sample intermediate frame;

[0164] Extracting planar features of the first sample frame and the second sample frame using the first feature extractor in training, extracting hierarchical features of the first sample frame and the second sample frame using the second feature extractor in training, and extracting first motion information using the motion information extractor in training;

[0165] Input the plane features of the first sample frame and the second sample frame into the trajectory estimator being trained so that it predicts the motion bias and motion coefficient;

[0166] Determining second motion information based on first motion information between the first sample frame and the second sample frame, and a motion bias and a motion coefficient predicted by a trajectory estimator in training;

[0167] Inputting the determined second motion information into an optical flow estimator in training, so that the optical flow estimator estimates and outputs an intermediate frame optical flow between the first sample frame and the second sample frame;

[0168] Inputting the hierarchical features of the first sample frame and the second sample frame, and the intermediate frame optical flow between the first sample frame and the second sample frame into the correction network being trained, so that the correction network predicts and outputs the intermediate frame residual between the first sample frame and the second sample frame;

[0169] Generate an intermediate frame between the first sample frame and the second sample frame according to the optical flow of the first sample frame, the second sample frame and the intermediate frame residual;

[0170] Calculating a loss value based on the difference between the sample intermediate frame and the intermediate frame generated during training;

[0171] According to the loss value, the parameters of the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the trajectory estimator, the optical flow estimator and the correction network in training are adjusted. After adjusting the parameters, the training is continued using new training samples until the training is completed.

[0172] Optionally, the loss value is calculated using the following loss function:

[0173] l=loss(I gt ,I t )+0.5×loss(I gt ,I warp );

[0174] Among them, I t represents the intermediate frames generated during the training process, I gt Indicates the intermediate frames contained in the training sample, I warp represents the intermediate frame obtained by optical flow distortion, loss(·) is the Laplace loss function, and l represents the loss value.

[0175] It should be noted that, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0177] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0178] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "a" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0179] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A video interpolation method based on plane feature-guided intermediate frame optical flow, characterized in that: include: Extracting plane features, hierarchical features and first motion information from a first frame and a second frame in a video to be inserted into an intermediate frame; The first motion information is inter-frame motion information between the first frame and the second frame; Predicting a motion offset and a motion coefficient based on the planar features; the motion offset and the motion coefficient are used to model inter-frame motion; Determining second motion information according to the first motion information, the motion offset, and the motion coefficient; the second motion information includes inter-frame motion information between the first frame and the intermediate frame, and inter-frame motion information between the second frame and the intermediate frame; generating an intermediate frame optical flow according to the second motion information; determining an intermediate frame residual according to the hierarchical features and the intermediate frame optical flow; An intermediate frame is generated according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual.

2. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 1, characterized in that: The predicting of the motion offset and the motion coefficient according to the plane feature includes: According to the planar features, a pre-trained trajectory estimator is used to predict motion bias and motion coefficients; wherein the trajectory estimator is constructed based on a convolutional layer.

3. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 1, characterized in that: The determining, according to the first motion information, the motion offset, and the motion coefficient, second motion information includes: Wherein, W represents the motion coefficient, B represents the motion offset, and t represents the interpolation position of the intermediate frame. represents inter-frame motion information from the first frame to the second frame, Indicates inter-frame motion information from the second frame to the first frame, the first motion information includes and represents inter-frame motion information from the first frame to the intermediate frame, represents the inter-frame motion information from the second frame to the intermediate frame, wherein the second motion information includes and 4. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 1, characterized in that: Generating an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual includes: I t =M×warp(I0,F t→0 )+(1-M)×warp(I1,F t→1 )+ΔI t ; Where M represents the mask used to process the occluder, I0 represents the first frame, I1 represents the second frame, and F t→0 represents the optical flow from the intermediate frame to the first frame, F t→1 represents the optical flow from the intermediate frame to the second frame, and the intermediate frame optical flow includes F t→0 and F t→1 , ΔI t represents the intermediate frame residual, warp(·) represents the operation of warping by optical flow, I t represents the intermediate frame.

5. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 1, characterized in that: The plane features are obtained by extracting plane features from the first frame and the second frame using a pre-trained first feature extractor; The hierarchical features are obtained by performing hierarchical feature extraction on the first frame and the second frame using a pre-trained second feature extractor; The first motion information is obtained by extracting inter-frame motion information from the first frame and the second frame using a pre-trained motion information extractor; The motion bias and motion coefficient are obtained by predicting the motion bias and motion coefficient using a pre-trained trajectory estimator; The intermediate frame optical flow is obtained by performing optical flow estimation using a pre-trained optical flow estimator; The intermediate frame residual is the residual of the intermediate frame predicted by the pre-trained correction network; Among them, the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network are all neural networks and are obtained through joint training.

6. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 5, characterized in that: The process of jointly training the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator, and the correction network includes: Constructing a video frame sample set; each training sample in the video frame sample set includes a first sample frame, a second sample frame, and a sample intermediate frame; Extracting planar features of the first sample frame and the second sample frame using the first feature extractor in training, extracting hierarchical features of the first sample frame and the second sample frame using the second feature extractor in training, and extracting first motion information using the motion information extractor in training; Input the plane features of the first sample frame and the second sample frame into the trajectory estimator being trained so that it predicts the motion bias and motion coefficient; Determining second motion information based on the first motion information extracted by the motion information extractor under training and the motion bias and motion coefficient predicted by the trajectory estimator under training; Inputting the determined second motion information into an optical flow estimator in training, so that the optical flow estimator estimates and outputs an intermediate frame optical flow between the first sample frame and the second sample frame; Inputting the hierarchical features of the first sample frame and the second sample frame, and the intermediate frame optical flow between the first sample frame and the second sample frame into the correction network being trained, so that the correction network predicts and outputs the intermediate frame residual between the first sample frame and the second sample frame; Generate an intermediate frame between the first sample frame and the second sample frame according to the optical flow of the first sample frame, the second sample frame and the intermediate frame residual; Calculating a loss value based on the difference between the sample intermediate frame and the intermediate frame generated during training; The parameters of the first feature extractor, the second feature extractor, the motion information extractor, the trajectory estimator, the optical flow estimator and the correction network in training are adjusted according to the loss value. After adjusting the parameters, the training is continued using the next training sample until the training is completed.

7. The video frame insertion method based on plane feature-guided intermediate frame optical flow according to claim 6, characterized in that: The loss value is calculated by the following loss function: l=loss(I gt ,I t )+0.5×loss(I gt ,I warp ); Among them, I t represents the intermediate frames generated during the training process, I gt Indicates the intermediate frames contained in the training sample, I warp represents the intermediate frame obtained by optical flow distortion, loss(·) is the Laplace loss function, and l represents the loss value.

8. A video frame insertion device based on plane feature-guided intermediate frame optical flow, characterized in that: include: An extraction module, configured to extract plane features, hierarchical features, and first motion information from a first frame and a second frame in a video to be inserted as intermediate frames; The first motion information is inter-frame motion information between the first frame and the second frame; A prediction module, configured to predict a motion offset and a motion coefficient based on the planar features; the motion offset and the motion coefficient are used to model inter-frame motion; a first determining module, configured to determine second motion information based on the first motion information, the motion offset, and the motion coefficient; the second motion information including inter-frame motion information between the first frame and the intermediate frame, and inter-frame motion information between the second frame and the intermediate frame; A first generating module, configured to generate an intermediate frame optical flow according to the second motion information; A second determining module is used to determine an intermediate frame residual according to the hierarchical features and the intermediate frame optical flow; The second generation module is configured to generate an intermediate frame according to the first frame, the second frame, the intermediate frame optical flow, and the intermediate frame residual.

9. The device according to claim 8, characterized in that The prediction module is specifically used to: predict the motion bias and motion coefficient using a pre-trained trajectory estimator according to the plane features; wherein the trajectory estimator is constructed based on a convolutional layer.

10. The device according to claim 8, characterized in that The first determining module is specifically configured to perform the following operations: Wherein, W represents the motion coefficient, B represents the motion offset, and t represents the interpolation position of the intermediate frame. represents inter-frame motion information from the first frame to the second frame, Indicates inter-frame motion information from the second frame to the first frame, the first motion information includes and represents inter-frame motion information from the first frame to the intermediate frame, represents the inter-frame motion information from the second frame to the intermediate frame, wherein the second motion information includes and

Images