Video frame interpolation method, device, equipment and storage medium

Abstract

This disclosure provides a video frame interpolation method, apparatus, electronic device, and computer-readable storage medium, relating to the field of computer technology. The method includes: performing a forward warp process on a group of reference frames and their corresponding motion vector information to obtain prior information of an intermediate frame to be interpolated; wherein the intermediate frame to be interpolated is located between two reference frames contained in the reference frame group; inputting the reference frame group and the prior information into a neural network model, so that the neural network model predicts the predicted optical flow of the intermediate frame to be interpolated based on the reference frame group and the prior information; and generating the intermediate frame to be interpolated based on the predicted optical flow of the intermediate frame to be interpolated and the reference frame group. This method can achieve a clear frame interpolation effect.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to a video frame interpolation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of digital video technology, high frame rate video is increasingly in demand across various video fields due to its ability to provide a smoother and clearer visual experience. However, limited by original shooting equipment, storage costs, or transmission bandwidth, a large amount of existing or real-time acquired video footage has a low frame rate, resulting in noticeable motion stuttering and jitter when played or used directly. Therefore, video frame rate upconversion technology, especially video frame interpolation technology, has become a key means to improve video temporal resolution and enhance visual quality. Summary of the Invention

[0003] This disclosure provides a video frame interpolation method, apparatus, electronic device, and computer-readable storage medium.

[0004] In a first aspect, embodiments of this disclosure propose a video frame interpolation method, which includes: performing forward warp processing on a reference frame group and its corresponding motion vector information to obtain prior information of an intermediate frame to be interpolated; wherein the intermediate frame to be interpolated is located between two reference frames contained in the reference frame group; inputting the reference frame group and the prior information into a neural network model, so that the neural network model predicts the predicted optical flow of the intermediate frame to be interpolated based on the reference frame group and the prior information, and generates the intermediate frame to be interpolated based on the predicted optical flow of the intermediate frame to be interpolated and the reference frame group.

[0005] In some embodiments, the two reference frames include a first reference frame and a second reference frame, with the second reference frame located after the intermediate frame to be inserted. The process of performing a forward warp processing on the reference frame group and its corresponding motion vector information to obtain prior information of the intermediate frame to be inserted includes: determining initial optical flow information from the second reference frame to the first reference frame based on the motion vector information of the second reference frame; calculating the initial optical flow information based on the proportional relationship between a first relative position parameter between the second reference frame and the intermediate frame to be inserted and a second relative position parameter between the second reference frame and the first reference frame; and performing a forward warp processing on the calculated result to obtain prior information of the intermediate frame to be inserted.

[0006] In some embodiments, the prior information includes prior optical flow. The process of inputting a reference frame group and prior information into a neural network model to predict the predicted optical flow of the intermediate frame to be inserted includes: performing multi-scale feature extraction on the reference frame group using the neural network model to obtain reference feature maps at N scales with feature fineness ranging from fine to coarse; where N is a positive integer greater than or equal to 2; and performing multi-level optical flow optimization processing based on the reference feature maps and prior optical flow using the neural network model to obtain the predicted optical flow of the intermediate frame to be inserted.

[0007] In some embodiments, the prior information further includes an occlusion mask. The process of obtaining the predicted optical flow of the intermediate frame to be inserted by performing multi-level optical flow optimization processing through a neural network model based on the reference feature map and the prior optical flow includes: obtaining an initial feature map of the intermediate frame to be inserted based on the reference feature map, the occlusion mask, and the prior optical flow through a neural network model; and performing multi-level optical flow optimization processing on the prior optical flow based on the reference feature map and the initial feature map to obtain the predicted optical flow of the intermediate frame to be inserted.

[0008] In some embodiments, the neural network model includes N feature extraction layers and N decoders corresponding to the feature extraction layers, and each feature extraction layer is used to output a reference feature map at a corresponding scale; wherein, based on the reference feature map and the initial feature map, the prior optical flow is optimized through the neural network model to obtain the predicted optical flow of the intermediate frame to be inserted, including: in the order of feature fineness from coarse features to fine features, each decoder outputs the final predicted optical flow and the intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, and outputs the final predicted optical flow and the intermediate feature map of the current level; wherein, the final predicted optical flow of the previous level of the first level decoder is the prior optical flow, the intermediate feature map of the previous level of the first level decoder is the initial feature map, and the final predicted optical flow of the current level output by the last level decoder is the predicted optical flow of the intermediate frame to be inserted.

[0009] In some embodiments, based on the previous level final predicted optical flow and the previous level intermediate feature map output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, the current level final predicted optical flow and the current level intermediate feature map are output, including: obtaining the current level initial predicted optical flow based on the previous level final predicted optical flow and the previous level intermediate feature map output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder; fusing the current level initial predicted optical flow and the prior optical flow to output the current level final predicted optical flow and the current level intermediate feature map.

[0010] In some embodiments, the neural network model includes a convolutional network; the initial predicted optical flow of the current level is obtained based on the final predicted optical flow of the previous level output by the previous level decoder, the intermediate feature map of the previous level, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, including: warping the reference feature map output by the feature extraction layer corresponding to the current level decoder based on the final predicted optical flow of the previous level output by the previous level decoder to obtain a warped reference feature map; calculating the correlation of the warped reference feature map to obtain correlated features; concatenating the correlated features, the warped reference feature map, and the intermediate feature map of the previous level to form a multi-source fusion feature, and inputting the formed multi-source fusion feature into the convolutional network to obtain optical flow deviation; updating the final predicted optical flow of the previous level based on the optical flow deviation to obtain the initial predicted optical flow of the current level.

[0011] In some embodiments, fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow and the current-level intermediate feature map includes: fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow; and performing a warp operation on the previous-level intermediate feature map based on the current-level final predicted optical flow to obtain the current-level intermediate feature map.

[0012] In some embodiments, fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow includes: determining the weighting coefficients of the current-level initial predicted optical flow and the prior optical flow; and fusing the current-level initial predicted optical flow and the prior optical flow according to the weighting coefficients to output the current-level final predicted optical flow.

[0013] In some embodiments, determining the weight coefficients of the current-level initial predicted optical flow and the prior optical flow includes: converting the current-level initial predicted optical flow and the prior optical flow into unfolded predicted optical flow and unfolded prior optical flow, respectively, in block representations containing spatial neighborhood information; performing a warp operation on the reference feature map output by the feature extraction layer corresponding to the current-level decoder based on the unfolded predicted optical flow and the unfolded prior optical flow, respectively, to obtain a warped predicted feature map and a warped prior feature map; calculating a first similarity of pixel neighborhoods within the warped predicted feature map and a second similarity of pixel neighborhoods within the warped prior feature map; concatenating the first similarity, the second similarity, the unfolded predicted optical flow, the unfolded prior optical flow, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder through channel concatenation to form a fusion tensor; and determining the weight coefficients of the prior optical flow and the current-level initial predicted optical flow based on the fusion tensor.

[0014] In some embodiments, multi-scale feature extraction of the reference frame group includes: concatenating the reference frame group with corresponding depth information to obtain a concatenated video frame; and performing multi-scale feature extraction on the concatenated video frame.

[0015] In some embodiments, the prior information includes prior optical flow and an occlusion mask. The reference frame group includes a first reference frame and a second reference frame. The prior optical flow includes a first prior optical flow from the intermediate frame to be inserted to the first reference frame and a second prior optical flow from the intermediate frame to be inserted to the second reference frame. The intermediate frame optical flow includes a first intermediate frame optical flow corresponding to the first prior optical flow and a second intermediate frame optical flow corresponding to the second prior optical flow. Generating the intermediate frame to be inserted based on the predicted optical flow of the intermediate frame and the reference frame group includes: performing a forward warp operation on the first reference frame based on the first intermediate frame optical flow to obtain a first warp feature map corresponding to the intermediate frame to be inserted; performing a forward warp operation on the second reference frame based on the second intermediate frame optical flow to obtain a second warp feature map corresponding to the intermediate frame to be inserted; and fusing the first warp feature map and the second warp feature map based on the occlusion mask to obtain the intermediate frame to be inserted.

[0016] In some embodiments, the method further includes: training a neural network model, wherein the training process of the neural network model includes: acquiring at least one set of sample reference frames; each set of sample reference frames includes: a first sample reference frame, a second sample reference frame, and a sample intermediate frame located between the first sample reference frame and the second sample reference frame; performing forward warp processing on at least one set of sample reference frames and the corresponding sample motion vector information to obtain sample prior information of the sample intermediate frame; and using at least one set of sample reference frames and the sample prior information as training data to iteratively train a preset initial neural network model to obtain a neural network model.

[0017] In some embodiments, forward warping is performed on the at least one set of sample reference frames and the corresponding sample motion vector information to obtain sample prior information of the sample intermediate frame, including: determining the initial sample optical flow information of the sample intermediate frame based on the motion vector information of the sample intermediate frame and the motion vector information of the second sample reference frame; and performing forward warping on the initial sample optical flow information to obtain the sample prior information of the sample intermediate frame.

[0018] Secondly, this disclosure also provides a video frame interpolation apparatus, which includes: a calculation module configured to perform forward warp processing on a reference frame group and its corresponding motion vector information to obtain prior information of an intermediate frame to be interpolated; wherein the intermediate frame to be interpolated is located between two reference frames contained in the reference frame group; and a prediction module configured to input the reference frame group and the prior information into a neural network model, so that the neural network model predicts the predicted optical flow of the intermediate frame to be interpolated based on the reference frame group and the prior information, and generates the intermediate frame to be interpolated based on the predicted optical flow of the intermediate frame to be interpolated and the reference frame group.

[0019] In some embodiments, the prior information includes prior optical flow. The prediction module includes a feature extraction unit configured to perform multi-scale feature extraction on a group of reference frames to obtain reference feature maps at N scales with feature fineness ranging from fine to coarse, where N is a positive integer greater than or equal to 2; and a multi-level optical flow optimization unit configured to perform multi-level optical flow optimization processing based on the reference feature maps and prior optical flow to obtain the predicted optical flow of the intermediate frame to be inserted.

[0020] In some embodiments, the prior information further includes an occlusion mask. The feature extraction unit includes N feature extraction subunits configured to output reference feature maps at corresponding scales. The multi-level optical flow optimization unit includes a construction unit and N decoders. The construction unit is configured to obtain an initial feature map of the intermediate frame to be inserted based on the reference feature map, the occlusion mask, and the prior optical flow. The N decoders are configured to correspond to the feature extraction subunits and are set to output the current-level final predicted optical flow and the current-level intermediate feature map in order from coarse to fine features, based on the previous-level final predicted optical flow and the previous-level intermediate feature map output by the previous-level decoder, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder. The previous-level final predicted optical flow of the first-level decoder is the prior optical flow, the previous-level intermediate feature map of the first-level decoder is the initial feature map, and the current-level final predicted optical flow output by the last-level decoder is the predicted optical flow of the intermediate frame to be inserted.

[0021] In some embodiments, the decoder includes: a fine-tuning unit configured to obtain the current-level initial predicted optical flow based on the previous-level final predicted optical flow and the previous-level intermediate feature map output by the previous-level decoder, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder; and an optical flow fusion unit configured to fuse the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow and the current-level intermediate feature map.

[0022] In some embodiments, the fine-tuning unit includes: a feature warping unit configured to warp the reference feature map output by the feature extraction subunit corresponding to the current level decoder based on the final predicted optical flow output by the previous level decoder, to obtain a warped reference feature map; a body unit configured to calculate the correlation of the warped reference features to obtain correlated features; a feature fusion unit configured to concatenate the correlated features, the warped reference feature map, and the intermediate feature map of the previous level to form a multi-source fusion feature; a convolution correction unit configured to perform convolution processing on the multi-source fusion feature to output optical flow deviation; and an optical flow update unit configured to update the final predicted optical flow of the previous level based on the optical flow deviation to obtain the initial predicted optical flow of the current level.

[0023] In some embodiments, the optical flow fusion unit includes: a coefficient determination unit configured to determine weight coefficients for the current-level initial predicted optical flow and the prior optical flow; and an optical flow output unit configured to fuse the current-level initial predicted optical flow and the prior optical flow according to the weight coefficients, and output the current-level final predicted optical flow.

[0024] In some embodiments, the coefficient determination unit includes: an optical flow unrolling subunit configured to transform the current-level initial prediction optical flow and the prior optical flow respectively to obtain an unrolled prediction optical flow and an unrolled prior optical flow in a block representation containing spatial neighborhood information; a feature warping subunit configured to warp the reference feature map output by the feature extraction layer corresponding to the current-level decoder based on the unrolled optical flow to obtain a warped prediction feature map and a warped prior feature map; a similarity calculation subunit configured to calculate a first similarity of pixel neighborhoods within the warped prediction feature map and a second similarity of pixel neighborhoods within the warped prior feature map; a concatenation and fusion subunit configured to concatenate the first similarity, the second similarity, the unrolled prediction optical flow, the unrolled prior optical flow, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder through channels to form a fusion tensor; and a weight prediction subunit configured to determine the weight coefficients of the prior optical flow and the current-level initial prediction optical flow based on the fusion tensor.

[0025] In some embodiments, the prior information includes prior optical flow and occlusion mask, the reference frame group includes a first reference frame and a second reference frame, the prior optical flow includes a first prior optical flow from the intermediary frame to be inserted to the first reference frame and a second prior optical flow from the intermediary frame to be inserted to the second reference frame; correspondingly, the intermediary frame optical flow includes a first intermediary frame optical flow corresponding to the first prior optical flow and a second intermediary frame optical flow corresponding to the second prior optical flow; wherein, the generation module includes: a feature warp synthesis unit, configured to perform a forward warp operation on the first reference frame based on the first intermediary frame optical flow to obtain a first warp feature map corresponding to the intermediary frame to be inserted; and to perform a forward warp processing on the second reference frame based on the second intermediary frame optical flow to obtain a second warp feature map corresponding to the intermediary frame to be inserted; and a frame generation unit, configured to fuse the first warp feature map and the second warp feature map based on the occlusion mask to obtain the intermediary frame to be inserted.

[0026] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the video frame interpolation method as described in any implementation of the first aspect.

[0027] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions configured to enable a computer, when executed, to implement the video frame interpolation method as described in any implementation of the first aspect.

[0028] The video frame interpolation method disclosed herein first obtains prior information of the intermediate frame to be interpolated through forward warping based on a reference frame and its corresponding motion vector information. Next, based on the prior information, a neural network model predicts the optical flow of the intermediate frame to be interpolated. Finally, the reference frame is synthesized into the intermediate frame to be interpolated based on the obtained predictive optical flow. This embodiment of the disclosure, by introducing prior information, transforms the ill-conditioned problem caused by uncertain motion states during frame interpolation into a deterministic optimization problem, effectively avoiding the blurry interpolation effect in traditional methods and achieving clearer interpolation output. Furthermore, the source of the prior information is the motion vector of the reference frame, which is verified inter-frame displacement data during video encoding and directly reflects the real motion trajectory, fundamentally reducing the randomness of motion trajectory prediction and improving the reliability of the prior information. Furthermore, to address the time / displacement inconsistency problem that easily occurs in non-uniform motion scenarios such as acceleration, deceleration, and turning, forward warp processing derives prior optical flow through motion vector information, enabling prior information to fuse the bidirectional motion correlation between the interpolated intermediate frame and the preceding and following reference frames. This provides a clear initial benchmark for complex motion prediction, transforming the originally uncertain motion estimation into a deterministic coordinate mapping problem, and avoiding the trajectory distortion problem caused by the complexity of motion when directly calculating optical flow. Attached Figure Description

[0029] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of a video frame interpolation method provided in this disclosure embodiment; Figure 2 A schematic diagram illustrating a video frame interpolation method provided in accordance with the embodiments of this disclosure, incorporating a neural network model structure; Figure 3 This is a schematic diagram of a method for generating initial predicted optical flow provided in this disclosure; Figure 4 This is a schematic diagram of a method for predicting the final optical flow based on the initial predicted optical flow output provided in this disclosure; Figure 5 A structural block diagram of a video frame interpolation device provided in this embodiment of the present disclosure; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0031] Video frame interpolation technology refers to the synthesis of intermediate frames that are logically consistent and visually coherent between known adjacent video frames.

[0032] In some embodiments, the video frame interpolation method can use CNN (Convolutional Neural Network) or Transformer (a neural network model / architecture) to extract known image features of adjacent video frames, then directly calculate the offset of the pixel points based on the extracted image features, and optimize the offset of the pixel points step by step to finally obtain the image optical flow of the intermediate frame, and then generate the intermediate frame through the image optical flow of the intermediate frame.

[0033] This is a scheme for directly estimating optical flow, which calculates the intermediate frame based on the motion displacement of the preceding and following frames, or obtains the motion position of the interpolated frame by using a multi-frame acceleration calculation method. However, in real-world scenarios, there are complex motions such as acceleration, deceleration, and turning. The intermediate frames calculated using CNN / Transformer models are inconsistent with the real intermediate frames in terms of time and distance. Furthermore, the model training ultimately uses the known average time and distance between adjacent frames as the intermediate frame, resulting in a blurry interpolation effect.

[0034] In other embodiments, the optical flow of the intermediate frame can be calculated based on the MV (Motion Vector) in the prior G-Buffer, and then fine-tuned step by step to calculate the deviation of the optical flow to correct the optical flow, thereby generating the intermediate frame.

[0035] This is a priori-based method that uses the prior optical flow (MV) as the initial optical flow and gradually fine-tunes it. This approach typically involves fine-tuning the prior optical flow, which can lead to deviations in the model fitting and a decrease in accuracy. Furthermore, even when the MV is accurate, the accuracy of the fine-tuned flow may still be reduced.

[0036] Based on this, the present disclosure proposes a video frame interpolation scheme.

[0037] Figure 1This is a flowchart of a video frame interpolation method provided in an embodiment of the present disclosure.

[0038] like Figure 1 As shown, the method specifically includes the following steps: Step 101: Perform forward warp processing on the reference frame group and the corresponding motion vector information to obtain the prior information of the intermediate frame to be inserted.

[0039] The intermediate frame to be inserted is located between two reference frames contained in the reference frame group.

[0040] A reference frame set is a continuous set of video frames used to provide motion reference and feature support during video frame interpolation. It includes key reference frames before and after the intermediate frame to be interpolated, and can completely characterize the motion trajectory and feature correlation between frames. The reference frame set includes at least two consecutive video frames in the video frame sequence.

[0041] Optionally, the reference frame group may include a first reference frame and a second reference frame, wherein the first reference frame and the second reference frame are two known adjacent video frames in the video frame sequence, used to generate the intermediate frame to be inserted. The intermediate frame to be inserted is located between the first reference frame and the second reference frame and is an unknown video frame. For example, if the intermediate frame to be inserted is inserted between the first frame I1 and the second frame I2, then the first frame I1 and the third frame I2 are the first reference frame and the second reference frame in the reference frame group, respectively.

[0042] A motion vector (MV) is a two-dimensional vector used to describe the motion trajectory of pixels or image blocks between adjacent frames in a video sequence. It is typically obtained through block matching algorithms or optical flow estimation and includes horizontal and vertical displacement components. Motion vector information is calculated using motion estimation algorithms to obtain pixel-level or block-level motion vector fields between a reference frame and the frame to be interpolated, characterizing the direction and magnitude of motion of objects in the scene. Specifically, the motion vector information includes the motion vectors of each reference frame in the reference frame group pointing to the previous frame. For example, if the reference frame group includes an adjacent first frame I1 and a second frame I2, the motion vector information may include mv1 and mv2, where mv1 refers to the motion vector of the first frame I1 relative to the previous frame (e.g., the 0th video frame before the first frame I1 in the video sequence), and mv2 refers to the motion vector of the second frame I2 relative to the first frame I1. Optionally, this motion vector information can be taken from the G-Buffer (G-buffer) or obtained through other means; this disclosure does not limit this.

[0043] Based on the motion vector information of video frames, the corresponding optical flow can be calculated. Optical flow is the pixel-level motion vector of the same pixel / object within a video frame across consecutive video frames. In other words, optical flow describes the direction and distance a pixel moves from a first video frame to a second video frame adjacent to that first. For example, using the above example, the optical flow from the second frame to the first frame can be calculated based on mv2.

[0044] The prior information of the intermediate frame to be inserted can be used to describe the prior optical flow of the intermediate frame. Specifically, the prior optical flow of the intermediate frame to be inserted can be obtained based on the motion vector information of the reference frame. However, the motion vector information only contains a single-direction motion trajectory and does not relate to the global motion of the intermediate frame to be inserted. Furthermore, the obtained prior optical flow depends on the accuracy of the motion vector information; if the motion vector information is distorted, it will directly affect the accuracy of the prior optical flow.

[0045] Furthermore, this disclosure can obtain prior information of the intermediate frame to be inserted by performing a forward warp operation based on the motion vector of the reference frame. Specifically, the warp operation refers to mapping pixels in the reference frame to the corresponding feature space of the intermediate frame to be inserted based on the motion vector according to their motion trajectories.

[0046] Specifically, by performing a forward warp operation on the motion vector of the reference frame, the bidirectional motion correlation between the two frames before and after the interpolated intermediate frame (i.e., the reference frame group) can be fused, reflecting the motion consistency between multiple frames; moreover, this method is naturally adapted to non-uniform motion scenes, and the motion trajectory is more in line with the real physical laws.

[0047] Step 102: Input the reference frame group and prior information into the neural network model so that the neural network model can predict the predicted optical flow of the intermediate frame to be inserted based on the reference frame group and prior information, and generate the intermediate frame to be inserted based on the predicted optical flow of the intermediate frame to be inserted and the reference frame group.

[0048] Specifically, the neural network model can be a pre-trained optical flow prediction model that can predict the optical flow of the intermediate frame to be inserted based on the known reference frame and prior information; or it can be an optical flow prediction model to be trained, which can be trained using the known reference frame and prior information so that the trained optical flow prediction model can accurately predict the optical flow of the intermediate frame to be inserted based on the known reference frame and prior information.

[0049] Since the prior information is obtained through forward warping based on the reference frame and the corresponding motion vector information in step 101 above, prediction based on accurate prior information makes the predicted optical flow of the intermediate frame to be inserted predicted by the neural network model more accurate and has higher precision.

[0050] After obtaining a high-precision final predicted optical flow, the reference frame can be further synthesized into an intermediate frame to be inserted based on the predicted optical flow.

[0051] The video frame interpolation method disclosed herein firstly obtains prior information of the intermediate frame to be interpolated through forward warping based on a reference frame and its corresponding motion vector information; then, predicts the optical flow of the intermediate frame to be interpolated through a neural network model based on the prior information; and finally, synthesizes the reference frame into the intermediate frame to be interpolated based on the obtained predictive optical flow. This embodiment of the disclosure introduces prior information to transform the ill-conditioned problem caused by the uncertainty of motion state during frame interpolation into a deterministic optimization problem, effectively avoiding the blurry frame interpolation effect in traditional methods and achieving clearer frame interpolation output. Furthermore, the source of the prior information is the motion vector of the reference frame, which is the inter-frame displacement data verified during video encoding and can directly reflect the real motion trajectory, reducing the randomness of motion trajectory prediction from the root and improving the reliability of the prior information. Furthermore, for the time / displacement inconsistency problem that is prone to occur in non-uniform motion scenarios such as acceleration, deceleration, and turning, the forward warp processing derives the prior optical flow through motion vector information, enabling the prior information to fuse the bidirectional motion correlation between the intermediate frame to be interpolated and the preceding and following reference frames, providing a clear initial benchmark for complex motion prediction, transforming the originally uncertain motion estimation into a deterministic coordinate mapping problem, and avoiding the trajectory distortion problem caused by the complexity of motion when directly calculating the optical flow.

[0052] Regarding step 101 above, in the process of determining prior information, the initial optical flow information of the intermediate frame to be inserted can be determined first; then, the initial optical flow information can be processed by forward warp to obtain the final prior information. A specific implementation method is given below.

[0053] In some embodiments, the two reference frames include a first reference frame and a second reference frame, with the second reference frame located after the intermediate frame to be inserted. The step of performing a forward warp processing on the reference frame group and its corresponding motion vector information to obtain prior information of the intermediate frame to be inserted includes: determining initial optical flow information from the second reference frame to the first reference frame based on the motion vector information of the second reference frame; calculating the initial optical flow information based on a first relative position parameter between the second reference frame and the intermediate frame to be inserted, and a proportional relationship between a second relative position parameter between the second reference frame and the first reference frame; and performing a forward warp processing on the calculated result to obtain the prior information of the intermediate frame to be inserted.

[0054] The initial optical flow can include the optical flow from the reference frame to the intermediate frame to be inserted. By performing forward warping on the initial optical flow, the optical flow from the intermediate frame to the reference frame can be obtained, which serves as prior information for the intermediate frame to be inserted.

[0055] The prior optical flow includes a second prior optical flow from the intermediate frame to be inserted to the second reference frame.

[0056] For example, assuming the reference frame group includes a first reference frame I1 and a second reference frame I2, and corresponding motion vectors mv1 and mv2, then the optical flow flow_21 (flow2->1, simplified to flow_21) from the second reference frame I2 to the first reference frame I1 can be obtained based on mv2. The intermediate frame to be inserted is theoretically located at the middle of the first and second reference frames (this middle can be the middle in the time dimension or the middle in the spatial dimension, which is not limited in this disclosure). Furthermore, based on a time / distance of 0.5, a second initial optical flow from the second reference frame I2 to the intermediate frame to be inserted can be generated, and by performing a forward warp operation on the second initial optical flow, a second prior optical flow from the intermediate frame to be inserted to the second reference frame I2 can be obtained.

[0057] In some embodiments, the prior optical flow information may further include a first prior optical flow from the intermediary frame to be inserted to the first reference frame.

[0058] Furthermore, after obtaining the optical flow flow_21 from the second reference frame I2 to the first reference frame I1, flow_12 can be obtained through forward warp, and based on a time / distance of 0.5, a first initial optical flow from the first reference frame I1 to the intermediate frame to be inserted is generated. By performing a forward warp operation on this first initial optical flow, a first prior optical flow from the intermediate frame to be inserted to the first reference frame I1 is obtained.

[0059] The embodiments disclosed herein effectively preserve the original accuracy of the motion vector by determining the initial optical flow based on the motion vector of the second reference frame and then obtaining the prior information through forward warp processing. They also transform the ill-conditioned interpolation problem caused by non-uniform motion into a deterministic coordinate mapping problem, thus solving the defects of existing technologies such as frame interpolation blurring and decreased accuracy of prior optical flow fine-tuning.

[0060] In some embodiments, prior information includes prior optical flow.

[0061] Furthermore, the step of inputting the reference frame group and prior information into the neural network model to predict the optical flow of the intermediate frame to be inserted includes: performing multi-scale feature extraction on the reference frame group through the neural network model to obtain reference feature maps of N scales with feature fineness ranging from fine to coarse; where N is a positive integer greater than or equal to 2; and performing multi-level optical flow optimization processing through the neural network model based on the reference feature maps and prior optical flow to obtain the predicted optical flow of the intermediate frame to be inserted.

[0062] Specifically, feature refinement refers to the granularity or richness of detail of the information contained in the feature map. Finer feature refinement corresponds to higher spatial resolution and low-level features such as edges, textures, colors, and details; coarser feature refinement corresponds to lower spatial resolution and high-level features such as object, scene, and semantic features. This disclosure uses a neural network model to extract features at different scales from each reference frame in the reference frame group using a feature pyramid approach, obtaining N reference feature maps with feature refinement ranging from fine features (low-level features) to coarse features (high-level features). Then, based on the extracted N reference feature maps, the prior optical flow is optimized at N levels, finally outputting the predicted optical flow of the intermediate frame to be interpolated.

[0063] This embodiment of the disclosure achieves progressively refined optical flow optimization from coarse to fine by extracting features at multiple scales and combining them with multi-level optical flow optimization. This effectively improves the estimation accuracy of the predicted optical flow in the interpolated intermediate frames, making the optical flow more closely match the actual motion trajectory between frames. Specifically, the multi-scale reference feature map provides adaptive feature support for optical flow optimization at different levels, taking into account both the global trend and local details of inter-frame motion, thus enhancing the adaptability of optical flow optimization to complex motion scenes.

[0064] Furthermore, in some embodiments, the prior information also includes an occlusion mask.

[0065] Specifically, during the forward warp process in step 101 above, inter-frame occlusion regions are also detected synchronously, and an occlusion mask is generated. This occlusion mask can mark invalid regions of optical flow. Based on the marked invalid regions of optical flow, invalid information in the corresponding optical flow can be filtered out, thereby reducing the interference of occlusion on the accuracy of optical flow and further improving the accuracy of prior optical flow.

[0066] The steps of obtaining the predicted optical flow of the intermediate frame to be inserted by performing multi-level optical flow optimization processing through a neural network model based on the reference feature map and prior optical flow include: obtaining the initial feature map of the intermediate frame to be inserted based on the reference feature map, occlusion mask and prior optical flow through a neural network model; and performing multi-level optical flow optimization processing on the prior optical flow based on the reference feature map and the initial feature map to obtain the predicted optical flow of the intermediate frame to be inserted.

[0067] Specifically, after extracting multi-scale reference feature maps, the neural network model first maps the reference feature maps of the last level (i.e., high-level features), the prior optical flow obtained from forward warping, and the synchronously generated occlusion mask to the feature space of the intermediate frame to be inserted, based on the coordinate mapping relationship of the prior optical flow. This results in two sets of mapped feature maps that match the intermediate frame to be inserted. Then, the occlusion mask is used to filter invalid regions in the two sets of mapped feature maps, masking the feature information of occluded regions and invalid boundary regions, retaining only reliable features of the effective motion regions. Finally, the feature fusion layer of the neural network model performs pixel-level and feature-level fusion processing on the filtered two sets of mapped feature maps, integrating the effective feature information of the bidirectional reference frames to generate the initial feature map of the intermediate frame to be inserted. This initial feature map not only fits the real motion trajectory described by the prior optical flow but also avoids interference from invalid features, possessing a basic feature structure that matches the motion state of the intermediate frame to be inserted.

[0068] The initial feature map of the intermediate frame to be inserted is ft=warp(f0,flow_t1). (1-mask)+warp(f1,flow_t0) mask. Here, ft represents the initial feature map of the intermediate frame to be inserted, f0 and f1 represent the reference feature maps of the first reference frame and the second reference frame, respectively, flow_t0 and flow_t1 represent the first prior optical flow and the second prior optical flow, respectively, and mask represents the occlusion mask.

[0069] Furthermore, using multi-scale reference feature maps as global feature bases and initial feature maps as local feature bases for the frames to be interpolated, a neural network model is used to perform multi-level optical flow optimization from coarse to fine, gradually correcting the small trajectory deviations of the prior optical flow, and obtaining high-precision predicted optical flow for the intermediate frames to be interpolated.

[0070] The method disclosed herein uses multi-scale reference feature maps of reference frame groups as feature bases, combines invalid region constraints of occlusion masks with motion trajectory guidance of prior optical flow, and constructs an initial feature map of the intermediate frame to be inserted through a neural network model, providing a feature base that fits the real motion for subsequent optical flow optimization; then, the initial feature map provides a feature base that fits the motion state of the frame to be inserted for optical flow optimization, and gradually improves the optical flow accuracy through multi-level optical flow optimization, which not only avoids trajectory distortion of optical flow optimization without feature support, but also solves the problem of insufficient accuracy of single prior optical flow in complex motion scenes, laying a high-quality feature and optical flow dual foundation for subsequent accurate synthesis of the intermediate frame to be inserted.

[0071] This neural network model can be a pyramid structure network such as ResNet, MobileNet, or other custom-structured networks. The following example uses a custom-structured pyramid structure network to illustrate a specific implementation of the neural network model.

[0072] In some embodiments, the neural network model may include N feature extraction layers and N decoders corresponding to the feature extraction layers, and each feature extraction layer is used to output a reference feature map at a corresponding scale.

[0073] Furthermore, based on the reference feature map and the initial feature map, the process of performing multi-level optical flow optimization on the prior optical flow using a neural network model to obtain the predicted optical flow of the intermediate frame to be inserted includes: following the order of feature refinement from coarse to fine features, each decoder outputs the final predicted optical flow and intermediate feature map of the current level based on the final predicted optical flow and intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder. The final predicted optical flow of the previous level of the first level decoder is the prior optical flow, the intermediate feature map of the previous level of the first level decoder is the initial feature map, and the final predicted optical flow of the current level output by the last level decoder is the predicted optical flow of the intermediate frame to be inserted.

[0074] Figure 2 This diagram illustrates a video frame interpolation method provided by an embodiment of the present disclosure, incorporating a neural network model. Wherein, in Figure 2 The following example uses reference frame groups I0 and I1 as an illustration. It is the intermediate frame to be inserted between I0 and I1, and MV is the motion vector information corresponding to I0 and I1.

[0075] See Figure 2 The prior optical flow / occlusion mask 207 of the intermediate frame It to be inserted is obtained by warping 206 according to MV.

[0076] like Figure 2 As shown, the neural network model 200 includes multiple feature extraction layers 201 ( Figure 2 Taking a four-layer feature extraction layer as an example, and a decoder 202 corresponding to each feature extraction layer 201. The multiple feature extraction layers 201 extract image features from the reference frame group sequentially from fine to coarse, obtaining multiple reference feature maps at different scales, from low-level features to high-level features. The multiple decoders 202 correspond to the multiple feature extraction layers 201 in reverse order; that is, the first-level decoder corresponds to the last feature extraction layer, and so on, with the last-level decoder corresponding to the first feature extraction layer. It should be noted that the reference frame group includes at least a first reference frame and a second reference frame. Each feature extraction layer 201 extracts multi-scale reference feature maps of the first reference frame and the second reference frame, respectively.

[0077] according to Figure 2 It can be seen that the input of each decoder 202 is the output of the previous level decoder 202, the reference feature map output by the feature extraction layer 201 corresponding to that decoder 202, and the occlusion mask and / or prior optical flow obtained by warping the motion vector MV 206. The output of the decoder 202 can include the final predicted optical flow and intermediate feature maps. Specifically, the input of the first-level decoder 202 is the initial intermediate feature map 203, the reference feature map output by the feature extraction layer 201 corresponding to that first-level decoder 202, and the occlusion mask and / or prior optical flow obtained by warping the motion vector MV 206; that is, the intermediate feature map of the previous level of the first-level decoder 202 is the initial intermediate feature map 203. The final predicted optical flow of the current level output by the last level decoder is the predicted optical flow of the intermediate frame to be inserted.

[0078] This embodiment of the present disclosure performs multi-scale feature extraction on the reference frames in the reference frame group through multiple feature extraction layers in the neural network model, and performs multi-level optical flow optimization processing on the prior optical flow by combining the reference feature maps of each scale extracted by multiple feature extraction layers through multiple decoders in the neural network model, so as to make the final predicted optical flow more accurate.

[0079] The following is a specific implementation process for the process of each decoder 202 in the neural network model outputting the final predicted optical flow of the previous stage and the intermediate feature map of the current stage.

[0080] In some embodiments, the step of outputting the final predicted optical flow and intermediate feature map of the current level based on the final predicted optical flow and intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder includes: obtaining the initial predicted optical flow of the current level based on the final predicted optical flow and intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder; fusing the initial predicted optical flow and the prior optical flow of the current level to output the final predicted optical flow and intermediate feature map of the current level.

[0081] Specifically, in the optical flow optimization process of the current-level decoder, the final predicted optical flow and intermediate feature map of the previous level decoder are first used as the basis for matching and mapping features with the optical flow, and then combined with the reference feature map output by the feature extraction layer of the current-level decoder, to obtain the initial predicted optical flow of the current level. This initial predicted optical flow of the current level is the result obtained after feature linkage optimization between the current-level features and the optical flow of the previous level.

[0082] After obtaining the initial predicted optical flow of the current level, it can be fused with the prior optical flow obtained through forward warping. During the fusion process, the advantages of the prior optical flow—namely, its motion vector derivation verified by video coding, its alignment with real inter-frame motion trajectories, and its ability to resolve inconsistencies in time / displacement during non-uniform motion—are fully utilized. Furthermore, the advantages of the current-level initial predicted optical flow—which has undergone iterative optimization at the previous level and adapts to the scale and feature information of the current-level reference feature map—are combined to achieve complementary fusion of the optical flow information optimized by model iteration and the prior motion information, effectively avoiding the accuracy defects of single optical flow information in complex motion scenarios.

[0083] This fusion process uses prior optical flow as the core motion benchmark to calibrate and optimize the initial predicted optical flow of the current level. Simultaneously, the initial predicted optical flow of the current level compensates for detail deviations in local feature matching of the prior optical flow, preserving the original motion accuracy of the prior optical flow while incorporating the optical flow details optimized at each level of the model. After fusion, the fused and optimized final predicted optical flow of the current level is output. This final predicted optical flow combines the motion realism of prior information with the accuracy advantages of multi-level iterative optimization, significantly improving the accuracy and robustness of optical flow estimation. Simultaneously, an intermediate feature map of the current level, matching the final predicted optical flow, is generated. This intermediate feature map is the feature result after fusing optical flow information with current-level reference features and previous-level intermediate features, providing a suitable feature and optical flow input basis for optical flow optimization in the next level decoder.

[0084] Figure 3 This is a schematic diagram of a method for generating initial predicted optical flow provided in this disclosure; Figure 4 This is a schematic diagram of a method for predicting the final optical flow based on the initial predicted optical flow output provided in this disclosure.

[0085] The following is combined Figure 3 and Figure 4 The process of outputting the final predicted optical flow of the previous stage and the intermediate feature map of the current stage for each decoder is described in detail.

[0086] In some embodiments, the neural network model includes a convolutional network. The step of obtaining the initial predicted optical flow of the current level based on the final predicted optical flow of the previous level output by the previous level decoder, the intermediate feature map of the previous level, and the reference feature map output by the feature extraction layer corresponding to the current level decoder includes: warping the reference feature map output by the feature extraction layer corresponding to the current level decoder based on the final predicted optical flow of the previous level output by the previous level decoder to obtain a warped reference feature map; calculating the correlation of the warped reference feature map to obtain correlated features; concatenating the correlated features, the warped reference feature map, and the intermediate feature map of the previous level to form a multi-source fusion feature, and inputting the formed multi-source fusion feature into the convolutional network to obtain the optical flow deviation; updating the final predicted optical flow of the previous level based on the optical flow deviation to obtain the initial predicted optical flow of the current level.

[0087] Specifically, firstly, based on the final predicted optical flow output by the previous level decoder, a warp operation is performed on the reference feature map output by the feature extraction layer corresponding to the current level decoder. The reference feature map is mapped to the feature space where the intermediate frame to be inserted is located according to the motion relationship indicated by the optical flow, thereby aligning the feature map with the feature position of the intermediate frame to be inserted, and thus obtaining the warped reference feature map.

[0088] Subsequently, feature correlation calculations are performed on the distorted reference feature map to construct a feature correlation volume, obtaining relevant features that can characterize the degree of matching and positional correspondence between features, providing a matching basis for subsequent estimation of optical flow deviation.

[0089] The obtained relevant features, the distorted reference feature map, and the intermediate feature map output from the previous level decoder are concatenated and fused along the channel dimension to form a multi-source fused feature that includes motion alignment information, feature matching information, and optimized feature information from the previous level. This multi-source fused feature is then input into a convolutional network for feature learning and inference, and the convolutional network outputs the optical flow deviation used to correct the optical flow trajectory.

[0090] Finally, based on the optical flow deviation obtained above, the final predicted optical flow of the previous level is updated and corrected to make the optical flow trajectory more consistent with the actual motion relationship at the current scale, thereby obtaining the initial predicted optical flow of the current level, which provides the basis for the optimized initial optical flow of this level for subsequent fusion with the prior optical flow.

[0091] See Figure 3 The prior optical flow includes a first prior optical flow from the interpolated intermediate frame to the first reference frame and a second prior optical flow from the interpolated intermediate frame to the second reference frame. Correspondingly, the final predicted optical flow output by the decoder includes a first final predicted optical flow corresponding to the first prior optical flow and a second final predicted optical flow corresponding to the second prior optical flow.

[0092] exist Figure 3In this diagram, pflowt0 301 represents the first final predicted optical flow output by the previous stage decoder, pflowt1 302 represents the second final predicted optical flow output by the previous stage decoder; F0 303 represents the first reference feature map corresponding to the first reference frame in the current stage decoder, F1 304 represents the second reference feature map corresponding to the second reference frame in the current stage decoder, Ft305 represents the intermediate feature map corresponding to the intermediate frame to be inserted output by the previous stage decoder; Pflowt0 312 represents the first initial predicted optical flow of the current stage output by the current stage decoder, and Pflowt0 313 represents the second initial predicted optical flow of the current stage output by the current stage decoder.

[0093] The decoder's inputs include pflowt0 301, pflowt1 302, F0 303, F1 304, and Ft 305, and the decoder's intermediate outputs include pflowt0 312 and pflowt1 313.

[0094] Specifically, firstly, based on the first final predicted optical flow pflowt0 301 output from the previous level decoder, the first reference feature map F0 303 is warped (206) and based on the second final predicted optical flow pflowt1 302 output from the previous level decoder, the second reference feature map F1 304 is forward-warped (307) to obtain warped reference feature maps corresponding to the first reference feature map F0 303 and the second reference feature map F1 304, respectively. Then, the correlation of the warped reference feature maps can be calculated through the cost body 308, or the correlation can be calculated through other methods to obtain the relevant features. Further, the relevant features obtained from the cost body 308, the warped reference feature maps obtained from the warp (206) operation and the forward-warp (307) operation, and the intermediate feature map Ft 305 from the previous level are concatenated (204) to form a multi-source fusion feature, and the formed multi-source fusion feature is input into the convolutional network Conv1. 310. Obtain the optical flow deviation; finally, concatenate the optical flow deviation with the first final predicted optical flow pflowt0 301 of the previous level by channel dimension to obtain the first initial predicted optical flow Pflowt0 312 of the current level, and concatenate the optical flow deviation with the second final predicted optical flow pflowt0 301 of the previous level by channel dimension to obtain the second initial predicted optical flow Pflowt1 313 of the current level.

[0095] Furthermore, in some embodiments, the step of fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow and the current-level intermediate feature map includes: fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow; and performing a warp operation on the previous-level intermediate feature map based on the current-level final predicted optical flow to obtain the current-level intermediate feature map.

[0096] Specifically, firstly, the current-level initial predicted optical flow and the prior optical flow are fused. By fully combining the motion prior information of the prior optical flow with the optimization information of the current-level initial predicted optical flow, a more accurate and stable current-level final predicted optical flow is obtained. Then, based on the obtained current-level final predicted optical flow, the previous-level intermediate feature map is warped to align its spatial position according to the motion relationship represented by the current-level final predicted optical flow, thereby obtaining a current-level intermediate feature map that matches the current-level feature scale and motion information.

[0097] Furthermore, in some embodiments, the step of fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow includes: determining the weighting coefficients of the current-level initial predicted optical flow and the prior optical flow; and fusing the current-level initial predicted optical flow and the prior optical flow according to the weighting coefficients to output the current-level final predicted optical flow.

[0098] Specifically, in the current-level decoding process of multi-level optical flow optimization, in order to efficiently fuse the current-level initial predicted optical flow and the prior optical flow and give full play to the advantages of their motion information, it is necessary to first determine the weight coefficients of the two types of optical flow, then complete the fusion based on the weight coefficients, and finally output the current-level final predicted optical flow.

[0099] In some embodiments, the weighting coefficients can be dynamically allocated based on the information reliability and adaptability of the two types of optical flows, ensuring that the more reliable optical flow occupies a higher weight. Specifically, the initial predicted optical flow weight and prior optical flow weight can be obtained by combining the current-level reference feature map, the previous-level intermediate feature map, and the consistency of the motion trajectory and pixel matching degree of the two types of optical flows through the neural network weight prediction branch, thus ensuring the rationality of the fusion.

[0100] If the prior optical flow has a high matching degree with the reference feature map pixels, it indicates that the prior optical flow is more reliable. Therefore, the prior optical flow is assigned a higher weight to retain its advantage of conforming to the real motion trajectory. If the current level initial prediction optical flow is more suitable for the current level scale and local details, the weight of the prediction optical flow is appropriately increased to give full play to the advantages of decoder layer optimization and avoid the limitations of a single optical flow.

[0101] Finally, optical flow fusion is completed based on weighted coefficients, and the result is output. The fusion can be performed using a weighted summation method, which calculates the initial predicted optical flow and the prior optical flow of the current level by weighting the predicted optical flow and the prior optical flow respectively, and then obtains the fusion result by pixel-level superposition, which is the final predicted optical flow of the current level.

[0102] This disclosure fuses the initial predicted optical flow and the prior optical flow of the current stage based on the weight coefficients of the prior optical flow and the predicted optical flow. The output final predicted optical flow of the current stage combines prior reliability and optimization accuracy. It can be used as input for subsequent warp operations to generate intermediate feature maps of the current stage, while providing high-quality input for the next stage decoder, ensuring accurate progress of multi-stage optical flow optimization.

[0103] Furthermore, in some embodiments, the step of determining the weight coefficients of the current-level initial predicted optical flow and the prior optical flow includes: converting the current-level initial predicted optical flow and the prior optical flow into unfolded predicted optical flow and unfolded prior optical flow in block representations containing spatial neighborhood information, respectively; performing a warp operation on the reference feature map output by the feature extraction layer corresponding to the current-level decoder based on the unfolded predicted optical flow and the unfolded prior optical flow, respectively, to obtain a warped predicted feature map and a warped prior feature map; calculating the first similarity of the pixel neighborhood in the warped predicted feature map and the second similarity of the pixel neighborhood in the warped prior feature map; concatenating the first similarity, the second similarity, the unfolded predicted optical flow, the unfolded prior optical flow, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder through channel concatenation to form a fusion tensor; and determining the weight coefficients of the prior optical flow and the current-level initial predicted optical flow based on the fusion tensor.

[0104] Specifically, firstly, the initial predicted optical flow and the prior optical flow of the current level are transformed into block representations to obtain the unfolded predicted optical flow and the unfolded prior optical flow. Specifically, both types of optical flow are divided into several spatial neighborhood blocks according to a preset size. Each neighborhood block contains local spatial neighborhood information of the corresponding optical flow. Through this block unfolding operation, the original single pixel-level optical flow information is transformed into a block representation containing local spatial correlation features, thereby more comprehensively preserving the spatial motion correlation of the optical flow and providing richer information support for subsequent feature distortion and similarity calculation.

[0105] Then, based on the unfolded predicted optical flow and the unfolded prior optical flow, respectively, a warp operation is performed on the reference feature map output by the feature extraction layer corresponding to the current level decoder. According to the motion trajectory and spatial position relationship represented by the two types of unfolded optical flows, the reference feature map is mapped to the feature space corresponding to the unfolded predicted optical flow and the unfolded prior optical flow, respectively, to achieve spatial alignment between the reference feature map and the two types of unfolded optical flows. Finally, the warped predicted feature map and the warped prior feature map are obtained, ensuring the accuracy of subsequent similarity calculations.

[0106] Furthermore, the pixel neighborhood similarities of the distorted predicted feature map and the distorted prior feature map are calculated, namely, the first similarity and the second similarity. For the distorted predicted feature map, the feature similarity of each pixel with its pixels within a preset neighborhood is calculated to obtain the first similarity, which characterizes the local pixel consistency of the feature map. Similarly, for the distorted prior feature map, the same neighborhood range and calculation method are used to calculate the neighborhood feature similarity of each pixel to obtain the second similarity. This similarity index can directly reflect the local feature reliability of the two types of distorted feature maps. The higher the similarity, the stronger the feature matching degree and stability of the corresponding region.

[0107] After similarity calculation, the first similarity, second similarity, unfolded predicted optical flow, unfolded prior optical flow, and the reference feature map output by the feature extraction layer corresponding to the current level decoder are concatenated and fused along the channel dimension to form a fusion tensor. This fusion tensor includes at least: alignment quality cues, local motion cues, appearance content cues, and implicit spatial location cues.

[0108] The alignment quality cue corresponds to the first and second similarities, directly measuring the local reliability of the predicted optical flow and the prior optical flow. High similarity indicates that in the current local region, after distorting the reference feature using the optical flow, the resulting feature is highly consistent with the target feature (or another reference feature), indicating accurate optical flow estimation and good alignment in that region. Low similarity indicates that the optical flow may be inaccurate in that region, possibly due to occlusion, large displacement, or motion blur. The alignment quality cue is the most direct and crucial signal determining the weights.

[0109] Local motion cues correspond to the predicted optical flow and the prior optical flow after unfolding. These cues provide the original motion vector information of the current local region, including not only the motion of the center point but also the motion distribution and trends of its neighborhood (such as rotational deformation and shearing). Local motion cues help the network understand the motion patterns of the region (including translation, rotation, or complex deformation) and aid in interpreting similarity. For example, a low similarity in a region, if its optical flow value is large (large displacement), might be interpreted as a difficulty in matching; a smooth optical flow but low similarity might suggest occlusion. Furthermore, as local motion cues serve as direct input to weight prediction, the network needs to adjust the fusion strategy based on the characteristics of the motion itself (such as amplitude and directional consistency).

[0110] The appearance content cue corresponds to the original reference feature map (e.g., features from F0 or F1), representing the visual content information of the spatial location itself, such as texture, edges, color, and semantic features. The appearance content cue provides the basis for discrimination, and similarity is calculated based on it. This cue can handle content dependencies; different regions (e.g., textured areas versus smooth skies) have different sensitivity and reliability criteria for optical flow errors, requiring the network to incorporate this appearance content to interpret motion and alignment quality. Furthermore, the appearance content cue can also be used to detect special regions. For example, if features indicate an object edge (abrupt change in content), the likelihood of motion discontinuity or occlusion is higher, requiring more careful integration of optical flow.

[0111] Implicit spatial location cues retain their original spatial arrangement in the concatenated tensor, and are not used as explicit channel inputs, but the spatial coordinates of the feature maps themselves are implicit information. This allows the network to learn priors related to image location. For example, image edges, central regions, or areas where specific types of objects frequently appear may exhibit different optical flow fusion patterns.

[0112] This fusion tensor integrates the blocky spatial information of optical flow, the neighborhood similarity information of the distorted features, and the original reference feature information, achieving comprehensive integration of multi-source features and providing sufficient feature basis for the accurate determination of subsequent weight coefficients.

[0113] Finally, the fusion tensor formed above is input into a pre-defined convolutional network. Through feature learning and inference, the network adaptively learns and outputs the weight coefficients corresponding to the initial predicted optical flow and the prior optical flow of the current stage. Based on the multi-source features in the fusion tensor, the convolutional network automatically judges the reliability of the two types of optical flows and their corresponding features, assigning higher weight coefficients to the optical flows with higher reliability. Furthermore, the two types of weight coefficients satisfy the normalization condition, ensuring the rationality and stability of the subsequent optical flow fusion process, thus laying the foundation for the efficient fusion of the initial predicted optical flow and the prior optical flow of the current stage.

[0114] See Figure 4 flowt0 401 represents the first prior optical flow, flowt1 402 represents the second prior optical flow; F0 303 represents the first reference feature map corresponding to the first reference frame in the current level decoder, F1 304 represents the second reference feature map corresponding to the second reference frame in the current level decoder; Pflowt0 312 represents the current level first initial prediction optical flow output by the current level decoder, and Pflowt0 313 represents the current level second initial prediction optical flow output by the current level decoder.

[0115] like Figure 4 As shown, firstly, the first prior optical flow flowt0 401, the second prior optical flow flowt1 402, the current-level first initial prediction optical flow Pflowt0 312, and the current-level second initial prediction optical flow Pflowt0 313 are expanded using operation 407. Then, the first reference feature map F0 303 is warped using operation 206 based on the first prior optical flow flowt0 401 and the current-level first initial prediction optical flow Pflowt0 312, and the second reference feature map F0 is warped using operation 206 based on the second prior optical flow flowt1 402 and the current-level second initial prediction optical flow Pflowt0 313. 304 performs a warp operation 206; further, a similarity correlation 409 is calculated, specifically calculating the first similarity of the warped predicted feature map (including the warped first predicted feature map and the warped second predicted feature map) and the second similarity of the warped prior feature map (including the warped first prior feature map and the warped second prior feature map); further still, the first and second similarities obtained from correlation 409, the unfolded predicted optical flow and the unfolded prior optical flow obtained from unfolding 407, and the first reference feature map F0 303 and the second reference feature map F1 are combined. 304 is concatenated with 204 to form a fusion tensor, which is then input into Conv2411. The weighting coefficients of the prior optical flow and the current-level initial predicted optical flow are obtained through softmax412. Finally, the prior optical flow is concatenated with the unfolded first and second prior optical flows based on the weighting coefficients of the prior optical flow to obtain the final prior optical flow. The predicted optical flow is then concatenated with the unfolded first and second predicted optical flows based on the weighting coefficients of the predicted optical flow to obtain the final predicted optical flow. Furthermore, using the final predicted optical flow and the final prior optical flow, the predicted optical flow of the intermediate frame to be inserted is obtained.

[0116] The embodiments provided in this disclosure can accurately determine the weight coefficients of the current-level initial predicted optical flow and the prior optical flow, fully combine the spatial neighborhood information and feature reliability of the two types of optical flow, improve the rationality of weight allocation, and thus ensure the accuracy of subsequent optical flow fusion.

[0117] In some embodiments, the prior information includes a prior optical flow and an occlusion mask, the reference frame group includes a first reference frame and a second reference frame, the prior optical flow includes a first prior optical flow from the intermediary frame to be inserted to the first reference frame and a second prior optical flow from the intermediary frame to be inserted to the second reference frame; correspondingly, the intermediary frame optical flow includes a first intermediary frame optical flow corresponding to the first prior optical flow and a second intermediary frame optical flow corresponding to the second prior optical flow.

[0118] The step of generating an intermediate frame to be inserted based on the predicted optical flow of the intermediate frame and the reference frame group includes: performing a forward warp operation on the first reference frame based on the optical flow of the first intermediate frame to obtain a first warp feature map corresponding to the intermediate frame to be inserted; performing a forward warp operation on the second reference frame based on the optical flow of the second intermediate frame to obtain a second warp feature map corresponding to the intermediate frame to be inserted; and fusing the first warp feature map and the second warp feature map based on an occlusion mask to obtain the intermediate frame to be inserted.

[0119] Specifically, in the video frame interpolation process, the intermediate frame to be interpolated is obtained by filtering the reference frame distortion and occlusion areas and fusing effective features to obtain an intermediate frame result that fits the real frame. Specifically, it is based on the optical flow of the first intermediate frame, the optical flow of the second intermediate frame, the first reference frame, the second reference frame, and the occlusion mask.

[0120] Specifically, firstly, based on the obtained optical flow of the first intermediate frame, a forward warp operation is performed on the first reference frame to obtain a first warp feature map corresponding to the intermediate frame to be inserted. The optical flow of the first intermediate frame represents the pixel motion trajectory and displacement relationship between the first reference frame and the intermediate frame to be inserted. The core of the forward warp operation is to map the pixels and feature information of the first reference frame point by point to the feature space corresponding to the intermediate frame to be inserted, according to the motion law indicated by the optical flow of the first intermediate frame. This achieves spatial alignment between the first reference frame and the intermediate frame to be inserted. The resulting first warp feature map contains all the effective feature information of the first reference frame adapting to the position of the intermediate frame to be inserted, providing forward reference feature support for subsequent intermediate frame fusion.

[0121] Similarly, based on the obtained second intermediate frame optical flow, the same forward warp processing can be performed on the second reference frame to obtain a second warp feature map corresponding to the intermediate frame to be inserted. The second intermediate frame optical flow characterizes the pixel motion trajectory and displacement relationship between the second reference frame and the intermediate frame to be inserted. This forward warp processing is consistent with the warp operation logic of the first reference frame; both align the pixels and feature information of the second reference frame to the feature space of the intermediate frame to be inserted through displacement mapping indicated by the optical flow, ensuring that the second warp feature map and the first warp feature map are at the same spatial scale.

[0122] After acquiring the two types of warp feature maps, the first and second warp feature maps can be further fused based on the occlusion mask to obtain the intermediate frame to be inserted. The occlusion mask is used to mark the occluded and valid regions in the two types of warp feature maps, accurately distinguishing invalid warp features caused by inter-frame motion (such as object occlusion or rapid movement), and avoiding interference from invalid features in occluded regions with the accuracy of intermediate frame generation. During the fusion process, the occlusion mask plays a filtering and weighting role. For features marked as valid regions by the mask, the corresponding valid features in the first and second warp feature maps are retained and fused complementaryly. For features marked as occluded regions by the mask, invalid warp features are automatically filtered out, retaining only the unoccluded valid features for fusion.

[0123] like Figure 2 As shown, the predicted optical flow output by the last stage decoder 202 corresponds to the intermediate frame optical flow of the intermediate frame to be inserted (including the first intermediate frame optical flow and the second intermediate frame optical flow). Based on the predicted optical flow output by the last stage decoder 202 and the reference frame group (including reference frame I0 and reference frame I1), the final intermediate frame to be inserted It is obtained through the construction operation 205.

[0124] The final intermediate frame to be inserted is It==warp(I0, flow_t0). (1-mask)+warp(I1,flow_t1) mask+res; where I0 represents the first reference frame, flow_t0 represents the optical flow of the first intermediate frame, I1 represents the second reference frame, flow_t1 represents the optical flow of the second intermediate frame, mask represents the occlusion mask, and res represents the residual.

[0125] The embodiments provided in this disclosure can fully integrate the effective features of the first reference frame and the second reference frame to adapt to the intermediate frame to be inserted, while avoiding feature deviations caused by occlusion areas. This ensures that the generated intermediate frame to be inserted not only matches the texture and color features of the preceding and following reference frames, but also conforms to the inter-frame motion rules. The clarity and realism of the image are effectively guaranteed, thus meeting the visual effect requirements of video frame interpolation.

[0126] In addition, feature extraction of the reference frame group can be performed by combining the depth information corresponding to each reference frame in the reference frame group.

[0127] In some embodiments, the step of performing multi-scale feature extraction on the reference frame group includes: concatenating the reference frame group with corresponding depth information to obtain a concatenated video frame; and performing multi-scale feature extraction on the concatenated video frame.

[0128] Specifically, the depth information corresponding to the reference frame group can be directly obtained from the G-buffer. First, the reference frame group and its corresponding depth information are concatenated along the channel dimension to obtain the concatenated video frame. The depth information of the reference frame group is used to supplement the spatial position differences of pixels within the reference frame group, clarify the hierarchical relationships and spatial distances of different objects, and mitigate feature extraction biases in occluded scenes. During the concatenation process, the spatial resolution (width and height) of the reference frame group and the depth information is kept consistent, and concatenation is performed only along the channel dimension. This ensures that the concatenated video frame simultaneously contains the texture features, color features, and spatial depth features of the reference frames, providing a more comprehensive and richer input basis for subsequent multi-scale feature extraction and improving the completeness of feature representation. After the concatenation process is completed, multi-scale feature extraction is performed on the resulting concatenated video frame.

[0129] like Figure 2 As shown, firstly, the reference frame I0 and the corresponding depth information D0 are concatenated using operation 204, and the reference frame I1 and the corresponding depth information D1 are concatenated using operation 204. Then, the concatenated video frame is input into the neural network model 200 to perform multi-scale feature extraction through multiple feature extraction layers 201 in the neural network model 200.

[0130] This embodiment of the present disclosure obtains a stitched video frame by stitching together a reference frame group with corresponding depth information, and performs multi-scale feature extraction on the stitched video frame. This can comprehensively capture the multi-scale feature information of the reference frame group, improve the accuracy of subsequent optical flow estimation and intermediate frame generation, and adapt to inter-frame motion scenes of different scales.

[0131] In some embodiments, the method may further include training a neural network model, wherein the training process of the neural network model includes: acquiring at least one set of sample reference frames; each set of sample reference frames includes: a first sample reference frame, a second sample reference frame, and a sample intermediate frame located between the first sample reference frame and the second sample reference frame; performing forward warp processing on at least one set of sample reference frames and the corresponding sample motion vector information to obtain sample prior information of the sample intermediate frame; and using at least one set of sample reference frames and the sample prior information as training data to iteratively train a preset initial neural network model to obtain a neural network model.

[0132] In some embodiments, the at least one set of sample reference frames and sample prior information are used as training data to iteratively train a preset initial neural network model to obtain a neural network model, including: the initial neural network model predicts the sample predicted optical flow of the sample intermediate frames of the corresponding sample reference frame group based on the first sample reference frame, the second sample reference frame, and the sample prior information of each set of sample reference frames; and generates predicted intermediate frames based on the sample predicted optical flow and the corresponding first and second sample reference frames; calculates a loss value based on the predicted intermediate frames and the sample intermediate frames; and iteratively trains the initial neural network model by adjusting the network parameters based on the loss value; when the loss value reaches a target loss value, the network parameters corresponding to the target loss value are saved as target network parameters, and the initial neural network model with the target network parameters is used as the neural network model.

[0133] It should be noted that the training process of the neural network model is basically the same as the process of predicting intermediate frames based on the trained neural network model in the above embodiment. Both require obtaining the prior information of the intermediate frame based on the reference frame group and the corresponding motion vector information, and then inputting the prior information and the reference frame group into the model to obtain the predicted intermediate frame. The only difference between the two is that the method of calculating the prior information is slightly different. The other processes are exactly the same, and will not be repeated here.

[0134] In the above embodiments, during the process of predicting intermediate frames based on the trained neural network model, the initial optical flow information is calculated based on the proportional relationship between the first relative position parameter between the second reference frame and the intermediate frame to be inserted, and the second relative position parameter between the second reference frame and the first reference frame. The calculated result is then subjected to forward warping to obtain the prior information of the intermediate frame to be inserted. During the training process of the neural network model, the intermediate frame corresponds to the real image frame. Therefore, the calculation of the prior information can be based on the real intermediate frame, thereby making the intermediate frame predicted by the trained neural network model more accurate.

[0135] In some embodiments, forward warping is performed on at least one set of sample reference frames and their corresponding sample motion vector information to obtain sample prior information for intermediate frames. This includes: determining initial sample optical flow information for intermediate frames based on the motion vector information of intermediate frames and the motion vector information of second sample reference frames; and performing forward warping on the initial sample optical flow information to obtain sample prior information for intermediate frames. For example, if a second frame I2 is inserted between the first frame I1 and the third frame I3, then during the training phase, the sample reference frame set may include the first frame I1 and the third frame I3, and the sample intermediate frame is the second frame I2. Here, the first frame I1, the second frame I2, and the third frame I3 are all real frames in the video frame sequence and all correspond to real motion vector information.

[0136] Since the second frame I2 to be inserted corresponds to a real image frame during the training phase, it is not necessary to determine the prior information based on the proportional relationship as in the above embodiment. Instead, the prior information of the sample can be calculated directly based on the motion vector information of the intermediate sample frame and the motion vector information of the sample reference frame group. Specifically, during the training phase, taking the insertion of the second frame I2 between the first frame I1 and the third frame I3 as an example, flow_1t (referring to the optical flow from the third frame to the second frame) and flow_t0 (referring to the optical flow from the second frame to the first frame) can be directly calculated based on the motion vector information of the first frame I1, the second frame I2, and the third frame I3. Then, through a forward warp operation, the prior optical flow flow_t1 is obtained based on flow_1t. In other words, since the intermediate frame I2 is known to be a known ground truth frame during training, flow1t can be calculated based on the real second frame I, thus obtaining a more accurate prior optical flow.

[0137] The video frame interpolation methods provided in the above embodiments can be applied to training scenarios for video frame interpolation, and also to video frame interpolation scenarios where an intermediate frame to be interpolated is inserted between adjacent known frames. The video frame interpolation method provided in this disclosure is described in detail below with reference to a training scenario. The method specifically includes the following steps: First, obtain training data.

[0138] Specifically, assume the video frame sequence includes video frames L0, L1, L2, and the corresponding motion vector information mv0, mv1, mv2 for each video frame. flow_21 can be obtained based on mv2, and flow10 can be calculated based on mv1. flow_21 can be forward warped to obtain flow_12 and a mask (occlusion region). Furthermore, flow_20 = flow_21 + warp(flow_21, flow_10), and w_flow_10 = warp(flow_20, flow_12).

[0139] Secondly, training is performed based on the training data.

[0140] Specifically, based on L0 and L2, the intermediate frame L1 is estimated. flow_12 and w_flow_10 are used as prior information to train the neural network model, addressing the training degradation (blurred interpolation) caused by the temporal inconsistency between the predicted frame and the true value (L1) due to non-uniform motion. The neural network model can predict the optical flow based on the prior information. During training, flow_1t = (1-t). Similarly, we can obtain flow_t1 and flow_t0 from flow_10.

[0141] Specifically, we first use a neural network model to extract features at different scales from LL0 and L2 using a feature pyramid approach, and then estimate optical flow from high-level features to low-level features, gradually optimizing the results. For details, please refer to the relevant content above; it will not be repeated here.

[0142] In some embodiments, the depth map information can be concatted with LL0 and L2 frames, and then features of different scales can be extracted from LL0 and L2 using a feature pyramid method to improve the frame interpolation effect.

[0143] Furthermore, in the process of estimating optical flow from high-level features to low-level features and progressively optimizing the effect, the initial feature map of the intermediate frame to be inserted can be obtained first based on the extracted highest-level feature map: ft=warp(f0, flow_t1). (1-mask)+warp(f1, flow_t0) The optical flow is then gradually optimized based on the initial feature map to obtain the final predicted optical flow.

[0144] Specifically, after the feature maps at each scale are warped by the previous level of optical flow, they are processed through concat, multi-layer convolutional structures (such as resblock), and transConv (deconvolution, stride=2) to estimate the optical flow deviations delt_t0 and delt_t1. Then, the predicted optical flow of the current level is obtained based on the optical flow deviations: Pflow_t0 = resize(Pflow_t0, scale=2). 2+delt_t0,Pflow_t1= resize(Pflow_t1, scale=2) 2 + delt_t1. Then, the coefficients alpha0 and alpah1 are obtained through sortmax, and the predicted optical flow Pflow and the prior optical flow flow are fused based on these coefficients to obtain the final predicted optical flow for the current stage, thus achieving high-precision optical flow. Where flow_t0 = sum(cat(flow_t0, Pflow_t0)). alpha0).Flow_t1=sum(cat(flow_t1,Pflow_t1) alpha1).

[0145] Finally, output flow_t0, flow_t1, res, and mask, and obtain the intermediate frame to be inserted based on the output: It=warp(L0, flow_t0). (1-mask)+warp(L1, flow_t1) mask+res.

[0146] Other details of the embodiments disclosed herein can be found in the relevant content above, and will not be repeated here.

[0147] The video frame interpolation method disclosed herein uses prior optical flow as a prior training model, which can effectively solve the blurring effect caused by acceleration, deceleration and turning problems during frame interpolation training. At the same time, the prior optical flow and the predicted optical flow are fused together as the final predicted optical flow, which can effectively integrate the information of the predicted optical flow and the prior optical flow, improve the accuracy of the final predicted optical flow, and thus obtain a clear frame interpolation effect.

[0148] Based on the same inventive concept as the video frame interpolation method described above, this disclosure also provides a video frame interpolation apparatus.

[0149] Figure 5 This is a structural block diagram of a video frame interpolation device provided in an embodiment of the present disclosure.

[0150] Further reference Figure 5 As an implementation of the methods shown in the above figures, this device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0151] like Figure 5 As shown, the frame interpolation device 500 in this embodiment may include: a calculation module 501 and a prediction module 502.

[0152] The calculation module 501 is configured to perform forward warp processing on the reference frame group and the corresponding motion vector information to obtain the prior information of the intermediate frame to be inserted; wherein the intermediate frame to be inserted is located between the two reference frames contained in the reference frame group; the prediction module 502 is configured to input the reference frame group and the prior information into the neural network model so that the neural network model predicts the predicted optical flow of the intermediate frame to be inserted based on the reference frame group and the prior information, and generates the intermediate frame to be inserted based on the predicted optical flow of the intermediate frame to be inserted and the reference frame group.

[0153] In this embodiment, the specific processing of the calculation module 501 and the prediction module 502 in the frame interpolation device 500 and the resulting technical effects can be referred to respectively. Figure 1 The relevant descriptions of steps 101-102 in the corresponding embodiments will not be repeated here.

[0154] The frame interpolation apparatus 500 provided in this embodiment first obtains prior information of the intermediate frame to be interpolated through forward warping based on a reference frame and corresponding motion vector information; then, it predicts the optical flow of the intermediate frame to be interpolated through a neural network model based on the prior information; finally, it synthesizes the reference frame into the intermediate frame to be interpolated based on the obtained predicted optical flow. This embodiment of the disclosure introduces prior information to transform the ill-conditioned problem caused by the uncertainty of motion state during frame interpolation into a deterministic optimization problem, effectively avoiding the blurry frame interpolation effect in traditional methods and achieving clearer frame interpolation output. Furthermore, the source of the prior information is the motion vector of the reference frame, which is the inter-frame displacement data verified during video encoding and can directly reflect the real motion trajectory, reducing the randomness of motion trajectory prediction from the root and improving the reliability of the prior information. Furthermore, for the time / displacement inconsistency problem that is prone to occur in non-uniform motion scenarios such as acceleration, deceleration, and turning, the forward warp processing derives the prior optical flow through motion vector information, enabling the prior information to fuse the bidirectional motion correlation between the intermediate frame to be interpolated and the preceding and following reference frames, providing a clear initial benchmark for complex motion prediction, transforming the originally uncertain motion estimation into a deterministic coordinate mapping problem, and avoiding the trajectory distortion problem caused by the complexity of motion when directly calculating the optical flow.

[0155] In some embodiments, the prior information includes prior optical flow, and the prediction module 502 includes a feature extraction unit and a multi-level optical flow optimization unit. The feature extraction unit is configured to perform multi-scale feature extraction on a group of reference frames to obtain reference feature maps at N scales with feature fineness ranging from fine to coarse, where N is a positive integer greater than or equal to 2. The multi-level optical flow optimization unit is configured to perform multi-level optical flow optimization processing based on the reference feature maps and the prior optical flow to obtain the predicted optical flow of the intermediate frame to be inserted.

[0156] In some embodiments, the prior information further includes an occlusion mask. The feature extraction unit includes N feature extraction sub-units, which are configured to output reference feature maps at corresponding scales. The multi-level optical flow optimization unit includes a construction unit and N decoders corresponding to the feature extraction sub-units. The construction unit is configured to obtain an initial feature map of the intermediate frame to be inserted based on the reference feature map, the occlusion mask, and the prior optical flow. The N decoders are configured to output the final predicted optical flow and the intermediate feature map of the current level, based on the final predicted optical flow and the intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, in order from coarse features to fine features. The final predicted optical flow of the previous level of the first level decoder is the prior optical flow, the intermediate feature map of the previous level of the first level decoder is the initial feature map, and the final predicted optical flow of the current level output by the last level decoder is the predicted optical flow of the intermediate frame to be inserted.

[0157] In some embodiments, the decoder includes a fine-tuning unit and an optical flow fusion unit. The fine-tuning unit is configured to obtain the current-level initial predicted optical flow based on the previous-level final predicted optical flow and the previous-level intermediate feature map output by the previous-level decoder, and the reference feature map output by the feature extraction layer corresponding to the current-level decoder. The optical flow fusion unit is configured to fuse the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow and the current-level intermediate feature map.

[0158] In some embodiments, the fine-tuning unit includes a feature warping unit, a cost unit, a feature fusion unit, a convolution correction unit, and an optical flow update unit. Specifically, the feature warping unit is configured to warp the reference feature map output by the feature extraction subunit corresponding to the current-level decoder based on the final predicted optical flow output by the previous-level decoder, obtaining a warped reference feature map; the cost unit is configured to calculate the correlation of the warped reference features to obtain correlated features; the feature fusion unit is configured to concatenate the correlated features, the warped reference feature map, and the intermediate feature map from the previous level to form a multi-source fused feature; the convolution correction unit is configured to perform convolution processing on the multi-source fused feature to output an optical flow deviation; and the optical flow update unit is configured to update the final predicted optical flow from the previous level based on the optical flow deviation to obtain the initial predicted optical flow of the current level.

[0159] In some embodiments, the optical flow fusion unit includes a coefficient determination unit and an optical flow output unit. The coefficient determination unit is configured to determine the weighting coefficients of the current-level initial predicted optical flow and the prior optical flow. The optical flow output unit is configured to fuse the current-level initial predicted optical flow and the prior optical flow according to the weighting coefficients and output the current-level final predicted optical flow.

[0160] In some embodiments, the coefficient determination unit includes an optical flow unrolling subunit, a feature warping subunit, a similarity calculation subunit, a splicing and fusion subunit, and a weight prediction subunit. The optical flow unrolling subunit is configured to transform the current-level initial prediction optical flow and the prior optical flow respectively, obtaining an unrolled prediction optical flow and an unrolled prior optical flow in a block-like representation containing spatial neighborhood information. The feature warping subunit is configured to warp the reference feature map output by the feature extraction layer corresponding to the current-level decoder based on the unrolled optical flow, obtaining a warped prediction optical flow. The system calculates a predictive feature map and a distorted prior feature map. The similarity calculation subunit is configured to calculate the first similarity of pixel neighborhoods within the distorted predictive feature map and the second similarity of pixel neighborhoods within the distorted prior feature map. The stitching and fusion subunit is configured to stitch together the first similarity, the second similarity, the unfolded predictive optical flow, the unfolded prior optical flow, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, forming a fusion tensor. The weight prediction subunit is configured to determine the weight coefficients of the prior optical flow and the current level initial predictive optical flow based on the fusion tensor.

[0161] In some embodiments, the prior information includes prior optical flow and an occlusion mask. The reference frame group includes a first reference frame and a second reference frame. The prior optical flow includes a first prior optical flow from the intermediate frame to be inserted to the first reference frame and a second prior optical flow from the intermediate frame to be inserted to the second reference frame. Correspondingly, the intermediate frame optical flow includes a first intermediate frame optical flow corresponding to the first prior optical flow and a second intermediate frame optical flow corresponding to the second prior optical flow. The generation module includes a feature warp synthesis unit and a frame generation unit. The feature warp synthesis unit is configured to perform a forward warp operation on the first reference frame based on the first intermediate frame optical flow to obtain a first warp feature map corresponding to the intermediate frame to be inserted; and to perform a forward warp operation on the second reference frame based on the second intermediate frame optical flow to obtain a second warp feature map corresponding to the intermediate frame to be inserted. The frame generation unit is configured to fuse the first warp feature map and the second warp feature map based on the occlusion mask to obtain the intermediate frame to be inserted.

[0162] The specific implementation details and technical effects of the video frame interpolation device provided in this disclosure are the same as the implementation details and technical effects of the video frame interpolation method described above, and will not be repeated here.

[0163] This embodiment is a device embodiment corresponding to the above method embodiment. The video frame interpolation device provided in this embodiment first obtains the prior information of the intermediate frame to be interpolated by forward warping based on the reference frame and the corresponding motion vector information; then, it predicts the optical flow of the intermediate frame to be interpolated by a neural network model based on the prior information; finally, it synthesizes the reference frame into the intermediate frame to be interpolated based on the obtained predicted optical flow. This embodiment of the disclosure introduces prior information to transform the ill-conditioned problem caused by the uncertainty of motion state during frame interpolation into a deterministic optimization problem, effectively avoiding the blurry frame interpolation effect in traditional methods and achieving clearer frame interpolation output. Furthermore, the source of the prior information is the motion vector of the reference frame, which is the inter-frame displacement data verified during video encoding and can directly reflect the real motion trajectory, reducing the randomness of motion trajectory prediction from the root and improving the reliability of the prior information. Furthermore, for the time / displacement inconsistency problem that is prone to occur in non-uniform motion scenarios such as acceleration, deceleration, and turning, the forward warp processing derives the prior optical flow through motion vector information, enabling the prior information to fuse the bidirectional motion correlation between the intermediate frame to be interpolated and the preceding and following reference frames, providing a clear initial benchmark for complex motion prediction, transforming the originally uncertain motion estimation into a deterministic coordinate mapping problem, and avoiding the trajectory distortion problem caused by the complexity of motion when directly calculating the optical flow.

[0164] According to embodiments of this disclosure, this disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the video frame interpolation method described in any of the above embodiments.

[0165] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 600 of this embodiment includes a processor 601 and a memory 602; wherein, the memory 602 is used to store computer execution instructions; the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 600 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0166] In some embodiments, the memory 602 can be either standalone or integrated with the processor 601.

[0167] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0168] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0169] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0170] Bus 603 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0171] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the video frame interpolation method in any of the above method embodiments.

[0172] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video frame interpolation method according to any of the above embodiments.

[0173] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0174] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0175] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0176] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0177] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0178] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0179] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0180] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0181] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video frame interpolation method, characterized in that, The method includes: The reference frame group and its corresponding motion vector information are subjected to forward warp processing to obtain the prior information of the intermediate frame to be inserted; wherein the intermediate frame to be inserted is located between two reference frames contained in the reference frame group. The reference frame group and the prior information are input into the neural network model so that the neural network model predicts the predicted optical flow of the intermediate frame to be inserted based on the reference frame group and the prior information, and generates the intermediate frame to be inserted based on the predicted optical flow and the reference frame group.

2. The method according to claim 1, characterized in that, The two reference frames include a first reference frame and a second reference frame, and the second reference frame is located after the intermediate frame to be inserted; The forward warp processing of the reference frame group and its corresponding motion vector information to obtain prior information of the intermediate frame to be inserted includes: Based on the motion vector information of the second reference frame, determine the initial optical flow information from the second reference frame to the first reference frame; Based on the ratio of the first relative position parameter between the second reference frame and the intermediate frame to be inserted, and the second relative position parameter between the second reference frame and the first reference frame, the initial optical flow information is calculated, and the calculated result is subjected to forward warp processing to obtain the prior information of the intermediate frame to be inserted.

3. The method according to claim 1, characterized in that, The prior information includes prior optical flow; The step of inputting the reference frame group and the prior information into a neural network model, so that the neural network model predicts the predicted optical flow of the intermediate frame to be inserted based on the reference frame group and the prior information, includes: By inputting the reference frame group and the prior information into a neural network model, multi-scale feature extraction is performed on the reference frame group to obtain N reference feature maps with feature fineness ranging from fine to coarse; where N is a positive integer greater than or equal to 2; Based on the reference feature map and the prior optical flow, the predicted optical flow of the intermediate frame to be inserted is obtained by performing multi-level optical flow optimization processing through the neural network model.

4. The method according to claim 3, characterized in that, The prior information also includes an occlusion mask; The step of performing multi-level optical flow optimization processing based on the reference feature map and the prior optical flow to obtain the predicted optical flow of the intermediate frame to be inserted includes: The neural network model is used to process the reference feature map, the occlusion mask, and the prior optical flow to obtain the initial feature map of the intermediate frame to be inserted. Based on the reference feature map and the initial feature map, the prior optical flow is optimized through a neural network model to obtain the predicted optical flow of the intermediate frame to be inserted.

5. The method according to claim 4, characterized in that, The neural network model includes N feature extraction layers and N decoders corresponding to the feature extraction layers, and each feature extraction layer is used to output a reference feature map at a corresponding scale; The step of performing multi-level optical flow optimization processing on the prior optical flow using a neural network model based on the reference feature map and the initial feature map to obtain the predicted optical flow of the intermediate frame to be inserted includes: Following the order of feature refinement from coarse to fine, each decoder outputs the current-level final predicted optical flow and the current-level intermediate feature map based on the previous-level decoder's output final predicted optical flow and intermediate feature map, as well as the reference feature map output by the feature extraction layer corresponding to the current-level decoder. Specifically, the previous-level final predicted optical flow of the first-level decoder is the prior optical flow, the previous-level intermediate feature map of the first-level decoder is the initial feature map, and the current-level final predicted optical flow output by the last-level decoder is the predicted optical flow of the intermediate frame to be inserted.

6. The method according to claim 5, characterized in that, The process of outputting the final predicted optical flow and intermediate feature map of the current level based on the output of the previous level decoder, the previous level intermediate feature map, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, includes: Based on the final predicted optical flow and intermediate feature map of the previous level output by the previous level decoder, and the reference feature map output by the feature extraction layer corresponding to the current level decoder, the initial predicted optical flow of the current level is obtained. The initial predicted optical flow of the current level and the prior optical flow are fused to output the final predicted optical flow of the current level and the intermediate feature map of the current level.

7. The method according to claim 6, characterized in that, The neural network model includes a convolutional network; the process of obtaining the initial predicted optical flow of the current level based on the final predicted optical flow output of the previous level decoder, the intermediate feature map of the previous level, and the reference feature map output by the feature extraction layer corresponding to the current level decoder includes: Based on the final predicted optical flow output by the previous level decoder, the reference feature map output by the feature extraction layer corresponding to the current level decoder is warped to obtain the warped reference feature map. The correlation of the distorted reference feature map is calculated to obtain the relevant features; The relevant features, the distorted reference feature map, and the previous intermediate feature map are spliced ​​together to form a multi-source fusion feature, and the formed multi-source fusion feature is input into the convolutional network to obtain optical flow deviation. The final predicted optical flow of the previous stage is updated based on the optical flow deviation to obtain the initial predicted optical flow of the current stage.

8. The method according to claim 6, characterized in that, The step of fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow and the current-level intermediate feature map includes: The current-level initial predicted optical flow and the prior optical flow are fused to output the current-level final predicted optical flow; Based on the final predicted optical flow of the current level, a warp operation is performed on the intermediate feature map of the previous level to obtain the intermediate feature map of the current level.

9. The method according to claim 8, characterized in that, The step of fusing the current-level initial predicted optical flow and the prior optical flow to output the current-level final predicted optical flow includes: Determine the weighting coefficients for the current-level initial predicted optical flow and the prior optical flow; Based on the weighting coefficients, the current-level initial predicted optical flow and the prior optical flow are fused to output the current-level final predicted optical flow.

10. The method according to claim 9, characterized in that, The determination of the weighting coefficients for the current-level initial predicted optical flow and the prior optical flow includes: The current-level initial predicted optical flow and the prior optical flow are respectively converted into expanded predicted optical flow and expanded prior optical flow in block representations containing spatial neighborhood information; Based on the unfolded predicted optical flow and the unfolded prior optical flow, respectively, a warp operation is performed on the reference feature map output by the feature extraction layer corresponding to the current level decoder to obtain the warped predicted feature map and the warped prior feature map. Calculate the first similarity of the pixel neighborhood within the distorted predicted feature map and the second similarity of the pixel neighborhood within the distorted prior feature map; The first similarity, the second similarity, the unfolded predicted optical flow, the unfolded a priori optical flow, and the reference feature map output by the feature extraction layer corresponding to the current level decoder are channel-stitched together to form a fusion tensor. The weighting coefficients of the prior optical flow and the current-level initial predicted optical flow are determined based on the fusion tensor.

11. The method according to claim 3, characterized in that, The multi-scale feature extraction of the reference frame group includes: The reference frame group is stitched together with the corresponding depth information to obtain the stitched video frame; The multi-scale feature extraction is performed on the spliced ​​video frames.

12. The method according to any one of claims 1-11, characterized in that, The prior information includes prior optical flow and occlusion mask; the reference frame group includes a first reference frame and a second reference frame; the prior optical flow includes a first prior optical flow from the interstitial frame to be inserted to the first reference frame and a second prior optical flow from the interstitial frame to be inserted to the second reference frame; the interstitial frame optical flow includes a first interstitial frame optical flow corresponding to the first prior optical flow and a second interstitial frame optical flow corresponding to the second prior optical flow. The step of generating the intermediate frame to be interpolated based on the predicted optical flow and the reference frame group includes: Based on the optical flow of the first intermediate frame, a forward warp operation is performed on the first reference frame to obtain a first warp feature map corresponding to the intermediate frame to be inserted. Based on the optical flow of the second intermediate frame, the second reference frame is subjected to forward warp processing to obtain a second warp feature map corresponding to the intermediate frame to be inserted. Based on the occlusion mask, the first warp feature map and the second warp feature map are fused to obtain the intermediate frame to be inserted.

13. The method according to any one of claims 1-11, characterized in that, The method further includes: The neural network model is trained, wherein the training process of the neural network model includes: At least one set of sample reference frames is obtained; each set of sample reference frames includes: a first sample reference frame, a second sample reference frame, and a sample intermediate frame located between the first sample reference frame and the second sample reference frame; The at least one set of sample reference frames and the corresponding sample motion vector information are subjected to forward warp processing to obtain the sample prior information of the sample intermediate frame; The at least one set of sample reference frames and the sample prior information are used as training data to iteratively train a preset initial neural network model to obtain the neural network model.

14. The method according to claim 13, characterized in that, The step of performing forward warp processing on the at least one set of sample reference frames and the corresponding sample motion vector information to obtain the sample prior information of the intermediate frames includes: Based on the motion vector information of the intermediate sample frame and the motion vector information of the second sample reference frame, the initial sample optical flow information of the intermediate sample frame is determined; The initial sample optical flow information is subjected to forward warp processing to obtain the sample prior information of the intermediate frame.

15. A video frame interpolation device, characterized in that, The device includes: The calculation module is configured to perform forward warp processing on the reference frame group and the corresponding motion vector information to obtain the prior information of the intermediate frame to be inserted; wherein the intermediate frame to be inserted is located between two reference frames contained in the reference frame group. The prediction module is configured to input the reference frame group and the prior information into a neural network model, so that the neural network model predicts the predicted optical flow of the intermediate frame to be inserted based on the reference frame group and the prior information, and generates the intermediate frame to be inserted based on the predicted optical flow and the reference frame group.

16. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video frame interpolation method according to any one of claims 1-14.

17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the video frame interpolation method according to any one of claims 1-14.

Images