A video frame interpolation method and system based on all-to-many field transformation

Abstract

The application relates to the technical field of video frame interpolation, and provides a video frame interpolation method and system based on all-to-many field conversion, which comprises the following steps: based on a pair of video frames, obtaining a bidirectional correlation container, bilateral optical flow, intermediate frame features and a plurality of levels of input frame feature pairs, and updating the bilateral optical flow and the intermediate frame features through a plurality of intermediate layers to obtain an output video frame; in each intermediate layer, the bilateral optical flow is first scaled and converted into bidirectional optical flow, and then the correlation features in the bidirectional correlation container are taken out according to the bidirectional optical flow, the residual error of the bidirectional optical flow and the residual error of the intermediate frame features are calculated, the bilateral optical flow and the intermediate frame features are updated according to the residual errors, the updated bilateral optical flow is used for performing a distortion operation on the input frame features, and the distorted input frame features and the intermediate frame features are input into a decoder to be enlarged in a spatial scale. The fidelity of the cross-scale optical flow is ensured, the accumulation and amplification of errors are reduced, and the video frame interpolation result is more accurate.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video frame interpolation, and particularly relates to a video frame interpolation method and system based on all-to-many field transformation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Video frame interpolation is a video processing technique that aims to synthesize intermediate frames with the help of reference frames to improve the temporal resolution of input videos. This technique has been applied to various downstream tasks, including slow-motion generation, novel view synthesis, video compression, text-to-video generation, etc. The advent of deep learning has promoted the development of video frame interpolation technology.

[0004] Video frame interpolation methods can be basically divided into three categories: kernel estimation-based methods, hallucination-based methods and optical flow-based methods. Kernel estimation-based methods capture inter-frame motion information by estimating dynamic kernel weights and offsets. With the help of the development of existing model architectures, hallucination-based methods directly generate interpolated frames from the features of input video frame pairs. Due to the robustness of optical flow, optical flow-based video frame interpolation methods dominate related research due to their effectiveness. Previous optical flow-based methods obtain estimated optical flow with the help of pre-trained optical flow models or jointly trained optical flow estimation modules. Some methods propose intermediate supervision to encourage the model to generate task-oriented optical flow in order to extract motion knowledge from pseudo-real optical flow. Subsequently, backward warping and forward warping became the standard scheme for optical flow estimation. Similar to UNet architecture, the architecture is a common model choice in the frame interpolation task, and the Transformer, as a general architecture, is introduced into the recent frame interpolation architecture to obtain better synthesis results. Recent work considers efficiency issues and gives up using additional synthesis networks in the frame interpolation framework. However, these methods still have defects in modeling large displacements and handling occlusions.

[0005] A common approach for optical flow based video interpolation methods is as follows: bilateral or bi-directional optical flow is estimated from a given pair of video frames, and then pixels or features are moved to the target time step by backward or forward warping. Therefore, the quality of the synthesized intermediate frame heavily depends on the result of the optical flow estimation. However, it is not directly convenient to approximate the intermediate optical flow (since the intermediate frame is not visible) by a pre-trained optical flow model, and these estimated optical flows are not suitable for applying to the video interpolation task. To alleviate this problem, the latest methods generate task-oriented optical flow by training end-to-end. However, some major and challenging issues in the interpolation task (such as large displacement and large occlusion cases) remain to be solved. These challenges mainly come from the defective estimation of the optical flow. Recent studies show that task-oriented optical flow is generally consistent with the true optical flow, but there are differences in local details.

[0006] The existing video interpolation methods have the following defects:

[0007] 1) The existing video interpolation methods predict optical flow that is not consistent with the true displacement at a coarse granularity, which is particularly serious when large displacement is encountered. Most existing methods use a structure similar to UNet to establish a video interpolation model using ordinary convolution. However, this type of structure is prone to accumulating errors at an early stage, and this structure is prone to accumulating errors when modeling large motion.

[0008] 2) The existing methods can only predict a pair of optical flows, so the solution set is limited to a narrow space. This makes it difficult for them to handle occlusions and details around the motion boundary, thereby deteriorating the interpolation results. SUMMARY

[0009] To solve the technical problems in the above background art, the present application provides a video interpolation method and system based on all-to-many field transformation, the retrieved relevant features help to jointly update the bilateral optical flow and the intermediate frame features in a cross-scale manner, ensuring the fidelity of the cross-scale optical flow, reducing the accumulation and amplification of errors, and facilitating the generation of more accurate video interpolation results.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] The first aspect of the present application provides a video interpolation method based on all-to-many field transformation, comprising:

[0012] Obtaining a pair of video frames;

[0013] Based on the obtained pair of video frames, obtaining a bi-directional relevance container, a bilateral optical flow, an intermediate frame feature, and a plurality of hierarchical input frame feature pairs, and updating the bilateral optical flow and the intermediate frame feature through a plurality of intermediate layers;

[0014] The updated bilateral optical flow and the intermediate frame feature pass through an output layer to obtain an output video frame.

[0015] In each intermediate layer, the bilateral optical flow is first converted into bidirectional optical flow by scaling, and the pyramid correlation feature is extracted from the bidirectional correlation container according to the bidirectional optical flow. Based on the correlation feature, the residual of the bidirectional optical flow and the residual of the intermediate frame feature are calculated. The bilateral optical flow and the intermediate frame feature are updated according to the residuals. The updated bilateral optical flow is used to warp the input frame feature. The warped input frame feature and the intermediate frame feature enter a decoder for spatial scale up to generate the bilateral optical flow and the intermediate frame feature of the next intermediate layer input.

[0016] Further, the bidirectional correlation container is indexed by the bidirectional optical flow using a scaling index method to obtain the pyramid correlation feature.

[0017] Further, the dimension of the transpose convolution used by the decoder of the output layer is greater than the dimension of the transpose convolution used by the decoder of the intermediate layer.

[0018] Further, the output of the decoder of the output layer includes multiple groups of bilateral optical flow, occlusion masks and residual images.

[0019] The bilateral optical flow of each group warps a pair of obtained video frames to obtain an interpolated image, and the occlusion mask and the residual image of the group are used when warping.

[0020] Further, the interpolated image is:

[0021]

[0022] wherein, represents a reverse warping operation, is the nth group of bilateral optical flow, is the nth group of occlusion mask, is the nth group of residual image, is a pair of obtained video frames.

[0023] Further, the interpolated images of all groups are fused by a plurality of convolution layers to obtain the output video frame.

[0024] Further, the calculation method of the bidirectional correlation container is:

[0025] Based on a pair of obtained video frames, a first frame feature and a second frame feature are obtained.

[0026] The first frame feature and the second frame feature are point multiplied and summed in the feature dimension to obtain a first one-way correlation container.

[0027] performing a plurality of continuous average pooling operations on the first one-way correlation container to obtain a one-way correlation pyramid;

[0028] performing a transposition operation on the one-way correlation pyramid to obtain an inverse correlation pyramid;

[0029] the one-way correlation pyramid and the inverse correlation pyramid form a two-way correlation container.

[0030] A second aspect of the present application provides a video interpolation system based on full-to-multiple field transformation, which comprises:

[0031] a data acquisition module configured to acquire a pair of video frames;

[0032] a feature update module configured to, based on the acquired pair of video frames, obtain a two-way correlation container, bilateral optical flow, intermediate frame features and a plurality of levels of input frame feature pairs, and update the bilateral optical flow and the intermediate frame features through a plurality of intermediate layers;

[0033] an interpolation module configured to, after the update, pass the updated bilateral optical flow and the intermediate frame features through an output layer to obtain an output video frame;

[0034] In each intermediate layer, the bilateral optical flow is first converted into two-way optical flow by scaling, and the pyramid-shaped correlation features are extracted from the two-way correlation container according to the two-way optical flow, the residual error of the two-way optical flow and the residual error of the intermediate frame features are calculated based on the correlation features, the bilateral optical flow and the intermediate frame features are updated according to the residual errors, the updated bilateral optical flow is used to perform a warping operation on the input frame features, the warped input frame features and the intermediate frame features are input into a decoder together to be enlarged in spatial scale, and the bilateral optical flow and the intermediate frame features of the next intermediate layer input are generated.

[0035] A third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a video interpolation method based on full-to-multiple field transformation as described above.

[0036] A fourth aspect of the present application provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of a video interpolation method based on full-to-multiple field transformation as described above when executing the program.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] The present application provides a video interpolation method based on full-to-multiple field transformation, which fully models the dense correspondence relationship between frames through a two-way 4D correlation container, and has strong robustness to large displacement.

[0039] The application provides a video interpolation method based on full-to-multi-field transformation, which introduces a scaling lookup strategy to solve the coordinate mismatch problem caused by invisible frames.

[0040] The application provides a video interpolation method based on full-to-multi-field transformation, which searches for relevant features to help the model jointly update bilateral optical flow and intermediate frame features in a cross-scale manner; thus ensuring the basic fidelity of cross-scale optical flow and reducing the accumulation and amplification of errors.

[0041] The application provides a video interpolation method based on full-to-multi-field transformation, which derives multiple sets of fine-grained bilateral optical flow from a pair of updated coarse-grained bilateral optical flow, and additionally predicts an occlusion mask and a residual image for each set of optical flow, thereby alleviating the interpolation ambiguity problem in the occlusion area.

[0042] The application provides a video interpolation method based on full-to-multi-field transformation, which is more efficient in training and deployment, and is also conducive to generating more accurate video interpolation results. BRIEF DESCRIPTION OF DRAWINGS

[0043] The drawings accompanying the specification of the application form a part of the application and serve to further understand the application. The illustrative embodiments of the application and their descriptions serve to explain the application, and do not constitute an improper limitation of the application.

[0044] Figure 1 FIG. 1 is a flowchart of a video interpolation method based on full-to-multi-field transformation according to an embodiment of the application;

[0045] FIG. 2(a) is an overlapping frame diagram according to an embodiment of the application;

[0046] FIG. 2(b) is a real frame diagram according to an embodiment of the application;

[0047] FIG. 2(c) is an optical flow diagram of IFRNet according to an embodiment of the application;

[0048] FIG. 2(d) is an optical flow diagram of a video interpolation method based on full-to-multi-field transformation according to an embodiment of the application;

[0049] FIG. 2(e) is a result diagram of IFRNet according to an embodiment of the application;

[0050] FIG. 2(f) is a result diagram of a video interpolation method based on full-to-multi-field transformation according to an embodiment of the application;

[0051] Figure 3 FIG. 3 is a comparison diagram of results according to an embodiment of the application. DETAILED DESCRIPTION

[0052] The application will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0054] Embodiment I

[0055] The embodiment provides a video frame interpolation method based on full-to-many field transformation, as shown in the following steps. Figure 1

[0056] Step 1, obtaining a pair of video frames (image pair) and sending them into a content encoder for encoding, and the output of the encoding is an initial bilateral optical flow estimation (i.e. the first level bilateral optical flow) and the first level intermediate frame feature and L a pair of input frame features of the first level .

[0057] The pair of video frames includes a first video frame I 0 and a second video frame I 1. t t represents the time of the frame to be interpolated, the first video frame I 0 is collected before the time t, and the second video frame t 1 is collected after the time t. I t The value of n is 3, and the pair of input frame features of the three levels includes the first level input frame feature pair , the second level input frame feature pair

[0058] , and the third level input frame feature pair L .

[0059] Step 2, sending the input pair of video frames into a correlation encoder to obtain a bidirectional 4D correlation container.

[0060] First, the input pair of video frames is sent into a correlation encoder to obtain a first frame feature and a second frame feature at an eighth of the original resolution scale, and according to the obtained first frame feature and the second frame feature , a first one-way 4D correlation container is calculated.

[0061] The calculation method of the first one-way 4D correlation container may be represented as:​​​​

[0062]

[0063] wherein, , , , , V represents the length of the video frame, W represents the width of the video frame; , H represents the dimension of the feature; is the first video frame at position of the first video frame; dimensional feature at position of the second video frame; dimensional feature at position of the second video frame; dimension of ; is the correlation of the first video frame at position for the second video frame at position .

[0064] Secondly, the is subjected to 3 repetitive average pooling operations with kernel size of 2 and step size of 2, to obtain a second one-way 4D correlation container , a third one-way 4D correlation container and a fourth one-way 4D correlation container , thus obtaining a one-way 4D correlation pyramid with level of 4 , with different dimensions.

[0065] Finally, the correlation encoder is used to construct a two-way 4D correlation container, which can be obtained from the one-way 4D correlation container by transposition operation , specifically: for each level of the one-way 4D correlation container , another direction of 4D correlation container can be obtained by transposition operation, thus obtaining the inverse correlation pyramid .

[0066] wherein, the network structure of the correlation encoder and the content encoder is not limited, but the output resolution is lower than the input resolution. In the embodiment, the correlation encoder and the content encoder are both composed of several convolutions and nonlinear operations, the difference lies in the use at different positions of the network, thus providing different information to the network.

[0067] Step 3: Based on the bidirectional correlation container, the first-level bilateral optical flow, the first-level intermediate frame features, and several levels of input frame feature pairs, the bilateral optical flow and intermediate frame features are updated through several intermediate layers; the updated bilateral optical flow and intermediate frame features are then passed through the output layer to obtain the output video frame.

[0068] Step 301: Index (lookup operation) the bidirectional 4D correlation container established in Step 2 using bilateral optical flow estimation. Specifically:

[0069] (1) In order to index the bidirectional 4D correlation container, first, the first... Hierarchical bilateral optical flow is converted to the first level by scaling. Hierarchical bidirectional optical flow, namely:

[0070]

[0071] in, Indicates the first Hierarchical bilateral optical flow, For the first Hierarchical bidirectional optical flow The moment when the frame is to be interpolated.

[0072] (2) After obtaining the bidirectional optical flow, according to the first The coordinates of the hierarchical bidirectional optical flow are used to find the position of the corresponding bidirectional 4D correlation container, and a fixed range is defined at the corresponding position. Extract the pyramid-shaped correlation features from within.

[0073] Specifically, in the first video frame The optical flow at the location is That is, the first l Hierarchical bidirectional optical flow of The value of the position Then there is a new pixel position. The relevance container is used as an index to navigate forward. Extract the pyramid-shaped forward correlation features. Specifically, taking the first-level correlation container as an example, the neighborhood range can be defined as follows. , where r is the neighborhood radius. Based on this neighborhood range... exist Then, the relevance features of the first level can be extracted. This indexing operation is performed in all levels of the pyramid-shaped relevance container, and the neighborhood range can therefore be represented as... , where k is the level of the relevance container; similarly, it can be based on the second video frame. The optical flow at location, from the inverse correlation container The pyramid-shaped inverse correlation features are extracted. The extracted forward and inverse correlation features can be collectively referred to as correlation features.

[0074] wherein only the first search uses the initial optical flow (V0) and the subsequent level searches use the updated optical flow (V1) of the previous level .

[0075] The optical flow is used to index the bidirectional 4D correlation container in a scaling manner, i.e., the bilateral optical flow displacement size is changed; each time the decoder passes, the optical flow output by the decoder is indexed once.

[0076] Step 302, using the correlation features indexed in step 3 and the intermediate frame features and bilateral optical flow of the corresponding level as input, the correlation feature processing module obtains the residual of the optical flow and the residual of the intermediate frame features.

[0077] Figure 1 The first, second and third level correlation feature processing modules are represented by V1, V2 and V3 respectively.

[0078] The correlation feature processing module uses multiple convolution layers as the skeleton; the correlation feature processing module finally has two output heads, which output the residual of the optical flow and the residual of the intermediate frame features (V1, V2 and V3).

[0079] Step 5, determine whether the level number reaches the maximum value, if yes, execute step 502; otherwise, execute step 501.

[0080] Step 501, update the bilateral optical flow estimation of step 1 and the intermediate frame features of the current level according to the optical flow residual and the intermediate frame feature residual (i.e., feature addition ); the updated bilateral optical flow estimation is used to perform a warping operation on the input frame features of the corresponding level, and the warped input frame features are sent to the decoder of the intermediate layer together with the intermediate frame features D i to perform spatial scale expansion to generate the optical flow estimation and the intermediate frame features of the next level .

[0081] Each scaling operation is performed once, and each scaling operation is performed once. Each time, the updated optical flow of the previous level is used instead of the initial optical flow.

[0082] The indexed correlation features need to update the bilateral optical flow and the intermediate frame features of the current level.

[0083] Step 502, the output layer decoder uses a higher-dimensional transpose convolution to output multiple groups of results, and the output is multiple groups of bilateral optical flow , and the occlusion mask​ ) and residual images (R ). Each bilateral optical flow is used to warp the input video frame pair to generate the final output video frame. Each interpolated frame is denoted as I The processing procedure can be represented as:

[0084]

[0085] wherein, denotes the inverse warping operation. The above equation is performed N times, where N denotes the number of predicted bilateral optical flow groups. The N interpolated frames are fused by two convolutional layers to generate the final interpolated result.

[0086] Coarse-to-fine bilateral optical flow estimation and intermediate frame feature extraction. This method can simultaneously perform bilateral optical flow estimation and intermediate frame feature extraction.

[0087] This method can be scaled to different network sizes to meet the needs of different application scenarios.

[0088] Using this method, efficient video interpolation training and deployment can be performed, and accurate and robust video interpolation results can be quickly obtained. This method can better handle large displacement and occlusion interpolation scenarios. This video interpolation method can be applied in scenarios including but not limited to slow motion generation, video frame rate improvement, new view synthesis, video compression, and text-to-video generation scenarios.

[0089] The present application contains two important designs. The first design is the bidirectional full correlation calculation (i.e., bidirectional 4D correlation container), which fully models the dense correspondence between frames and has strong robustness for large displacement. The present application introduces a scaling search strategy to solve the coordinate mismatch problem caused by invisible frames. In addition, the retrieved correlation features help the model to jointly update the bilateral optical flow and the intermediate frame features in a cross-scale manner. Therefore, the basic fidelity of the cross-scale optical flow is guaranteed, and the accumulation and amplification of errors are reduced. The second design of the present application is to derive multiple groups of fine-grained bilateral optical flow from a pair of updated coarse-grained bilateral optical flow, and additionally predict an occlusion mask and a residual image for each group of optical flow. This design alleviates the interpolation ambiguity problem in the occlusion area. The combination of these two designs makes the training and deployment process more efficient, and is also conducive to generating more accurate video interpolation results.

[0090] The application realizes more accurate and efficient video frame interpolation by designing a neural network based on full-to-many field transformation. The method of the application first sends a pair of input video frames into a content encoder for encoding, and the output after encoding is an initial bilateral optical flow estimation and an intermediate frame feature; at the same time, a pair of input video frames are sent into a correlation encoder in parallel to calculate a bidirectional 4D correlation container. Then, different levels of optical flow are used to index the bidirectional 4D correlation container by scaling to obtain different levels of correlation features. These correlation features are used to synchronously update the corresponding level of optical flow and the intermediate frame feature; finally, a pair of updated coarse-grained bilateral optical flows pass through the last decoder to obtain a plurality of groups of fine-grained optical flows, occlusion masks and residual images, and the input video pair is obtained by reverse warping to obtain the final output video frame. The method can be scaled to different network sizes to meet the needs of different application scenarios. In addition, the method can be applied to slow motion generation, video frame rate improvement, new view synthesis, video compression and text-to-video generation scenarios.

[0091] To demonstrate the effectiveness of the design of the present application, the current advanced methods are compared with the method of the present application using Vimeo90K dataset on different model scales. Among them, the advanced methods include ToFlow (Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. IJCV, 2019), DAIN (Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In CVPR, 2019), CAIN (Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention is all you need for video frame interpolation. In AAAI, 2020), AdaCoF (Hyeongmin Lee, Taeoh Kim, Tae young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof: Adaptive collaboration of flows for video frame interpolation. In CVPR, 2020), BMBC (Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. Bmbc: Bilateral motion estimation with bilateral cost volume for video interpolation. In ECCV, 2020), RIFE (Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation.In ECCV, 2022), ABME (Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation. In CVPR, 2021), M2M-VFI (Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko. Many-to-many splatting for efficient video frame interpolation. In CVPR, 2022), and IFRNet. The metrics for comparison include: accuracy metrics: peak signal-to-noise ratio (PSNR) and structure similarity (SSIM); efficiency metrics: speed, parameter amount, and computational amount. The inference speed and computational complexity are calculated by RTX 3090 GPU at 1280x720 resolution. The results are shown in Table 1 (small model) and Table 2 (large model), and the method of the present application is more accurate and efficient than the current advanced method. Specifically, the small model version of the method of the present application is about 0.17 dB higher than the previous most advanced method, and its parameters and FLOPs are only about 60% of the previous most advanced method. While the large model version of the method of the present application is about 0.15 dB higher than the previous most advanced method, and only needs 65% of the parameters and 75% of the FLOPs of the previous most advanced method.

[0092] Table 1, Comparison results of small model and other methods

[0093]

[0094] Table 2, Comparison results of large model and other methods

[0095]

[0096] As shown in FIG. 2(a), FIG. 2(b), FIG. 2(c), FIG. 2(d), FIG. 2(e), and FIG. 2(f), in the video frame interpolation task, the previous video frame interpolation method cannot generate sharp edges of moving objects when the motion is complex. The method of the present application can accurately synthesize the content of the motion boundary, and the generated texture structure has fewer artifacts. And generate reasonable texture with less artifacts. When the background object is severely occluded by the foreground, the method of the present application can still obtain content guidance from the reference frame in other directions, while other methods cannot synthesize these occluded objects.

[0097] AsFigure 3 As shown in the middle results, the method of the present application ensures the global consistency of the intermediate optical flow and can accurately synthesize the fast-moving object covering the occluded area, while the previous most advanced method IFRNet cannot ensure the accurate estimation of the optical flow and thus cannot ensure the accurate generation of the object.

[0098] Embodiment Two

[0099] The embodiment provides a video interpolation system based on full-to-multiple field transformation, which specifically comprises:

[0100] The data acquisition module is configured to acquire a pair of video frames;

[0101] The feature updating module is configured to obtain a bidirectional correlation container, a bilateral optical flow, an intermediate frame feature and a plurality of levels of input frame feature pairs based on the acquired pair of video frames, and update the bilateral optical flow and the intermediate frame feature through a plurality of intermediate layers;

[0102] The interpolation module is configured to pass the updated bilateral optical flow and intermediate frame feature through an output layer to obtain an output video frame;

[0103] In each intermediate layer, the bilateral optical flow is first scaled and converted into a bidirectional optical flow, and the pyramid-shaped correlation features in the bidirectional correlation container are taken out according to the bidirectional optical flow, the residual error of the bidirectional optical flow and the residual error of the intermediate frame feature are calculated based on the correlation features, the bilateral optical flow and the intermediate frame feature are updated according to the residual errors, the updated bilateral optical flow is used to perform a warping operation on the input frame feature, and the warped input frame feature and the intermediate frame feature are input into a decoder to be enlarged in spatial scale to generate the bilateral optical flow and the intermediate frame feature of the next intermediate layer input.

[0104] It should be noted that each module in the embodiment corresponds to each step in Embodiment One, and the specific implementation process is the same, which will not be repeated here.

[0105] Embodiment Three

[0106] The embodiment provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the video interpolation method based on full-to-multiple field transformation according to Embodiment One.

[0107] Embodiment Four

[0108] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the steps in the video interpolation method based on full-to-multiple field transformation according to Embodiment One when executing the program.

[0109] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage and so forth) embodying computer-readable program code.

[0110] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the

[0111] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for carrying out each of the

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0114] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video frame interpolation method based on full-pair multi-field transform, characterized in that, include: Get a pair of video frames; Based on the acquired pair of video frames, a bidirectional correlation container, bilateral optical flow, intermediate frame features, and several levels of input frame feature pairs are obtained. After passing through several intermediate layers, the bilateral optical flow and intermediate frame features are updated. The calculation method for the bidirectional correlation container is as follows: Based on the acquired pair of video frames, the features of the first frame and the features of the second frame are obtained; The first frame features and the second frame features are multiplied by a dot product along the feature dimension and summed to obtain the first unidirectional correlation container. Perform several consecutive average pooling operations on the first unidirectional correlation container to obtain the unidirectional correlation pyramid. Transpose the one-way correlation pyramid to obtain the inverse correlation pyramid. The unidirectional correlation pyramid and the inverse correlation pyramid together form a bidirectional correlation container; The updated bilateral optical flow and intermediate frame features are passed through the output layer to obtain the output video frame; In each intermediate layer, the bilateral optical flow is first scaled and converted into a bidirectional optical flow. Based on the bidirectional optical flow, the pyramid-shaped correlation features are extracted from the bidirectional correlation container. Based on the correlation features, the residuals of the bidirectional optical flow and the intermediate frame features are calculated. The bilateral optical flow and intermediate frame features are updated based on the residuals. The updated bilateral optical flow estimates distort the input frame features. The distorted input frame features and intermediate frame features are entered into the decoder together for spatial scale magnification to generate the bilateral optical flow and intermediate frame features for the next intermediate layer input.

2. The video frame interpolation method based on full-pair multi-field transform as described in claim 1, characterized in that, A scaling indexing method is used to index the bidirectional correlation container using bidirectional optical flow, resulting in pyramid-shaped correlation features.

3. The video frame interpolation method based on full-pair multi-field transform as described in claim 1, characterized in that, The output layer decoder uses a transposed convolution with a larger dimension than the intermediate layer decoder.

4. The video frame interpolation method based on full-pair multi-field transform as described in claim 1, characterized in that, The output of the decoder in the output layer includes multiple sets of bilateral optical flows, occlusion masks, and residual images; Each pair of bilateral optical flows performs a warping operation on the acquired pair of video frames to obtain an interpolated image, and the occlusion mask and residual image of that pair are used when performing the warping operation.

5. A video frame interpolation method based on full-pair multi-field transform as described in claim 4, characterized in that, The interpolated image is: in, Indicates the reverse twist operation. For the nth group of bilateral optical flow, For the nth occlusion mask, For the nth group of residual images, For acquiring a pair of video frames.

6. The video frame interpolation method based on full-pair multi-field transform as described in claim 1, characterized in that, The interpolated images from all groups are fused together through several convolutional layers to obtain the output video frame.

7. A video frame interpolation system based on full-pair multi-field transform, characterized in that, include: The data acquisition module is configured to acquire a pair of video frames; The feature update module is configured to: based on a pair of acquired video frames, obtain a bidirectional correlation container, bilateral optical flow, intermediate frame features, and several levels of input frame feature pairs, and update the bilateral optical flow and intermediate frame features through several intermediate layers; The calculation method for the bidirectional correlation container is as follows: Based on the acquired pair of video frames, the features of the first frame and the features of the second frame are obtained; The first frame features and the second frame features are multiplied by a dot product along the feature dimension and summed to obtain the first unidirectional correlation container. Perform several consecutive average pooling operations on the first unidirectional correlation container to obtain the unidirectional correlation pyramid. Transpose the one-way correlation pyramid to obtain the inverse correlation pyramid. The unidirectional correlation pyramid and the inverse correlation pyramid together form a bidirectional correlation container; The frame interpolation module is configured to: pass the updated bilateral optical flow and intermediate frame features through the output layer to obtain the output video frame; In each intermediate layer, the bilateral optical flow is first scaled and converted into a bidirectional optical flow. Based on the bidirectional optical flow, the pyramid-shaped correlation features are extracted from the bidirectional correlation container. Based on the correlation features, the residuals of the bidirectional optical flow and the intermediate frame features are calculated. The bilateral optical flow and intermediate frame features are updated based on the residuals. The updated bilateral optical flow estimates distort the input frame features. The distorted input frame features and intermediate frame features are entered into the decoder together for spatial scale magnification to generate the bilateral optical flow and intermediate frame features for the next intermediate layer input.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the video frame interpolation method based on full-pair multi-field transformation as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the video frame interpolation method based on full-pair multi-field transformation as described in any one of claims 1-6.

Images