A nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network

Abstract

The method discloses a nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network, high-level semantic features of multiple continuous video frames are extracted in a coarse frame synthesis stage, and coding features are fused, nonlinear motion features between the video frames are adaptively obtained through interactive operation of the features in a double-branch up-sampling module and layer-by-layer refinement operation of a single optical flow, more accurate motion estimation and a coarse frame are obtained, then in a fine frame synthesis stage, a residual error is generated through a refinement network to compensate for the coarse frame, and a refined intermediate frame is obtained. The application solves the problem of video frame interpolation of high-order nonlinear motion scenes, extracts high-level semantic information of multiple video frames, adaptively establishes a motion model, effectively solves the problem that it is difficult to accurately describe with a fixed mathematical model in the prior art, and through the help of deep learning technology and the optical flow strategy, complex motion scenes can be better processed, so that the synthesized intermediate frame image is more real in vision.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology and relates to a nonlinear motion estimation video interpolation method based on a multi-branch symmetric network. Background Technology

[0002] Video frame interpolation is an important research area in video image coding and computer vision. It explores the problem of inserting extra frames into video sequences to improve image quality or smooth animation, and has many practical applications in multimedia services, such as video frame rate conversion, video compression, and the synthesis of virtual viewpoint images.

[0003] With the continuous advancements in deep learning and optical flow networks, optical flow-based video interpolation methods have achieved significant breakthroughs with the help of deep learning. This is because optical flow technology can provide pixel-level motion estimation of objects or scenes in a video sequence, thus providing more accurate dynamic information for video interpolation. Flow-based methods typically follow three steps: 1) Estimating the optical flow between the target frame and the input frame. 2) Twisting the input frame or contextual features according to the optical flow. 3) Refining the distorted frame or features and generating the target frame through a synthesis network.

[0004] Most current video frame interpolation algorithms use two consecutive frames as input to generate an intermediate frame, primarily capturing motion information by establishing a linear model. However, dynamic scenes often involve high-order nonlinear motion, posing a significant challenge to frame interpolation techniques. To capture motion information in complex dynamic scenes, some solutions employ quadratic motion models to approximate the actual scene's motion, which is effective for low-order motion scenes. However, limited by fixed motion models, when motion is discontinuous or irregular over time, it is difficult to accurately describe it with a fixed mathematical model, resulting in severe distortion of the video interpolation results. To overcome these problems, further improvements in technical performance are needed. Although deep learning-based techniques have made some progress, further research is required to address these challenges. Summary of the Invention

[0005] To address the aforementioned technical problems in the existing technology, this invention proposes a nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network, the specific technical solution of which is as follows:

[0006] A nonlinear motion estimation video interpolation method based on multi-branch symmetric networks:

[0007] Acquire consecutive frame images;

[0008] Four frames with the same interval in a continuous frame image are preprocessed and then input into a coarse frame synthesis network for feature extraction, fusion, dual-branch upsampling and frame synthesis operations to obtain a coarse frame image.

[0009] The coarse frame image is input into the thinning network for residual information compensation to obtain the intermediate frame;

[0010] At the same time, a guiding network is used for joint iterative training to obtain more realistic intermediate frames.

[0011] Furthermore, the preprocessing of the consecutive frame images includes: cropping four frames with the same interval to ensure consistent scale, and then performing data enhancement by randomly flipping, rotating, swapping color channels, and reversing the sequence order of the images.

[0012] Furthermore, the coarse frame synthesis network includes an encoder, an encoding feature fusion module, a dual-branch upsampling module, and a decoding feature fusion module. The encoder extracts features from the input frame image. The encoding feature fusion module arranges the extracted features symmetrically and performs a fusion operation to obtain two sets of fused features. The dual-branch upsampling module performs input feature interaction operations and layer-by-layer refinement operations on the two sets of fused features to generate intermediate optical flow and context features in two directions. The decoding feature fusion module performs convolution operations on the intermediate optical flow and context features to obtain optical flow in two directions and a spatial sampling mask. Then, it combines these with a reference frame to perform frame synthesis to obtain a coarse frame. The four frames are I, ... -1 Frames I0, I1, and I2 are used as reference frames, and the fourth frame is used as the target frame I at time t. gt , 0≤t≤1.

[0013] Furthermore, the encoder consists of four coding layers of a pyramid network with the same core, each encoding one of the four input frames I. -1 Feature extraction is performed using I0, I1, and I2. Each pyramid network has four downsampling layers used to extract features and continuously halve the feature size. Each layer consists of two 3×3 convolutions with strides of 2 and 1, respectively. Each convolutional layer is followed by a PReLU activation function.

[0014] Furthermore, the coding feature fusion module specifically fuses the fourth-layer coding features of the four frames to obtain two sets of fused features, namely... and in Indicate I p The i-th layer coding features of the frame, i = 1, 2, 3, 4, p = -1, 0, 1, 2.

[0015] Furthermore, the dual-branch upsampling module consists of two parallel and parameter-sharing upsampling networks forming upper and lower branches, and the upsampling network on each branch has four unidirectional optical flow generation modules with the same structure.

[0016] Furthermore, the unidirectional optical flow generation module SFB (i)The input features (i = 4, 3, 2, 1) consist of one convolutional layer, one residual block, and one deconvolutional layer, outputting the intermediate optical flow and contextual features in a single direction. First, the features input to this module undergo convolution in the first convolutional layer to fuse and extract key information. Next, the residual block performs adaptive motion compensation on the convolutional features to obtain more accurate motion information. The residual block contains two skip connections and three convolutional layers, each followed by a PReLU activation function. The first skip connection adds the input of the residual block to the output of the second convolutional layer, and the second skip connection adds the input of the residual block to the output of the third convolutional layer. Each skip connection is followed by a PReLU activation function before outputting. Finally, the deconvolutional layer performs upsampling convolution to obtain the intermediate optical flow of that layer. and context features u = 0, 1, i = 4, 3, 2, 1; the first unidirectional optical flow generation module SFB (4) The input feature is the fused feature, while the inputs of the other three modules are the spliced ​​features obtained through interactive operations of the input features.

[0017] Furthermore, the input feature interaction operation involves using the reference features of the upper and lower branches as common features. During the upsampling process, the two branches will call each other's reference features. Specifically, the reference features are... and Symmetrically concatenate the optical flow and context features from the two branches respectively; the concatenation method is as follows: and Reference features It is composed of coding features By skipping connections, the intermediate optical flows of corresponding sizes in the two branches are respectively connected. The image obtained after performing a backward image warping operation is expressed as follows:

[0018]

[0019] Furthermore, the layer-by-layer refinement operation refers to the operation of adding the current intermediate optical flow in each branch to the intermediate optical flow of the previous layer after performing double bilinear interpolation, resulting in a refined intermediate optical flow. This serves as the input feature for the next unidirectional optical flow output module, and so on, repeating the operation until the last unidirectional optical flow generation module. The specific expression is as follows:

[0020]

[0021] Furthermore, the frame synthesis operation includes a backward warping operation and a spatial sampling operation. Specifically, it uses two reference frames I0 and I1 to perform a backward warping operation with the optical flow in two directions to obtain a warped frame. and Then, spatial sampling is performed using a spatial sampling mask to obtain a coarse frame. The spatial sampling operation is the process of combining two distorted frames into one frame according to the weights assigned by the spatial sampling mask.

[0022]

[0023]

[0024] Furthermore, the refinement network is composed of a Unet network, which combines two reference frames I0 and I1, and a distorted frame. and optical flow and The spatial sampling mask M is fed into the Unet network to generate three-channel residual features R. The residuals are added to the coarse frame to generate the intermediate frame. The specific expression is as follows:

[0025]

[0026] Furthermore, the guiding network consists of two parameterized Unet networks, with the same structure as the thinning network. The two Unet networks generate guiding optical flows in two directions respectively, and then the guiding intermediate frame is obtained by using backward warping and spatial sampling operations. Specifically, the optical flows generated in the coarse frame synthesis stage are used to obtain the guiding intermediate frame. Reference frame I0 and target frame I gt After being assembled in sequence, the optical stream is fed into one of the Unets to generate a guiding optical flow in the t→0 direction. In the same way, the optical flow generated in the coarse frame synthesis stage Reference frame I1 and target frame I gt After being assembled sequentially, the optical stream is fed into another Unet to generate a guiding optical flow in the t→1 direction. As in the formula:

[0027]

[0028] The guide intermediate frame is synthesized using backward warping and spatial sampling operations. As in the formula:

[0029]

[0030]

[0031] The guiding optical flow is represented by a two-channel feature map, where each pixel is represented by a two-dimensional coordinate. in, It is the guided optical flow in the t→u direction generated by the network, x u and y u They represent in Horizontal and vertical component feature maps.

[0032] Furthermore, the total loss function used in the joint iterative training is L. total =λ n L n +λ t L t ; where L n For nonlinear reconstruction loss, L t To guide the loss, λ n and λ t These are the nonlinear reconstruction loss coefficient and the guiding loss coefficient, respectively.

[0033] The nonlinear reconstruction loss includes: (βW+1) represents the overall weight, and β is the coefficient of the additional weight graph. It is a three-channel additional weight map, where U represents the difference binarization operation between the two parts of the input, used to mark the parts with large errors in each color channel between the synthesized frame and the target frame. and Pixel values ​​with a difference greater than 0 are set to 1, otherwise to 0. Then, the weight map of a single channel is adjusted. Sampling is performed to obtain a three-channel additional weight map W that matches each color channel, where α represents the tolerance error factor; single-channel weight map The nonlinear motion estimation is obtained from the guiding optical flow generated by the guiding network.

[0034] Guided loss:

[0035] Beneficial effects: This invention solves the problem of video frame interpolation in high-order nonlinear motion scenes by using a deep convolutional neural network to capture motion information between multiple consecutive input frames; it extracts high-level semantic information from multiple video frames and adaptively builds a motion model, effectively solving the problem that is difficult to accurately describe with a fixed mathematical model in the prior art; by using deep learning technology and adopting an optical flow strategy, it can better handle complex motion scenes, making the synthesized intermediate frame images more visually realistic. Attached Figure Description

[0036] Figure 1 This is a flowchart of the overall process of a nonlinear motion estimation video interpolation method based on a multi-branch symmetric network according to the present invention.

[0037] Figure 2 This is a flowchart of the coarse frame synthesis network of the present invention;

[0038] Figure 3 This is a flowchart illustrating the parallel unidirectional optical flow generation module and residual block of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0040] like Figure 1 As shown, the present invention provides a nonlinear motion estimation video interpolation method based on a multi-branch symmetric network. First, in the coarse frame synthesis stage, high-level semantic features of multiple consecutive video frames are extracted and coded features are fused. Then, through interactive operations of features in the dual-branch upsampling module and layer-by-layer refinement operations of single optical flow, nonlinear motion features between video frames are adaptively obtained, resulting in more accurate motion estimation and coarse frames. Finally, in the fine frame synthesis stage, residuals generated by a refinement network are used to compensate for the coarse frames, resulting in refined intermediate frames. Specifically, as follows:

[0041] (1) Data preprocessing.

[0042] (1-1) Extracting seven consecutive frames from a fixed frame rate video: This embodiment uses a seven-tuple dataset from the Vimeo-90K dataset extracted from a 30FPS video. The 1st, 3rd, 5th, and 7th video frames are selected from the seven-tuple as input images at times -1, 0, 1, and 2, respectively. -1 Frames I0, I1, and I2 are used as reference frames, and the fourth frame is used as the target frame I at time t. gt 0≤t≤1, take t=0.5. Use 4 frames of images with the same interval to ensure that a nonlinear motion model can be constructed.

[0043] (1-2) Crop each image to make each image have the same size N×N, 128≤N≤512, take N=224, which helps to maintain the consistency of the data.

[0044] (1-3) Data augmentation is achieved by randomly flipping, rotating, swapping color channels, and reversing the sequence order of images to increase the input data for the model. Among these, reversing the sequence order enhances the model's inductive ability. Specifically, it involves changing the I... -1 The input order of frames I0, I1, and I2 becomes I2, I1, I0, and I2. -1 The target frame remains unchanged while the first frame is inserted, achieving a consistent structure. This data preprocessing method better meets the needs of video frame interpolation technology and improves the model's effectiveness in practical tasks.

[0045] The performance of the proposed network was evaluated using the Vimeo-90K seven-tuple test set, 100 five-tuples generated by UCF101, and 2847 five-tuples generated by DAVIS.

[0046] (2) Construct a coarse frame synthesis network, which includes an encoder, an encoding feature fusion module, a double-branchup-sampling module, and a decoding feature fusion module.

[0047] (2-1) The encoder is located at the encoding end, such as Figure 2 As shown, the coding layer consists of four identical PyramidBlock networks, each encoding one of the four input frames I. -1 Feature extraction is performed using I0, I1, and I2. Each pyramid network has four downsampling layers used to extract features and continuously halve the feature size. Each layer consists of two 3×3 convolutions with strides of 2 and 1, respectively. Each convolutional layer is followed by a PReLU activation function. Each input frame generates a set of encoded features after encoding. in Indicate I p The i-th layer coding features of the frame, i = 1, 2, 3, 4, p = -1, 0, 1, 2.

[0048] (2-2) The coding feature fusion module is located after the encoder. To adaptively obtain nonlinear information, this module fuses the fourth-layer coding features of the four frames and sends them to the two decoding ends respectively. Depending on the different tasks of the two branches of the decoding end, the coding feature fusion module performs a symmetrical arrangement and concatenation fusion operation on the fourth-layer coding features encoded from the four input frames, resulting in two sets of fused features, namely… and This is adapted to the input of a decoder with two branches, namely a two-branch upsampling module.

[0049] The splicing and fusion operation involves arranging multiple features in a specified order and merging them into a single feature along the channel dimension.

[0050] (2-3) The dual-branch upsampling module is located at the decoding end. The decoding end performs an upsampling process. The dual-branch upsampling module consists of two parallel upsampling networks forming upper and lower branches. Each branch's upsampling network has four identical unidirectional optical flow generation modules (SFBs). The fusion features... and The inputs are fed into the upper and lower branches respectively, and then the intermediate optical flow and context features are generated and the intermediate optical flow is refined through input feature interaction operations and layer-by-layer refinement operations.

[0051] The features on the upper and lower branches represent motion estimations in the directions I0 and I1 from the target frame to the reference frame, respectively. The intermediate optical flow and context features on the upper branch are respectively... and The intermediate optical flow and context features on the lower branch are respectively and The superscript i represents the output feature of the i-th unidirectional optical flow generation module, and the features with subscripts t→0 and t→1 represent the features in the upper and lower branches, respectively.

[0052] The aforementioned input feature interaction operation refers to the mutual reuse of reference features between the two branches during the upsampling decoding process. Specifically, reference features belonging to their respective branches are used as common features. During upsampling, the upper branch calls the reference features of the lower branch, and similarly, the lower branch calls the reference features of the upper branch. For example, the reference features... and Symmetrically concatenate the optical flow and context features from the two branches respectively; the concatenation method is as follows: and The common information in the input information is the reference feature. and For decoding layers with the same parameters, controlling the position of input features generates corresponding optical flow information, enhancing the network's inductive ability and feature utilization efficiency. Among these, reference features... It is composed of coding features By skipping connections, the intermediate optical flows of corresponding sizes in the two branches are respectively connected. The result obtained after performing a backward image warping operation.

[0053] like Figure 3 As shown, the unidirectional optical flow generation module SFB (i)The input features (i = 4, 3, 2, 1) consist of one convolutional layer, one residual block, and one deconvolutional layer, outputting the intermediate optical flow and contextual features in a single direction. First, the features input to this module undergo convolution in the first convolutional layer to fuse and extract key information. Next, the residual block performs adaptive motion compensation on the convolutional features to obtain more accurate motion information. The residual block contains two skip connections and three convolutional layers, each followed by a PReLU activation function. The first skip connection adds the input of the residual block to the output of the second convolutional layer, and the second skip connection adds the input of the residual block to the output of the third convolutional layer. Each skip connection is followed by a PReLU activation function before outputting. Finally, the deconvolutional layer performs upsampling convolution to obtain the intermediate optical flow of that layer. and context features u = 0, 1, i = 4, 3, 2, 1. Except for the first unidirectional optical flow generation module (SFB). (4) The input features are fused features, while the inputs to other modules are concatenated features obtained through interactive operations on the input features. The specific expression is as follows:

[0054]

[0055]

[0056] The layer-by-layer refinement operation refers to the operation of adding the current intermediate optical flow in each branch to the intermediate optical flow of the previous layer after performing bilinear interpolation by a factor of 2, resulting in a refined intermediate optical flow. This process, using the input features of the next unidirectional optical flow output module, is repeated until the last unidirectional optical flow generation module. The details of the intermediate optical flow at each layer are continuously compensated to achieve accurate optical flow estimation. The specific expression is as follows:

[0057]

[0058] The backward image warping operation calculates the coordinates in the original image for each pixel of the generated image or feature based on optical flow information. The intensity value originates from these coordinates. The reference frame or its encoded features are then back-warped based on optical flow to obtain the warped frame or reference feature, as shown in the following expression:

[0059]

[0060] (2-4) The decoding feature fusion module is a convolutional layer with a kernel of 3×3 and a stride of 1. It performs a convolution operation on the output features of the dual-branch upsampling module to obtain the final optical flow in two directions. and And the spatial sampling mask M:

[0061]

[0062] A coarse frame is obtained by combining a reference frame, optical flow, and a spatial sampling mask through a frame combining operation. The frame combining operation includes a backward warping operation and a spatial sampling operation. Specifically, a warped frame is obtained by performing a backward warping operation on two reference frames I0 and I1 with the optical flow in two directions, respectively. and Then, spatial sampling is performed using a spatial sampling mask to obtain a coarse frame. The spatial sampling operation is the process of combining two distorted frames into one frame according to the weights assigned by the spatial sampling mask.

[0063]

[0064]

[0065] (3) Construct a refined network.

[0066] The refinement network is located in the fine-frame synthesis stage and is composed of a Unet network; it combines two reference frames I0 and I1, and distorts the frame. and optical flow and The spatial sampling mask M is fed into the Unet network to generate three-channel residual features R. The residual features are used to compensate for the inaccurate coarse frames generated during the coarse frame generation stage due to object motion occlusion and inter-frame color channel differences. The residuals and coarse frames are added together to generate intermediate frames. The specific expression is as follows:

[0067]

[0068] The Unet network described in (3-1) consists of four coding blocks and four decoding blocks. Each coding block is a downsampling layer, composed of two 3×3 convolutions with strides of 2 and 1 respectively. Each convolutional layer is followed by a PReLU activation function. As the number of coding block layers increases, the extracted feature size is halved and the number of channels gradually increases to 32, 64, 128, and 256. Each decoding block is an upsampling layer, performing upsampling convolution operations on the decoded output of the previous layer and the encoded features from the skip connections. Residual features are obtained through four convolution operations.

[0069] (3-2) The object motion occlusion mentioned refers to the situation in which an object or region is occluded by other objects in some frames of a continuous video frame, but is clearly visible in another frame. Especially in nonlinear motion, it is often the case that some regions of the target in the intermediate frame do not exist in the adjacent frames. Directly using the adjacent frames and optical flow information to synthesize the intermediate frame will cause frame interpolation blur. Therefore, residual is used to compensate for this part of the region.

[0070] (3-3) The inter-frame color channel difference refers to the color difference of the same object in adjacent video frames caused by factors such as lighting. In RGB images, it is manifested as the change in the values ​​of the three color channels at the same pixel position. The backward image distortion operation is a global mapping of the pixels of the three color channels of the video frame, thus ignoring the inter-frame color channel difference. Therefore, the residual of the three channels is used to compensate for the color difference, making the final result of the synthesis more realistic.

[0071] (4) Construct the total loss function, L total =λ n L n +λ t L t ; where L n For nonlinear reconstruction loss, L t To guide the loss, λ n and λ t These are the nonlinear reconstruction loss coefficient and the guiding loss coefficient, respectively.

[0072] Nonlinear reconstruction loss is used to better learn the nonlinear information between input frames. with I gt The weighted l1 distance between them is used as the nonlinear reconstruction loss. (βW+1) is the overall weight, W is a three-channel additional weight map, and β is the coefficient of the additional weight map, set to β = 0.1.

[0073] The nonlinear reconstruction loss can adaptively adjust the learning weights based on the nonlinear motion components in the video, enabling the model to effectively learn nonlinear motion information. Specifically, a guide network is used to generate more accurate guide optical flow, and this accurate optical flow is used for nonlinear motion estimation, resulting in a single-channel weight map that can represent the degree of nonlinearity. To address inter-frame color channel differences, intermediate frames generated by a bootstrap network are utilized. Perform channel differentiation operation with the real target frame to transform the weight map of a single channel. An additional weight map W is added to the three-channel model. This weight is used in the loss calculation. A separate weight map is given for each color channel to determine the learning intensity. Parts with high nonlinearity will receive additional learning weights. Therefore, the neural network can not only learn the nonlinear information between input frames in a targeted manner, but also effectively learn color differences, making the final result more realistic.

[0074] use with I gt The L1 distance between them is used as the guiding loss. This is to ensure the accuracy of the network reconstruction.

[0075] (4-1) The guiding network consists of two Unet networks with parameters. Its structure is the same as that of the thinning network. The two Unet networks generate guiding optical flows in two directions respectively, and then the guiding intermediate frame is obtained by using backward twisting operation and spatial sampling operation.

[0076] Specifically, the design philosophy of the guiding network in generating the guiding optical flow is consistent with that of generating the intermediate optical flow during the coarse frame synthesis process. Specified information is fed into Unet to generate the guiding optical flow in the corresponding direction, and the same parameters ensure content consistency. Since the target frame and reference frame serve as input information for the guiding network, the guiding optical flow generated by the guiding network more accurately represents the motion between the target frame and the reference frame. Specifically, it uses the optical flow generated during the coarse frame synthesis stage... Reference frame I0 and target frame I gt After being assembled in sequence, the optical stream is fed into one of the Unets to generate a guiding optical flow in the t→0 direction. In the same way, the optical flow generated in the coarse frame synthesis stage Reference frame I1 and target frame I gt After being assembled sequentially, the optical stream is fed into another Unet to generate a guiding optical flow in the t→1 direction. As in the formula:

[0077]

[0078] The guide intermediate frame is synthesized using backward warping and spatial sampling operations. As in the formula:

[0079]

[0080]

[0081] The guiding optical flow is represented by a two-channel feature map, where each pixel is represented by a two-dimensional coordinate. in, It is the guided optical flow in the t→u direction generated by the network, x u and y u They represent in Horizontal and vertical component feature maps.

[0082] The nonlinear motion estimation described in (4-2) is an estimation of the nonlinear motion of objects in a video using guided optical flow. Each pixel in the guided optical flow has two values ​​representing its motion method and motion length. The degree of nonlinear motion is determined by comparing the difference in motion lengths at the same pixel position in two directions; a larger difference indicates a greater degree of nonlinear motion, while a difference of 0 indicates linear motion. Specifically, the length of each pixel in the guided optical flow in both directions is calculated and normalized to obtain a single-channel weighted map that can express the degree of nonlinearity. As in the formula:

[0083]

[0084] The channel differentiation operation described in (4-3) takes into account the differences in color channels between frames, and like video frames, each color channel is independent, exhibiting non-linearity in continuous time. The resulting... The single-channel weight map is obtained from optical flow, so the results of the teacher model (guided network) are used. The difference between the channels and the actual target frame will Updated to a three-channel additional weight map W to match each color channel, represented as in These are intermediate frames synthesized by the teacher model. U is defined as the operation of performing difference binarization on the two input parts, used to mark the parts with large errors in each color channel between the synthesized frame and the target frame. Specifically... and Pixel values ​​with a difference greater than 0 are set to 1, otherwise to 0. Then, the weight map of a single channel is adjusted. Sampling yields a three-channel additional weight map W that can match each color channel, where α is set to 0.01, representing the tolerance error factor.

[0085] (5) Train the coarse frame synthesis network, the thinning network and the guide network, and use the coarse frame synthesis network and the thinning network to generate high-quality intermediate frames.

[0086] The coarse frame synthesis network, thinning network, and guiding network are jointly trained, and four consecutive frames of images are processed together. -1 I0, I1, and I2 are fed into the coarse frame synthesis network. Then, the features generated by the coarse frame synthesis network, along with the reference frames I0 and I1, are fed into the thinning network. Simultaneously, optical flow information is also included. and The reference frames I0 and I1 are fed into the pilot network for iterative training. After each iteration, all networks use the total loss function for backpropagation until the total loss function reaches a stable state, at which point training ends.

[0087] In summary, this invention first performs data preprocessing to obtain the input image and the target image, and then synthesizes intermediate frames through two stages, from coarse to fine. In the coarse frame synthesis stage, a coarse frame synthesis network is designed. First, a pyramid network with the same kernel sequentially extracts features from consecutive input frames to generate multiple sets of encoded features. Then, an encoded feature fusion module fuses the encoded features, and a dual-branch upsampling module with the same kernel adaptively upsamples and decodes the fused features and the encoded features connected by skip links to generate intermediate optical flow. During the upsampling process, the intermediate optical flow details are compensated for through layer-by-layer refinement, enhancing the accuracy of motion estimation. Finally, the features generated by the dual-branch upsampling module are convolved to generate the final optical flow and spatial sampling mask, resulting in the coarse frame of the coarse frame synthesis stage. In the fine frame synthesis stage, a Unet network with 4 layers for both encoder and decoder is used as a refinement network. The input frame and the intermediate frames from the coarse frame synthesis stage are fed into this network to generate residual feature information. The coarse frame and the residual feature information are then added together to obtain the intermediate frames for the fine frame synthesis stage. In addition, a nonlinear reconstruction loss function was designed to guide the network to learn nonlinear features, which enables the present invention to effectively solve the problem of inaccurate nonlinear motion estimation in video frame interpolation and obtain more realistic intermediate frames.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A nonlinear motion estimation video frame interpolation method based on multi-branch symmetric networks, characterized in that: Acquire consecutive frame images; Four frames with the same interval in a continuous frame image are preprocessed and then input into a coarse frame synthesis network for feature extraction, fusion, dual-branch upsampling and frame synthesis operations to obtain a coarse frame image. The coarse frame image is input into the thinning network for residual information compensation to obtain the intermediate frame; Simultaneously, by combining a guiding network for joint iterative training, more realistic intermediate frames are obtained. The guiding network consists of two parameterized Unet networks, with the same structure as the thinning network. The two Unet networks generate guiding optical flows in two directions respectively. Then, a backward warping operation and a spatial sampling operation are used to obtain the guiding intermediate frame. Specifically, the optical flows generated in the coarse frame synthesis stage are... Reference frame and target frame After being assembled in sequence, the optical stream is fed into one of the Unets to generate a guiding optical flow in the t→0 direction. Similarly, the optical flow generated in the coarse frame synthesis stage is... Reference frame and target frame After being assembled sequentially, the optical stream is fed into another Unet to generate a guiding optical flow in the t→1 direction. For example, in the formula: ， The guide intermediate frame is synthesized using backward warping and spatial sampling operations. For example, in the formula: ， ， The guiding optical flow is represented by a two-channel feature map, where each pixel is represented by a two-dimensional coordinate. ,in, It is the guiding optical flow in the t→u direction generated by the network. and They represent in Horizontal and vertical component feature maps; The total loss function used in the joint iterative training is: ;in For nonlinear reconstruction loss, To guide losses, and These are the nonlinear reconstruction loss coefficient and the guiding loss coefficient, respectively. The nonlinear reconstruction loss includes: , As a whole weight These are the additional weighted graph coefficients. It is a three-channel additional weight map, where U represents the difference binarization operation between the two parts of the input, used to mark the parts with large errors in each color channel between the synthesized frame and the target frame. and Pixel values ​​with a difference greater than 0 are set to 1, otherwise to 0. Then, the weight map of a single channel is adjusted. Sampling is performed to obtain a three-channel additional weight map W that matches each color channel, where Indicates the tolerance error factor; weighting graph of a single channel. The nonlinear motion estimation is obtained from the guiding optical flow generated by the guiding network. Guided loss: .

2. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 1, characterized in that: The coarse frame synthesis network includes an encoder, an encoding feature fusion module, a dual-branch upsampling module, and a decoding feature fusion module; The encoder extracts features from the input frame image; The encoding feature fusion module arranges the extracted features symmetrically and splices them together to obtain two sets of fused features; the dual-branch upsampling module performs input feature interaction operations and layer-by-layer refinement operations on the two sets of input fused features to generate intermediate optical flow and context features in two directions. The decoding feature fusion module performs convolution operations on the intermediate optical flow and context features to obtain optical flows in two directions and a spatial sampling mask. These are then combined with a reference frame for frame synthesis to obtain a coarse frame. The four frames are I... -1 Frames I0, I1, and I2 are used, where I0 and I1 are used as reference frames, and the frame between the two reference frames is used as the target frame I at time t. gt , 0≤t≤1; The encoder consists of four identical core pyramid network coding layers, which process the input four frames of image I. -1 Feature extraction is performed using I0, I1, and I2. Each pyramid network has four downsampling layers used to extract features and continuously halve the feature size. Each layer consists of two 3×3 convolutions with strides of 2 and 1, respectively. Each convolutional layer is followed by a PReLU activation function. Specifically, the coding feature fusion module fuses the fourth-layer coding features of four frames to obtain two sets of fused features, namely... and ,in Indicate I p The i-th layer coding features of the frame, i=1,2,3,4, p=-1,0,1,2.

3. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 2, characterized in that: The dual-branch upsampling module consists of two parallel and parameter-sharing upsampling networks forming two upper and lower branches. Each branch has four identical unidirectional optical flow generation modules in its upsampling network.

4. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 3, characterized in that: The unidirectional optical flow generation module SFB （i） The input features (i=4,3,2,1) consist of one convolutional layer, one residual block, and one deconvolutional layer, outputting the intermediate optical flow and contextual features in a single direction. First, the features input to this module undergo convolution in the first convolutional layer to fuse and extract key information. Next, the residual block performs adaptive motion compensation on the convolutionally processed features to obtain more accurate motion information. The residual block contains two skip connections and three convolutional layers, each followed by a PReLU activation function. The first skip connection adds the input of the residual block to the output of the second convolutional layer, and the second skip connection adds the input of the residual block to the output of the third convolutional layer. Each skip connection is then processed by a PReLU activation function before outputting. Finally, the deconvolutional layer performs upsampling convolution to obtain the intermediate optical flow of that layer. and context features u=0,1, i=4,3,2,1; the first unidirectional optical flow generation module SFB （4） The input feature is the fused feature, while the inputs of the other three modules are the spliced ​​features obtained through interactive operations of the input features.

5. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 4, characterized in that: The aforementioned input feature interaction operation uses the reference features of the upper and lower branches as common features. During the upsampling process, the two branches will call each other's reference features. Specifically, the reference features are... and Symmetrically concatenate the optical flow and context features from the two branches respectively; the concatenation method is as follows: and , among which reference features It is composed of coding features By skipping connections, the intermediate optical flows of corresponding sizes in the two branches are respectively connected. The image obtained after performing a backward image warping operation is expressed as follows: 。 6. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 4, characterized in that: The layer-by-layer refinement operation refers to the operation of adding the current intermediate optical flow in each branch to the intermediate optical flow of the previous layer after performing double bilinear interpolation, resulting in the refined intermediate optical flow. This serves as the input feature for the next unidirectional optical flow output module, and so on, repeating the operation until the last unidirectional optical flow generation module. The specific expression is as follows: 。 7. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 4, characterized in that: The frame synthesis operation includes a backward warping operation and a spatial sampling operation. Specifically, it uses two reference frames I0 and I1 to perform a backward warping operation with the optical flow in two directions to obtain a warped frame. and Then, spatial sampling is performed using a spatial sampling mask to obtain a coarse frame. The spatial sampling operation is the process of combining two distorted frames into a single frame based on the weights assigned by the spatial sampling mask. ， 。 8. The nonlinear motion estimation video frame interpolation method based on a multi-branch symmetric network as described in claim 7, characterized in that: The refinement network is composed of a Unet network, which combines two reference frames I0 and I1 and a distorted frame. and , optical flow and The spatial sampling mask M is fed into the Unet network to generate three-channel residual features R. The residuals are added to the coarse frame to generate the intermediate frame. The specific expression is as follows: ， 。

Images