Video frame interpolation method and system based on lightweight driving and three-scale coding

Abstract

The application discloses a video frame interpolation method and system based on lightweight driving and three-scale coding, obtains an i-th original image and an i+1-th original image of a video to be interpolated; performs scale reduction on the i-th original image and the i+1-th original image respectively to obtain an i-th reduced-scale image, an i+1-th reduced-scale image, an i-th enlarged-scale image and an i+1-th enlarged-scale image; inputs the original images and the images after scale reduction into a trained video frame interpolation model to output an interpolated image; wherein the trained video frame interpolation model performs feature extraction of three scales on the original images and the images after scale reduction respectively, adopts a channel attention mechanism module to fuse the features of the three scales, and adopts a dual attention mechanism module to enhance the fused features; and then performs frame warping operation on the enhanced features to obtain the interpolated image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video frame interpolation method and system based on lightweight driving and three-scale coding. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Video frame interpolation is a technique that synthesizes one or more intermediate frames by extracting spatiotemporal features from consecutive raw frames, which can significantly improve temporal resolution (frame rate). As a hot research topic in computer vision, video frame interpolation plays a crucial role in various applications such as slow-motion generation, image restoration, and video frame rate upconversion. However, the complex motion and occlusion in real-world videos pose significant challenges to video frame interpolation, making it difficult to accurately generate intermediate frames.

[0004] Early traditional methods relied on motion estimation and compensation, depending on motion vectors between input frames. Motion vector estimation can be block-level or pixel-level. Block-level methods somewhat affect the accuracy of motion information, while pixel-level methods require estimation for every pixel of the input frame, leading to a huge computational burden. Therefore, traditional methods cannot perform frame insertion well and may result in overly cumbersome models. In recent years, deep neural network-based methods have gradually replaced traditional methods as the mainstream. Most existing methods are stream-based, kernel-based, or hybrid approaches.

[0005] Some flow-based methods utilize existing flow models to extract optical flow, while others estimate specific flows required by the network as guidance for pixel-level tasks. The resulting flows are then used to warp the original frames to synthesize interpolated frames. However, pre-trained flow models increase model memory and complexity, and task-oriented optical flow cannot accurately handle complex motions and occlusions.

[0006] Kernel-based methods treat pixel interpolation as the convolution of corresponding local patches in two input frames and estimate the convolution kernel for each pixel in the output frame using a deep neural network. However, they are limited by kernel size and motion, unable to handle large motions exceeding the kernel size, and incur significant computational costs. While Niklaus et al. reduced the number of parameters by replacing normal convolution kernels with separable kernels, it still cannot handle large-scale motions. Hybrid methods combine the advantages of the above approaches, but the models are heavier and computationally more expensive. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a video frame interpolation method and system based on lightweight driving and three-scale coding. It employs a channel attention mechanism module to fuse three-scale features and a channel spatial attention mechanism to enhance the combination of encoded and decoded features while preserving motion information. Network performance is improved by using recurrent residual convolutional units and separable recurrent residual convolutional units, while simultaneously reducing model parameters and model size.

[0008] In a first aspect, the present invention provides a video frame interpolation method based on lightweight driving and three-scale coding;

[0009] Video frame interpolation methods based on lightweight driving and three-scale coding include:

[0010] Obtain the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale.

[0011] Both the original image and the scaled image are input into the trained video frame interpolation model, and the interpolated image is output.

[0012] The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image.

[0013] Secondly, this invention provides a video frame interpolation system based on lightweight driving and three-scale coding;

[0014] A video frame interpolation system based on lightweight driving and three-scale coding includes:

[0015] The acquisition module is configured to: acquire the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale.

[0016] The processing module is configured to input both the original image and the scaled image into the trained video frame interpolation model and output the interpolated image.

[0017] The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image.

[0018] Thirdly, the present invention also provides an electronic device, comprising:

[0019] Memory, used for non-transitory storage of computer-readable instructions; and

[0020] Processor, for executing the computer-readable instructions,

[0021] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0022] Fourthly, the present invention also provides a storage medium for non-transitory storage of computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions for the method described in the first aspect are executed.

[0023] Fifthly, the present invention also provides a computer program product, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] This invention uses a three-scale encoder to handle large motions and complex small motions, facilitating the extraction of more accurate depth features. It employs a channel attention mechanism to fuse the three-scale features and a dual attention mechanism to enhance the combination of encoded and decoded features to preserve motion information. By using recurrent residual convolutional units and separable recurrent residual convolutional units, network performance is improved while reducing model parameters and model size. Excellent interpolation results are achieved with fewer model parameters. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is the overall system framework of Embodiment 1 of this application;

[0028] Figure 2 This is a diagram showing the internal connection structure of the channel attention mechanism module in Embodiment 1 of this application;

[0029] Figure 3This is a diagram of the internal connection structure of the dual attention mechanism module CBAM in Embodiment 1 of this application;

[0030] Figure 4 This is a structural diagram of the Circular Residual Convolutional Unit (RRCU) of Embodiment 1 of this application;

[0031] Figure 5 This is a structural diagram of the separable cyclic residual convolutional unit S_RRCU according to Embodiment 1 of this application;

[0032] Figure 6 This is the internal structure of the recurrent convolutional layer when t=2, as shown in Embodiment 1 of this application. Detailed Implementation

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0036] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0037] Example 1

[0038] This embodiment provides a video frame interpolation method based on lightweight driving and three-scale coding;

[0039] Video frame interpolation methods based on lightweight driving and three-scale coding include:

[0040] S101: Obtain the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale.

[0041] S102: Input both the original image and the scaled image into the trained video frame interpolation model and output the interpolated image;

[0042] The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image.

[0043] Furthermore, the trained video frame interpolation model has a network structure that includes an encoder, a decoder, and a frame warping module connected in sequence.

[0044] The encoder includes: three parallel branches; the three parallel branches are a first branch, a second branch, and a third branch; each branch includes three sequentially connected recurrent residual convolutional units;

[0045] The first branch includes: a cyclic residual convolution unit RRCU11, a cyclic residual convolution unit RRCU12, and a cyclic residual convolution unit RRCU13 connected in sequence;

[0046] The second branch includes: a circular residual convolution unit RRCU21, a circular residual convolution unit RRCU22, and a circular residual convolution unit RRCU23 connected in sequence;

[0047] The third branch includes: a cyclic residual convolution unit RRCU31, a cyclic residual convolution unit RRCU32, and a cyclic residual convolution unit RRCU33 connected in sequence;

[0048] In this system, the cyclic residual convolutional units of adjacent branches are connected through a channel attention mechanism module;

[0049] The outputs of the cyclic residual convolution unit RRCU11 and the cyclic residual convolution unit RRCU31 are both connected to the input of the cyclic residual convolution unit RRCU21 through their respective channel attention mechanism modules.

[0050] The outputs of the cyclic residual convolution unit RRCU12 and RRCU32 are both connected to the input of the cyclic residual convolution unit RRCU22 through their respective channel attention mechanism modules.

[0051] The outputs of the cyclic residual convolution unit RRCU13 and RRCU33 are both connected to the input of the cyclic residual convolution unit RRCU23 through their respective channel attention mechanism modules.

[0052] Specifically, the output of the cyclic residual convolution unit RRCU23 is connected to the input of the separable cyclic residual convolution unit S_RRCU1; the output of the separable cyclic residual convolution unit S_RRCU1 is connected to the input of the separable cyclic residual convolution unit S_RRCU2.

[0053] Furthermore, such as Figure 2 As shown, the channel attention mechanism module has the following internal structure:

[0054] Parallel branches R1 and R2;

[0055] The branch R1 includes an average pooling layer, a two-dimensional convolutional layer, a linear rectified function layer, a two-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence; the input of the average pooling layer serves as the input of the channel attention mechanism module.

[0056] The branch R2 includes: an adder J1; the input of the adder J1 is connected to the input of the average pooling layer, and the input of the adder J1 is also connected to the output of the sigmoid activation function layer; the output of the adder J1 serves as the output of the channel attention module.

[0057] Furthermore, the working principle of the channel attention mechanism module includes: two two-dimensional convolutional layers to obtain non-linear inter-channel relationships, an average pooling layer to aggregate channel statistical information, obtaining the weights of each channel in the feature map, assigning a weight value to each channel through the weights of each channel, and focusing on channels with weight values ​​higher than a set threshold in subsequent operations.

[0058] Furthermore, the first branch is used to input the scaled-down image of the i-th frame and the scaled-down image of the (i+1)-th frame;

[0059] The second branch is used to input the original image of the i-th frame and the original image of the (i+1)-th frame;

[0060] The third branch is used to input the image at the magnified scale of the i-th frame and the image at the magnified scale of the (i+1)-th frame.

[0061] For example, this invention belongs to kernel-based video frame insertion technology, which takes two consecutive frames I0 and I2 as input, and then obtains half-scale, original-scale, and double-scale input images from the original-scale input through upsampling and subsampling.

[0062] Furthermore, adjacent recurrent residual convolutional units in the same branch are connected through an average pooling layer;

[0063] The cyclic residual convolutional unit RRCU11 and the cyclic residual convolutional unit RRCU12 are connected through an average pooling layer; the cyclic residual convolutional unit RRCU12 and the cyclic residual convolutional unit RRCU13 are connected through an average pooling layer.

[0064] The cyclic residual convolutional unit RRCU21 and the cyclic residual convolutional unit RRCU22 are connected through an average pooling layer; the cyclic residual convolutional unit RRCU22 and the cyclic residual convolutional unit RRCU23 are connected through an average pooling layer.

[0065] The cyclic residual convolutional unit RRCU31 and the cyclic residual convolutional unit RRCU32 are connected through an average pooling layer; the cyclic residual convolutional unit RRCU32 and the cyclic residual convolutional unit RRCU33 are connected through an average pooling layer.

[0066] The recurrent residual convolutional unit RRCU23 and the separable recurrent residual convolutional unit S_RRCU1 are connected through an average pooling layer; the separable recurrent residual convolutional unit S_RRCU1 and the separable recurrent residual convolutional unit S_RRCU2 are connected through an average pooling layer.

[0067] Furthermore, the cyclic residual convolutional units RRCU11, RRCU12, RRCU13, RRCU21, RRCU22, RRCU23, RRCU31, RRCU32, and RRCU33 are collectively referred to as a three-scale feature extractor.

[0068] Among them, the three scales refer to the 32-dimensional circular residual convolutional unit RRCU11, circular residual convolutional unit RRCU21, and circular residual convolutional unit RRCU31.

[0069] The cyclic residual convolutional unit RRCU12, cyclic residual convolutional unit RRCU22, and cyclic residual convolutional unit RRCU32 adopt 64 dimensions;

[0070] The cyclic residual convolutional units RRCU13, RRCU23, and RRCU33 employ 128 dimensions.

[0071] Furthermore, the internal structures of the cyclic residual convolution units RRCU11, RRCU12, RRCU13, RRCU21, RRCU22, RRCU23, RRCU31, RRCU32, and RRCU33 are identical.

[0072] like Figure 4 As shown, the cyclic residual convolution unit RRCU11 includes:

[0073] Parallel branches A1 and A2;

[0074] The branch A1 includes: a first recurrent convolutional layer and a second recurrent convolutional layer connected in sequence;

[0075] The branch A2 includes: adder J2;

[0076] The input of the first recurrent convolutional layer serves as the input of the recurrent residual convolutional unit RRCU11;

[0077] The input of the adder J2 is connected to the input of the cyclic residual convolution unit RRCU11;

[0078] The input of the adder J2 is also connected to the output of the second recurrent convolutional layer;

[0079] The output of adder J2 serves as the output of cyclic residual convolution unit RRCU11.

[0080] Furthermore, the working principle of the recurrent residual convolutional unit RRCU11 includes: using the residual module, performing pixel-level addition between the features processed by two recurrent convolutional layers and the original features to accumulate features and achieve feature extraction.

[0081] Furthermore, to describe the internal structure of the first recurrent convolutional layer within the recurrent residual convolutional unit RRCU11, (i,j) represents a pixel on the k-th feature map in the first recurrent convolutional layer. The output uses... This means that, where t represents the time step:

[0082]

[0083] Where * denotes a convolution operation. and This represents the feedforward input and the loop input of the l-th layer of the circular convolution. and The weights of the feedforward and recurrent connections are represented by f, r, and b, respectively. kThis is the bias value. The output of the l-th recurrent convolution layer is input into the ReLU activation function, which is expressed as:

[0084]

[0085] In the cyclic residual convolution unit RRCU11, The input residual module performs feature accumulation, and the output of the recurrent residual convolution unit RRCU11 can be expressed as (3):

[0086]

[0087] Among them, u l+1 This represents the output of the cyclic residual convolution unit RRCU11, u l This represents the input to the cyclic residual convolution unit RRCU11.

[0088] It should be understood that the first and second recurrent convolutional layers incorporate recurrent connections into the convolutions, and the state of the recurrent convolutional layers evolves with the time step t. Setting the time step t = 2 results in a feedforward subnetwork of depth 3. The recurrent convolutional layers contain two recurrent connections and three feedforward connections. The inputs to the feedforward connections remain constant throughout all iterations, while the inputs to each recurrent connection change with each iteration. The recurrent connections and feedforward connections have local connections and share weights.

[0089] Furthermore, the encoder operates by processing two adjacent input frames, I0 and I2, at three scales to increase the range of motion reception. Increasing the width of the low-dimensional portion of the encoder improves the overall encoder performance; the width is increased only in dimensions 32, 64, and 128, and weights are shared between different scales to ensure no additional model parameters are added. A channel attention mechanism is used to fuse features from different scales, reducing information loss. This encoder expands the range of motion reception and improves its ability to handle complex small and large motions.

[0090] Furthermore, the decoder includes: a separable cyclic residual convolutional unit S_RRCU3, a separable cyclic residual convolutional unit S_RRCU4, a cyclic residual convolutional unit RRCU24, and a cyclic residual convolutional unit RRCU25 connected in sequence;

[0091] The input of the separable cyclic residual convolution unit S_RRCU3 is connected to the output of the separable cyclic residual convolution unit S_RRCU2.

[0092] The output of the cyclic residual convolution unit RRCU25 is connected to the input of the adaptive flow cooperative twisting module.

[0093] Furthermore, the separable cyclic residual convolutional unit S_RRCU3 and the separable cyclic residual convolutional unit S_RRCU2 are connected through an upsampling layer; the separable cyclic residual convolutional unit S_RRCU3 and the separable cyclic residual convolutional unit S_RRCU4 are connected through an upsampling layer; the separable cyclic residual convolutional unit S_RRCU4 and the cyclic residual convolutional unit RRCU24 are connected through an upsampling layer; and the cyclic residual convolutional unit RRCU24 and the cyclic residual convolutional unit RRCU25 are connected through an upsampling layer.

[0094] Furthermore, the layers of the decoder are connected to the corresponding layers of the encoder via a dual attention mechanism module (CBAM).

[0095] Furthermore, the output of the recurrent residual convolution unit RRCU32 is connected to the input of the first dual attention mechanism module CBAM; the output of the first dual attention mechanism module CBAM and the output of the recurrent residual convolution unit RRCU25 are both connected to the input of the first splicing unit, and the output of the first splicing unit is connected to the input of the upsampling layer.

[0096] The output of the cyclic residual convolution unit RRCU33 is connected to the input of the second dual attention mechanism module CBAM; the output of the second dual attention mechanism module CBAM and the output of the cyclic residual convolution unit RRCU24 are both connected to the input of the second splicing unit, and the output of the second splicing unit is connected to the input of the upsampling layer.

[0097] The output of the separable cyclic residual convolution unit S_RRCU1 is connected to the input of the third dual attention mechanism module CBAM; the output of the third dual attention mechanism module CBAM and the output of the separable cyclic residual convolution unit S_RRCU4 are both connected to the input of the third splicing unit, and the output of the third splicing unit is connected to the input of the upsampling layer.

[0098] The output of the separable cyclic residual convolution unit S_RRCU2 is connected to the input of the fourth dual attention mechanism module CBAM; the output of the fourth dual attention mechanism module CBAM and the output of the separable cyclic residual convolution unit S_RRCU3 are both connected to the input of the fourth splicing unit, and the output of the fourth splicing unit is connected to the input of the upsampling layer.

[0099] Furthermore, the internal structures of the first dual attention mechanism module CBAM, the second dual attention mechanism module CBAM, the third dual attention mechanism module CBAM, and the fourth dual attention mechanism module CBAM are identical;

[0100] Furthermore, such as Figure 3 As shown, the first dual attention mechanism module CBAM includes: parallel branches B1 and B2; the branch B1 includes: parallel branches C1 and C2;

[0101] Branch C1 includes: a max pooling layer Z1, a two-dimensional convolutional layer E1, a linear rectified function layer, and a two-dimensional convolutional layer E2 connected in sequence;

[0102] Branch C2 includes: an average pooling layer P1, a two-dimensional convolutional layer E3, a linear rectified function layer, and a two-dimensional convolutional layer E4 connected in sequence;

[0103] The input of the max pooling layer in branch C1 serves as the input of the first dual attention mechanism module CBAM; the input of the average pooling layer is connected to the input of the max pooling layer.

[0104] The output of the two-dimensional convolutional layer E2 is connected to the input of adder J5; the output of the two-dimensional convolutional layer E4 is connected to the input of adder J5; the output of adder J5 is connected to the input of sigmoid activation function layer S1; the output of sigmoid activation function layer S1 is connected to the input of adder J3.

[0105] The branch B2 includes: adder J3 and adder J4 connected in sequence;

[0106] The output of adder J3 is connected to the input of max pooling layer Z2 and average pooling layer P2, respectively. The outputs of max pooling layer Z2 and average pooling layer P2 are connected to the input of channel splicing unit. The output of channel splicing unit is connected to the input of 2D convolutional layer E5. The output of 2D convolutional layer E5 is connected to the input of Sigmoid activation function layer S2. The output of Sigmoid activation function layer S2 is connected to the input of adder J4. The output of adder J4 serves as the output of the first dual attention mechanism module CBAM.

[0107] The channel splicing unit works by adding the number of channels while keeping the feature size unchanged, thus expanding only the number of channels.

[0108] Furthermore, the internal structures of the separable cyclic residual convolution unit S_RRCU1, separable cyclic residual convolution unit S_RRCU2, separable cyclic residual convolution unit S_RRCU3 and separable cyclic residual convolution unit S_RRCU4 are identical;

[0109] like Figure 5 As shown, the separable cyclic residual convolution unit S_RRCU1 has the following internal structure:

[0110] Parallel branches D1 and D2;

[0111] The branch D1 includes: a separable recurrent convolutional layer K1 and a separable recurrent convolutional layer K2 connected in sequence;

[0112] The branch D2 includes: adder J5;

[0113] The input of the separable circular convolutional layer K1 is used as the input of the separable circular residual convolutional unit S_RRCU1; the output of the separable circular convolutional layer K2 is connected to the input of the adder J5.

[0114] The input of adder J5 is connected to the input of separable circular convolutional layer K1, and the output of adder J5 is used as the output of separable circular residual convolutional unit S_RRCU1.

[0115] Furthermore, to describe the internal structure of the separable recurrent convolutional layer K1 within the separable recurrent residual convolutional unit S_RRCU1, (i,j) represents a pixel on the k-th feature map in the separable recurrent convolutional layer K1. The output uses... This means that, where t represents the time step:

[0116]

[0117] Where ⊙ represents a depthwise separable convolution operation. and This represents the feedforward input and the loop input of the l-th layer of the circular convolution. and The weights of the feedforward and recurrent connections are represented by f, r, and b, respectively. k This is the bias value. The output of the separable recurrent convolutional layer K1 is input into the ReLU activation function, which is expressed as:

[0118]

[0119] In the separable cyclic residual convolution unit S_RRCU1, The input residual module performs feature accumulation, and the output of the separable recurrent residual convolution unit S_RRCU1 can be expressed as (6):

[0120]

[0121] Among them, u l+1 This represents the output of the separable cyclic residual convolution unit S_RRCU1, u l This represents the input to the separable cyclic residual convolution unit S_RRCU1.

[0122] It should be understood that the working principle of the separable cyclic residual convolutional unit S_RRCU1 includes: First, depthwise separable convolution splits ordinary convolution into channel-wise convolution and pointwise convolution. In channel-wise convolution, each channel is convolved by only one convolution kernel, and the number of channels in the resulting feature map is exactly the same as the number of input channels. Pointwise convolution is a 1×1 convolution, allowing depthwise separable convolution to freely change the number of output channels and perform channel fusion on the feature map output by channel-wise convolution. This significantly reduces the parameters of ordinary convolutional layers. In the separable cyclic residual convolutional unit, we replace the ordinary convolution operations used internally with depthwise separable convolution to obtain an efficient convolutional unit.

[0123] Furthermore, the decoder works by taking high-dimensional motion semantics as input, decoding features through separable cyclic residual convolutional units (S_RRCU) and cyclic residual convolutional units (RRCU), using bilinear interpolation for upsampling to change the feature size, and using the dual attention mechanism module CBAM to fuse the encoded features into the decoded features to reduce information loss. The decoder obtains high-quality depth features, and the subsequent frame warping module extracts the required information.

[0124] Furthermore, such as Figure 1 As shown, the frame warping module includes: 7 parallel branches;

[0125] Branch F1 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, a convolutional layer, and a softmax function layer connected in sequence;

[0126] Branch F2 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, and a convolutional layer connected in sequence;

[0127] Branch F3 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, and a convolutional layer connected in sequence;

[0128] Branch F4 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, a convolutional layer, and a sigmoid function layer connected in sequence;

[0129] Branch F5 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, a convolutional layer, and a softmax function layer connected in sequence;

[0130] Branch F6 includes: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, and a convolutional layer connected in sequence.

[0131] Branch F7 consists of: a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, an upsampling layer, and a convolutional layer connected in sequence.

[0132] Furthermore, such as Figure 1 As shown, the frame warping module works by: using branches F1, F2, and F3 to extract the offset vectors and weights of pixels in the x and y directions of the previous input frame; using branches F5, F6, and F7 to extract the offset vectors and weights of pixels in the x and y directions of the next input frame; using branch F4 to estimate the fusion weights of the two frames; determining the importance of corresponding pixels in the two frames; and using these weights to fuse the feature maps of the two frames to obtain the interpolated frame.

[0133] Furthermore, the training process of the trained video frame interpolation model includes:

[0134] Construct a training set, which consists of known videos;

[0135] The training set is input into the video frame interpolation model to train the model. Training is stopped when the overall loss function value of the model no longer decreases, and the trained video frame interpolation model is obtained.

[0136] Understandably, the encoder is used to obtain accurate motion information and fuse multi-scale features to obtain high-level image features. Subsequent subnetworks use this to estimate image parameters, and then use Adaptive Collaboration of Flows (AdaCoF, derived from the paper "Adaptive Collaboration of Flows for Video Frame Interpolation") to perform the warping operation on the input frame, resulting in a high-quality interpolated frame. The lightweight approach is consistent throughout the model, reducing the number of model parameters while ensuring the accuracy of the interpolated frame.

[0137] This invention extracts multi-scale motion semantics only from the low-dimensional part of the encoder, sharing weights across different locations. This allows the invention to obtain more information without adding parameters. Furthermore, this invention uses a combination of depthwise separable convolution and recurrent convolution to achieve local lightweighting, significantly reducing the number of parameters. The key technologies of this invention are reflected in three aspects: a low-dimensional three-scale encoder, a two-level attention cascaded feature fusion method, and the joint local lightweighting of depthwise separable convolution and recurrent convolution.

[0138] The technical solution adopted in this invention is as follows:

[0139] A feature extractor based on three-scale coding: A novel feature extraction module with a three-scale coding and decoding style is designed to perform depth feature estimation on the preceding and following reference frames of the input in three dimensions (32, 64, and 128) to obtain instructive low-dimensional motion semantics.

[0140] A two-level attention cascade strategy based on three-scale features: The three-scale features are fused using a channel attention mechanism module, and the fused three-scale features and the decoded features are cascaded using cascaded channel attention and spatial attention mechanisms to achieve feature enhancement.

[0141] Frame warping operation based on features obtained from the codec: The features extracted by the three-scale codec are used to obtain the pixel-level parameters required by the adaptive stream cooperative warping module through three sub-networks and warp the input frame.

[0142] A joint local lightweighting strategy of depthwise separable convolution and circular convolution: ordinary convolutional units in the low-dimensional part of the encoding and decoding structure are replaced with circular residual convolutional units, and depthwise separable convolution and circular convolution are fused in the high-dimensional part of the encoding and decoding structure to minimize model parameters.

[0143] The two original frames are randomly cropped to obtain a 256×256 block; the scale of the image block is changed by upsampling and subsampling to obtain inputs at half scale, original scale, and double scale; the three scale inputs are respectively input into the low-dimensional encoders of the corresponding scales to obtain low-dimensional motion semantics.

[0144] In the 32, 64, and 128 dimensions, channel weight maps are extracted through a channel attention mechanism, and features at half scale and twice scale are fused into original scale features. Cascaded channel attention and spatial attention mechanisms are used to extract weight maps from the low-dimensional fused features and high-dimensional features, and feature enhancement is performed through pixel-level multiplication. The enhanced encoded features are then fused with the decoded features in the corresponding dimensions through pixel-level addition.

[0145] Based on the depth features obtained by the three-scale feature extractor, the offset vectors of each target pixel in the horizontal and vertical directions and the weights of the convolution kernels of each target pixel are estimated through three sub-networks. An adaptive flow collaboration is used as a warping module, which uses the offset vector to expand the motion sampling range and each pixel does not share weights to warp the input image into an output image. Pixel-level operations are performed to obtain the warped output frame.

[0146] The 32, 64, and 128-dimensional convolutional units in the encoder and decoder are replaced with cyclic residual convolutional units that fuse recurrent convolutional layers and residual operations. The cyclic residual convolutional units can reduce the number of parameters compared to the convolutional units. In the 256 and 512-dimensional dimensions, depthwise separable convolutions are fused with cyclic residual convolutional units, and the depthwise separable convolutions are used to reduce the number of parameters in the model by about 90%.

[0147] This invention proposes a frame insertion algorithm and system based on lightweight driving and a three-scale coding structure, which can insert frames into low frame rate videos with fewer parameters. The entire process is as follows: Figure 1 The specific implementation method is as follows:

[0148] (1) Data Preparation. This invention uses the most common open-source dataset Vimeo-90k as the training set. Vimeo-90k consists of 51,312 sets of three-frame images with a resolution of 256*448. The first and third frames are used as input, and the second frame is used as the ground truth frame to fine-tune the training results. This invention further randomly crops the dataset to obtain image patches with a resolution of 256*256. By randomly flipping the samples horizontally and vertically and scrambling the temporal order, data diversity is increased. The test set consists of common open-source datasets Middlebury, DAVIS, and UCF101.

[0149] (2) Three-Scale Motion Feature Extraction. This invention designs a three-scale feature extractor based on the U-Net encoder-decoder structure, which processes the input frame at three scales to increase the motion reception range. This invention only increases the encoder width in dimensions 32, 64, and 128, and shares weights across different scales, controlling parameters while improving the overall encoder performance. This three-scale encoder expands the motion reception range to handle complex and large-scale motions without increasing model parameters.

[0150]

[0151] L lowEn {·},L highEn {·},L De {·} represent the low-dimensional 3D encoder, the high-dimensional encoder, and the decoder, respectively. F represents the input at three scales. -1 ,F 0 ,F 1 This represents the low-dimensional motion semantics obtained at three scales. The three sets of low-dimensional semantics are then fused and input into a high-dimensional encoder to obtain high-dimensional motion semantics F. Finally, a decoder is used to obtain the final features OUT to represent motion details.

[0152] (3) Two-level attention cascade strategy based on three-scale features. In order to better integrate motion information from three scales and reduce information loss, this invention proposes a channel attention mechanism module and a dual attention mechanism module CBAM.

[0153] The channel attention mechanism module adaptively obtains channel weight maps of different layers (32, 64, and 128 dimensions) of the encoder to preserve information about motion content and occlusion. Information at each scale is fused into the original scale features through sampling. The structure of the channel attention mechanism module is as follows: Figure 2 As shown, the channel attention mechanism module consists of two 1×1 convolutional layers to obtain nonlinear inter-channel relationships, while average pooling is used to aggregate channel statistics. The channel attention mechanism module can be mathematically represented as (8).

[0154] EIC(F)=σ(W1*(ReLU(W0*avg(F)))) (8)

[0155] Where W0 and W1 represent the weights of two 1×1 convolutions, F represents the input features, avg(F) represents the average pooling of the feature map to obtain a new feature map, ReLU(·) represents the linear rectified function, * represents the convolution operation, σ(·) represents the sigmoid function, and EIC(F) represents the channel attention mechanism module.

[0156] The Dual Attention Mechanism (CBAM) module adaptively obtains spatial weight maps and channel weight maps for different layers of the entire encoder. After the channel attention mechanism module fuses the three low-dimensional scales, the Dual Attention Mechanism (CBAM) module extracts the weight maps and enhances the fused original scale features to preserve motion and occlusion information. Figure 3 The detailed architecture of the Dual Attention Mechanism (CBAM) module is shown. CBAM uses average pooling and max pooling in parallel to aggregate channel information. Channel attention feature maps are obtained through 1×1 convolutions and the sigmoid function. This invention multiplies the input features and the channel attention feature maps pixel-by-pixel to obtain feature F'. Using F' as input for the second stage, average pooling and max pooling aggregate the statistical information of one channel respectively, and then the two feature maps are concatenated based on the channel. Finally, a 7×7 convolution reduces the channels to one channel, and the sigmoid function is used to generate a spatial attention feature map. The final feature is obtained through pixel-by-pixel multiplication. The two stages of the Dual Attention Mechanism (CBAM) module can be represented as follows:

[0157]

[0158] In (3), W2 and W3 represent the weights of the two 1×1 convolutions in the first row of the first stage. The convolutional layers in the two rows share the weights. 7×7This represents a 7×7 convolution. F represents the input features, avg(·) and max(·) represent average pooling and max pooling operations respectively, σ(·) and * represent the σ function and convolution, ReLU(·) represents the rectified linear function, and... and × represent element addition and multiplication, respectively, and CIC(F) represents the dual attention mechanism module CBAM.

[0159] This invention represents the process of extracting structural features from the encoding / decoding process as feature acquisition. We first utilize a channel attention mechanism module to enhance the low-dimensional feature maps at the two additional scales. Then, we fuse these features into the original scale through sampling operations to reduce information loss due to large and small motions. We use a dual attention mechanism module (CBAM) to process the fused low-dimensional features and the original high-dimensional features, and then add them separately to the corresponding decoded features to preserve the encoded information.

[0160] (4) Frame warping operation based on features obtained from the codec. This invention uses Adaptive Stream Cooperative (AdaCoF) to synthesize interpolated frames. The features extracted by the three-scale codec are passed through three sub-networks to obtain the pixel-level parameters required by the Adaptive Stream Cooperative warping module and warp the input frame. It estimates the offset vector of each target pixel to expand the information sampling position. The weights of each pixel are no longer shared. Its operation can be mathematically represented by the following equation:

[0161]

[0162] I t =⊙I n +(JV)⊙I n+1

[0163] Where ⊙ represents pixel multiplication, J and V represent an M×N 1 matrix and an occlusion map, and I n and I n+1 Representing the two frames respectively, I t This represents the final result obtained by fusing two consecutive frames. S represents the convolution kernel size, (m,n) represents the pixel position, and W... p, (m,n), d and d are the weights, offset vectors, and inflation values ​​of the (p, q)th kernel.

[0164] (5) A joint local lightweighting strategy combining depthwise separable convolution and recurrent convolution. In the multi-scale encoding / decoding architecture, this invention introduces a Recurrent Residual Convolutional Unit (RRCU) to replace the original ordinary convolutional units in dimensions 32, 64, and 128. The key component of this unit is the Recurrent Convolutional Layer (RCL), which merges recurrent connections into ordinary convolutions. The state of the RCL evolves with time step t. Simultaneously, this invention merges recurrent connections into depthwise separable convolutions, proposing a separable RRCU (S_RRCU). The specific structures of RRCU and S_RRCU are as follows... Figure 4 and Figure 5 As shown. Each RRCU contains two RCLs. (See diagram) Figure 6 As shown, when the time step t = 2, this invention obtains a feedforward subnetwork with a depth of 3. The feedforward input remains constant throughout all iterations, while each recurrent input evolves with each iteration. When t = 2, the RCL consists of two recurrent connections and three feedforward connections, where the recurrent and feedforward connections are locally connected and share weights. The RRCU contains only two RCLs, and the RCLs share weights internally, while the original convolutional unit contains three ordinary convolutional layers. Therefore, the RRCU can achieve better interpolation results with fewer parameters. Ordinary convolutional units in the low-dimensional part of the encoding / decoding structure are replaced with RRCUs, and depthwise separable convolutions and recurrent convolutions are fused in the high-dimensional part of the encoding / decoding structure, replacing ordinary convolutional units with S_RRCUs to minimize model parameters.

[0165] This paper proposes a three-scale encoding / decoding structure with two-level attention cascades to represent multi-scale motion information. Then, it employs recurrent convolutional layers and residual operations to design recurrent residual convolutional units (RNRs) to replace the ordinary convolutional units in the three-scale structure. Finally, this invention applies the concept of local lightweighting to significantly reduce model parameters by fusing depthwise separable convolutions and recurrent convolutions in the high-dimensional part of the encoding / decoding structure to minimize model parameters. The proposed three-scale structure improves model performance without increasing model parameters, and the RRNs play a crucial role in reducing parameters.

[0166] Example 2

[0167] This embodiment provides a video frame interpolation system based on lightweight driving and three-scale coding;

[0168] A video frame interpolation system based on lightweight driving and three-scale coding includes:

[0169] The acquisition module is configured to: acquire the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale.

[0170] The processing module is configured to input both the original image and the scaled image into the trained video frame interpolation model and output the interpolated image.

[0171] The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image.

[0172] It should be noted that the acquisition module and processing module described above correspond to steps S101 to S102 in Embodiment 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the modules described above, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0173] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0174] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0175] Example 3

[0176] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0177] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0178] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0179] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0180] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0181] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0182] Example 4

[0183] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0184] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video frame interpolation method based on lightweight driving and three-scale coding, characterized by: include: Obtain the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale. Both the original image and the scaled image are input into the trained video frame interpolation model, and the interpolated image is output. The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image. The trained video frame interpolation model has a network structure consisting of an encoder, a decoder, and a frame warping module connected in sequence. The encoder includes: three parallel branches; the three parallel branches are a first branch, a second branch, and a third branch; each branch includes three sequentially connected recurrent residual convolutional units; the recurrent residual convolutional units of adjacent branches are connected through a channel attention mechanism module; The recurrent residual convolutional unit includes: parallel branches A1 and A2; branch A1 includes: a first recurrent convolutional layer and a second recurrent convolutional layer connected in sequence; branch A2 includes: an adder J2; the input of the first recurrent convolutional layer serves as the input of the recurrent residual convolutional unit; the input of the adder J2 is connected to the input of the recurrent residual convolutional unit; the input of the adder J2 is also connected to the output of the second recurrent convolutional layer; the output of the adder J2 serves as the output of the recurrent residual convolutional unit; the working principle of the recurrent residual convolutional unit includes: performing pixel-level addition between the features processed by the two recurrent convolutional layers and the original features to accumulate features and achieve feature extraction; The decoder includes: two separable cyclic residual convolutional units and two cyclic residual convolutional units connected in sequence; each layer of the decoder is connected to the corresponding layer of the encoder through a dual attention mechanism module CBAM. The separable cyclic residual convolution unit has the following internal structure: The parallel branches D1 and D2; branch D1 includes: a separable circular convolutional layer K1 and a separable circular convolutional layer K2 connected in sequence; branch D2 includes: an adder J5; The input of the separable circular convolutional layer K1 serves as the input of the separable circular residual convolutional unit; the output of the separable circular convolutional layer K2 is connected to the input of the adder J5. The input of adder J5 is connected to the input of separable circular convolutional layer K1, and the output of adder J5 serves as the output of separable circular residual convolutional unit.

2. The video frame interpolation method based on lightweight driving and three-scale coding as described in claim 1, characterized in that, The channel attention mechanism module has an internal structure including two parallel branches, R1 and R2. Branch R1 includes an average pooling layer, a two-dimensional convolutional layer, a linear rectified function layer, a two-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence. The input of the average pooling layer serves as the input of the channel attention mechanism module. The branch R2 includes: an adder J1; the input of the adder J1 is connected to the input of the average pooling layer, and the input of the adder J1 is also connected to the output of the sigmoid activation function layer; the output of the adder J1 serves as the output of the channel attention module. The working principle of the channel attention mechanism module includes: two two-dimensional convolutional layers to obtain non-linear inter-channel relationships, an average pooling layer to aggregate channel statistical information, obtaining the weights of each channel in the feature map, and assigning a weight value to each channel through the weights of each channel.

3. The video frame interpolation method based on lightweight driving and three-scale coding as described in claim 1, characterized in that, The dual attention mechanism module CBAM includes: parallel branches B1 and B2; branch B1 includes: parallel branches C1 and C2; Branch C1 includes: a max pooling layer Z1, a two-dimensional convolutional layer E1, a linear rectified function layer, and a two-dimensional convolutional layer E2 connected in sequence; Branch C2 includes: an average pooling layer P1, a two-dimensional convolutional layer E3, a linear rectified function layer, and a two-dimensional convolutional layer E4 connected in sequence; The input of the max pooling layer in branch C1 serves as the input of the dual attention mechanism module CBAM; the input of the average pooling layer is connected to the input of the max pooling layer. The output of the two-dimensional convolutional layer E2 is connected to the input of adder J5; the output of the two-dimensional convolutional layer E4 is connected to the input of adder J5; the output of adder J5 is connected to the input of sigmoid activation function layer S1; the output of sigmoid activation function layer S1 is connected to the input of multiplier J3. The branch B2 includes: multipliers J3 and J4 connected in sequence; The output of multiplier J3 is connected to the input of max pooling layer Z2 and average pooling layer P2, respectively. The outputs of max pooling layer Z2 and average pooling layer P2 are connected to the input of channel splicing unit. The output of channel splicing unit is connected to the input of 2D convolutional layer E5. The output of 2D convolutional layer E5 is connected to the input of Sigmoid activation function layer S2. The output of Sigmoid activation function layer S2 is connected to the input of multiplier J4. The output of multiplier J4 serves as the output of dual attention mechanism module CBAM.

4. A video frame interpolation system based on lightweight driving and three-scale coding, employing the video frame interpolation method based on lightweight driving and three-scale coding as described in any one of claims 1-3, characterized in that it comprises: The acquisition module is configured to: acquire the original image of the i-th frame and the original image of the (i+1)-th frame of the video to be interpolated; scale the original image of the i-th frame and the original image of the (i+1)-th frame respectively to obtain the image of the i-th frame at a reduced scale, the image of the (i+1)-th frame at a reduced scale, the image of the i-th frame at a magnified scale, and the image of the (i+1)-th frame at a magnified scale. The processing module is configured to input both the original image and the scaled image into the trained video frame interpolation model and output the interpolated image. The trained video frame interpolation model extracts features at three scales from both the original image and the scaled image. It then uses a channel attention mechanism to fuse the features at the three scales and a dual attention mechanism to enhance the fused features. Finally, it performs frame warping on the enhanced features to obtain the interpolated image.

5. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-3.

6. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-3 are executed.

Images