Method and system for video frame rate upscaling based on multi-scale coarse-to-fine transformer network

Abstract

The application provides a multi-scale coarse-to-fine converter network-based video frame rate improvement method and system, which comprises the following steps: extracting features from input integer frames by using a feature extraction network; performing implicit motion estimation on the integer frame features by using a coarse intermediate frame estimation network in a first stage network to obtain coarse intermediate frames; migrating the integer frame features to the positions of the coarse intermediate frames by using a feature refinement converter network in a second stage network to obtain fine intermediate frames; establishing a coarse intermediate frame loss function and a fine intermediate frame loss function, and training by adding the two loss functions; and inputting continuous multi-frame low frame rate videos into the trained first stage network and second stage network to reconstruct fine intermediate frames. In this embodiment, the video frame interpolation is performed in a two-stage step-by-step refinement manner, and the motion information is extracted by using a pure space-time convolution manner to avoid presetting a motion model to estimate the optical flow. Meanwhile, the converter network is used to estimate the many-to-one mapping between the integer frames and the intermediate frames to improve the robustness of the feature migration.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and more specifically, to a method and system for improving video frame rate based on a multi-scale coarse-to-fine transformer network. Background Technology

[0002] Video frame interpolation aims to synthesize missing intermediate frames from low frame rate videos to increase the frame rate, making the video smoother and more fluid. With the upgrading of display devices and the increasing demands for video viewing experiences, the need for high frame rate videos is growing in various scenarios, including video recording, slow-motion capture, new perspective synthesis, video-on-demand, and live streaming. However, limited by capture equipment and storage, existing older video sources often have low frame rates. During playback, low frame rate videos often cause perceptual discontinuities in moving scenes or areas, affecting the subjective visual experience. Furthermore, due to the increasing demand in fields such as virtual reality, perspective synthesis technology is becoming increasingly important; however, rendering a single frame requires significant complexity, posing considerable challenges to computation and applications. Therefore, using video frame interpolation to synthesize new perspectives can effectively reduce the enormous complexity of rendering.

[0003] Most existing video frame interpolation methods are stream-based, which estimate the dense optical flow from integer frames to intermediate frames, then transfer features to intermediate time points using the estimated flow, and finally synthesize intermediate frames using residual estimation. However, these methods typically have two problems: First, during motion estimation, stream-based methods usually require a pre-defined motion model (linear, quadratic, cubic). When encountering complex motion scenes or regions of the current frame with complex nonlinear motion, these low-order motion models struggle to achieve effective motion estimation, resulting in lower estimated flow and consequently affecting the correctness of the transferred features and the accuracy of the final synthesized frame. The second problem lies in the feature transfer process. Optical flow can be viewed as a one-to-one mapping between pixels, and its performance is highly dependent on the accuracy of motion estimation, exhibiting low tolerance for errors. If the motion estimation is inaccurate, the performance of feature transfer will be significantly affected. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for improving the video frame rate of a multi-scale, coarse-to-fine converter network.

[0005] According to one aspect of the present invention, a method for improving the video frame rate of a multi-scale coarse-to-fine converter network is provided, comprising:

[0006] A feature extraction network is used to extract integer frame features from the input integer frames;

[0007] The first-stage network uses a coarse intermediate frame estimation network to perform implicit motion estimation on the features of the integer frames to obtain coarse intermediate frames;

[0008] The second-stage network uses a feature refinement transformer network to migrate the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames.

[0009] Establish the coarse intermediate frame loss function and the fine intermediate frame loss function, and train by adding the two loss functions together;

[0010] Multiple consecutive low frame rate videos are input into the trained first-stage network and second-stage network to reconstruct fine intermediate frames.

[0011] Preferably, the feature extraction network includes three 2D convolutional layers, each of which extracts features from the input integer frames and downsamples them to obtain integer frame features at three scales, which serve as the input to the first-stage network.

[0012] Preferably, the coarse intermediate frame estimation network is a pure 3D convolutional U-shaped network consisting of three scales, each consisting of several 3D convolutional layers. The three scales are fine scale, intermediate scale, and coarse scale in order of decreasing resolution. The feature refinement transformer network includes four transformer blocks with different block sizes, varying from large to small, to extract associations at different scales from coarse to fine.

[0013] Preferably, the first-stage network uses a coarse intermediate frame estimation network to perform implicit motion estimation on the integer frame features to obtain coarse intermediate frames, including:

[0014] The integer frame features at the three scales are input into the coarse intermediate frame estimation network;

[0015] Each integer-scale frame feature is processed using a three-layer network; where...

[0016] The first layer is a 3D convolutional layer, used to fuse features from multiple integer frames;

[0017] The second layer is a 3D convolutional layer, which is responsible for feature refinement at the fine scale, for fusing and refining the features output from the first layer at the current scale and the fine scale at the intermediate scale, and for fusing and refining the features output from the first layer at the current scale and the intermediate scale at the coarse scale.

[0018] The third layer is a 3D convolutional residual block, which consists of two 3D convolutional layers connected by residuals. It is responsible for feature refinement at the coarse scale, and for fusing and refining the features output from the second layer at the current scale and the intermediate scale at the fine scale. At the intermediate scale, it is responsible for fusing and refining the features output from the second layer at the current scale and the coarse scale.

[0019] The time dimension of the refined features is expanded and merged into the channel dimension, and the time dimension is removed, thus reducing the dimensionality of the features.

[0020] The expanded and merged features are subjected to dimensionality reduction through three layers of 2D convolution to obtain coarse intermediate frame features, which are then synthesized into coarse intermediate frames, resulting in coarse intermediate frames at three scales.

[0021] In the coarse intermediate frame estimation network, except for the last 2D convolutional layer which does not use an activation layer, each other convolutional layer uses the LeakyReLU activation function, which is to apply a non-zero slope to negative values ​​based on the ReLU of the linear rectified layer.

[0022] Preferably, the second layer is a 3D convolutional layer, which downsamples the output features of the first layer at a fine scale and fuses them with features at an intermediate scale; it downsamples the output features of the first layer at an intermediate scale and fuses them with features at a coarse scale; the downsampling of the features is accomplished by additional 3D convolution.

[0023] Preferably, the third layer is a 3D convolutional residual block that upsamples the output features of the coarse-scale second layer and fuses them with intermediate-scale features; it also upsamples the output features of the intermediate-scale second layer and fuses them with fine-scale features; the upsampling of the features is accomplished by additional 3D deconvolution.

[0024] Preferably, the second-stage network employs a feature refinement transformer network to transfer the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames, including:

[0025] The coarse intermediate frame and integer frame features at the three scales are used as inputs to the second-stage network;

[0026] For each scale of coarse intermediate frame, four consecutive transformer blocks are used to migrate the integer frame features of the corresponding scale to the coarse intermediate frame position to enhance the quality of the coarse intermediate frame features.

[0027] The coarse intermediate frame features, after being processed by the transformer block, are aggregated by a frame synthesis module consisting of three residual densely connected blocks to obtain a fine intermediate frame.

[0028] Preferably, the process performed in each of the converter blocks includes:

[0029] The image features of coarse intermediate frames and integer frames are divided into blocks to obtain several image blocks;

[0030] Using a self-attention mechanism, the association between the image patches is estimated and matched. Specifically, image patches with coarse intermediate frame features are used as references to perform feature matching with image patches divided by integer frame features to generate weights.

[0031] The weighted sum of integer frame features is calculated based on the weights and used as the refined feature for the current position of the coarse intermediate frame, i.e., Attention(Q,K,V) = Softmax(QK). T )V, where Q is the vector to be queried, referring to the coarse intermediate frame features; K and V are the queried vector and the queried value, respectively, both being features of the input integer frame.

[0032] Preferably, a comprehensive loss function is established, consisting of a coarse intermediate frame loss function and a fine intermediate frame loss function:

[0033] L = L OC +L O

[0034] This includes the loss function L of the coarse intermediate frame. OC and the loss function L for fine intermediate frames O Both loss functions are implemented using L1 loss:

[0035] L(O,GT)=||O-GT||1

[0036] That is, calculating the L1 loss between the reconstructed intermediate frame and the true value, where GT is the true intermediate frame, O is the reconstructed intermediate frame output by the network, and L... OC In the text, O represents a coarse intermediate frame, and L represents a coarse intermediate frame. O The 'O' in the text represents a fine intermediate frame;

[0037] For reconstructed intermediate frames at different scales, the true values ​​are downsampled using bilinear interpolation to calculate the loss value.

[0038] According to a second aspect of the present invention, a video frame rate enhancement system based on a multi-scale coarse-to-fine converter network is provided, comprising:

[0039] The feature extraction module extracts integer frame features from the input integer frames;

[0040] The first stage module uses a coarse intermediate frame estimation network to perform implicit motion estimation on the features of the integer frames to obtain coarse intermediate frames.

[0041] The second-stage module uses a feature refinement transformer network to migrate the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames.

[0042] The constraint training module establishes the coarse intermediate frame loss function and the fine intermediate frame loss function, and trains by adding the two loss functions together.

[0043] The application module inputs multiple consecutive low frame rate video frames into the trained first-stage network and second-stage network to reconstruct fine intermediate frames.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The video frame rate enhancement method and system based on a multi-scale coarse-to-fine converter network in this invention employs a streamless approach to implicitly extract intermediate motion from the video, avoiding the need for pre-defined low-order motion models. This allows for better handling of nonlinear and complex motion scenes and content, and the estimated intermediate frames have a more accurate structure. In video viewing, a more accurate intermediate frame structure also results in a smoother visual experience.

[0046] The video frame rate enhancement method and system based on multi-scale coarse-to-fine converter network in this embodiment of the invention employs a converter network structure for feature transfer and refinement. The converter network uses a many-to-one matching and mapping method, which greatly improves the robustness and fault tolerance of feature matching and reduces the probability of the first-stage error accumulating in the second stage. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart of an embodiment of the present invention based on a multi-scale coarse-to-fine converter network video frame rate enhancement method;

[0049] Figure 2 This is a structural diagram of the coarse intermediate frame estimation network in a preferred embodiment of the present invention;

[0050] Figure 3 This is a structural diagram of the feature refinement converter network in a preferred embodiment of the present invention;

[0051] Figure 4 This is a structural diagram of the intermediate frame reconstruction network in a preferred embodiment of the present invention;

[0052] Figure 5 This is a comparison of the effect of the predicted intermediate frames in a preferred embodiment of the present invention with existing pairing methods. Detailed Implementation

[0053] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0054] See Figure 1 This invention provides an embodiment of a method for improving the video frame rate of a multi-scale coarse-to-fine converter network, comprising:

[0055] S100, a feature extraction network is used to extract integer frame features from the input integer frames;

[0056] S200, the first stage network uses a coarse intermediate frame estimation network to perform implicit motion estimation on the integer frame features extracted in S100 to obtain coarse intermediate frames;

[0057] In S300, the second-stage network uses a feature refinement transformer network to transfer the integer frame features obtained in S100 to the position of the coarse intermediate frame obtained in S200 for quality enhancement, thus obtaining a fine intermediate frame.

[0058] S400: Establish the coarse intermediate frame loss function of S200 and the fine intermediate frame loss function of S300, and train by adding the two loss functions together;

[0059] The S500 takes several consecutive low frame rate video frames as input to a first-stage network and a second-stage network, which are trained to reconstruct fine intermediate frames.

[0060] This embodiment constructs a two-stage, multi-scale video interpolation structure, performing motion estimation and feature transfer based on the estimated coarse intermediate frame features, respectively. Both stages involve computation and estimation at multiple scales. This embodiment employs a two-stage, progressively refined approach for video interpolation, using pure spatiotemporal convolution to extract motion information, thus avoiding the use of pre-defined motion models to estimate optical flow. A transformer network is also employed to estimate a many-to-one mapping between integer frames and intermediate frames, improving the robustness of feature transfer. Here, the many-to-one mapping refers to the association between blocks in the feature refinement transformer network. For one block, the similarity to all other blocks needs to be calculated; therefore, multiple blocks with high similarity can be found, which can be considered a many-to-one mapping.

[0061] Ideally, four consecutive low frame rate video frames (numbered frame 1, frame 2, frame 3, and frame 4) are used as input to fully estimate the underlying motion information. Intermediate frames are then synthesized: for 2x frame rate, frame 2.5 is synthesized; for 4x frame rate, frames 2.25, 2.5, and 2.75 are synthesized. Alternatively, two frames can be used as input, but the motion information extracted between two frames is less than that extracted from four frames. More input frames, such as six or eight, can also be used, but this increases computational complexity and buffer requirements. Therefore, using four frames as input helps extract richer motion information and introduces less buffering.

[0062] In a preferred embodiment of the present invention, step S100 is implemented, in which a feature extraction network is used to extract integer frame features from the input integer frames. In this embodiment, the feature extraction network consists of three 2D convolutional layers, whose main function is to extract features from the input integer frames and reduce their dimensionality to obtain the input for each scale. Specifically, each convolutional layer is followed by a Rectified Linear Activation Function (RELU). The first convolutional layer uses a 7×7 kernel and reduces dimensionality by setting a stride of 2; the other two layers use 3×3 kernels and also reduce dimensionality by setting a stride of 2. Finally, features with resolutions of 1 / 2, 1 / 4, and 1 / 8 of the original video are obtained as the input for each scale.

[0063] This embodiment employs a shallow neural network to extract features from integer frames, effectively improving the efficiency of feature extraction. Simultaneously, the features extracted by the shallow network can preserve the details and positional information of the original image to the greatest extent possible, which facilitates the implementation of the second-stage feature transfer.

[0064] In a preferred embodiment of the present invention, in step S200, the first-stage network uses a coarse intermediate frame estimation network to perform implicit motion estimation on the features of integer frames to obtain coarse intermediate frames.

[0065] In this embodiment, the coarse intermediate frame estimation network is based on a pure 3D U-shaped network structure for implicit motion estimation and coarse intermediate frame estimation. A pure 3D U-shaped network typically first extracts and aggregates features through multi-scale downsampling, and then uses the downsampled features to gradually upsample back to the original resolution. This model's structure resembles a "U" shape, hence the name U-shaped network. The U-shaped network in this embodiment consists of three scales, each composed of 3D convolutional layers, and each scale has a similar structure. The three scales are called the fine scale, intermediate scale, and coarse scale, corresponding to high, medium, and low resolution features, respectively. Thus, the pure 3D convolutional U-shaped network performs two downsampling operations to obtain features at three scales; each scale feature is processed separately; starting with the coarse scale features, it is upsampled and aggregated with the intermediate scale features, then upsampled again and aggregated with the fine scale features to obtain the coarse intermediate frame.

[0066] The coarse intermediate frame estimation network is deployed at three scales. Specifically, the process of obtaining coarse intermediate frames is as follows:

[0067] Integer frame features at three scales are input into the coarse intermediate frame estimation network;

[0068] Each integer-scale frame feature is processed using a three-layer network; where...

[0069] The first layer is a 3D convolutional layer, used to fuse features from multiple integer frames;

[0070] The second layer is a 3D convolutional layer, which is responsible for feature refinement at the fine scale, for fusing and refining the features output from the first layer at the current scale and the fine scale at the intermediate scale, and for fusing and refining the features output from the first layer at the current scale and the intermediate scale at the coarse scale.

[0071] The third layer is a 3D convolutional residual block, which consists of two 3D convolutional layers connected by residuals. It is responsible for feature refinement at the coarse scale, and for fusing and refining the features output from the second layer at the current scale and the intermediate scale at the fine scale. At the intermediate scale, it is responsible for fusing and refining the features output from the second layer at the current scale and the coarse scale.

[0072] The time dimension of the refined features is expanded and merged into the channel dimension, and the time dimension is removed, thus reducing the dimensionality of the features from 5 dimensions to 4 dimensions.

[0073] The expanded and merged features are subjected to dimensionality reduction through three layers of 2D convolution to obtain coarse intermediate frame features, which are then synthesized into coarse intermediate frames, resulting in coarse intermediate frames at three scales.

[0074] In a preferred embodiment, the second layer is a 3D convolutional layer that downsamples the output features of the first layer at a fine scale and fuses them with features at an intermediate scale; it also downsamples the output features of the first layer at an intermediate scale and fuses them with features at a coarse scale; the downsampling of the features is accomplished by additional 3D convolution.

[0075] In a preferred embodiment, the third layer is a 3D convolutional residual block that upsamples the output features of the coarse-scale second layer and fuses them with intermediate-scale features; it upsamples the output features of the intermediate-scale second layer and fuses them with fine-scale features; the upsampling of features is accomplished by additional 3D deconvolution.

[0076] In the first-stage network, except for the last convolutional layer, every other convolutional layer uses the LeakyReLU activation function, which applies a non-zero slope to negative values ​​on top of the ReLU layer. In this embodiment, the ReLU layer directly sets all negative values ​​to 0, which would result in the loss of information contained in the negative values. Applying a non-zero slope to the negative values ​​on this basis can reduce information loss and make full use of the information contained in the negative values.

[0077] To better understand the processing of the coarse intermediate frame estimation network, this invention provides an embodiment using an intermediate scale (1 / 4 resolution) as an example. Please refer to... Figure 2 The features extracted from four integer frames are combined and input into the first 3D convolutional layer. Then, they are concatenated with the features from the previous scale in the channel dimension using another 3D convolutional layer. The resulting features undergo dimensionality reduction and are simultaneously passed to the next network layer.

[0078] In the next layer, the input is a combination of the previously transmitted features and the features upscaled to the next scale. After two layers of 3D convolutions and residual connections, optimized features are obtained. These features are also used for upscaling and propagation to the previous scale.

[0079] At the same time, the channel dimension and time dimension of this feature are merged, that is, expanded from B×C×T×H×W to B×CT×H×W;

[0080] The features from four time steps are aggregated into one time step using a two-layer 2D convolutional network to generate intermediate frame features. Finally, a coarse intermediate frame is synthesized using a single 2D convolutional layer. Figure 2 In the diagram, k and s of the 3D convolutional layer have three numbers, corresponding to the kernel size and stride size in the T, H, and W dimensions. N corresponds to the number of output channels, which varies for each scale, ranging from 128, 192, and 256 from top to bottom.

[0081] In this embodiment, a streamless method is used to implicitly extract intermediate motion from the video, avoiding the need for pre-defined low-order motion models. This approach better handles non-linear and complex motion scenes and content, resulting in more accurate estimated intermediate frames. In video viewing, a more accurate intermediate frame structure also leads to a smoother visual experience.

[0082] In a preferred embodiment of the present invention, in step S300, the second-stage network uses a feature refinement transformer network to migrate the features of integer frames to the position of coarse intermediate frames and enhance them to obtain fine intermediate frames.

[0083] The second-stage network targets the intermediate frame features and integer frame features extracted from S200 at three scales. At each scale (except the coarse scale), it employs four consecutive transformer blocks to transfer the integer frame features to the intermediate frame positions to enhance the quality of the intermediate frame features. Each transformer block utilizes a self-attention mechanism, using the intermediate frame features as a reference, to perform feature matching with the integer frame features in the local block, generate weights, and calculate the weighted sum of the features as the refined features for the current intermediate frame position. Specifically:

[0084] Attention(Q,K,V)=Softmax(QK T V

[0085] Where Q is the vector to be queried, referring to the features of the intermediate frame; K and V are the queried vector and the retrieved value, respectively, both being features of the input frame. In this invention, a linear attention layer is used to improve efficiency. For the blocks of the four transformers, varying block sizes are used, from large to small, to extract feature associations at different scales.

[0086] After passing through the transformer, the refined intermediate frame features are processed by a frame synthesis module consisting of three residual densely connected blocks for feature aggregation and frame synthesis.

[0087] In a preferred embodiment, the feature refinement transformer network is illustrated using one scale as an example, see [link to example]. Figure 3 Except for the coarse scale, which does not use a transformer network, each scale consists of 4 transformer blocks.

[0088] In each transformer block, the coarse intermediate frame features and integer frame features are first divided into several non-overlapping blocks through an unfold operation;

[0089] Each transformer has a different block size. For the fine scale, the block sizes are 8, 4, 2, and 1, respectively; for the intermediate scale, the block sizes are 4, 4, 2, and 1, respectively.

[0090] The features of the blocks after unfolding are enhanced with coarse intermediate frame features through a linear attention layer, and refined intermediate frame features are obtained through a folding operation (the inverse operation of unfolding) and a 2D convolution layer.

[0091] For the linear attention layer, the coarse intermediate frame features are used as the Q vector, and the integer frame features are used as the K and V vectors. Q and K are first processed by an exponential linear unit, then K and V are multiplied together and then multiplied by Q. The resulting value is then normalized by layer normalization and concatenated with the original Q along the channel dimension. Finally, it is processed by a fully connected network, layer normalization, and residual connections to obtain the final feature.

[0092] Finally, feature aggregation and frame synthesis are performed; see [link to documentation]. Figure 4 The coarse intermediate frame features refined in the previous step are first enhanced by three ResDenseBlocks, and the outputs of the three ResDenseBlocks are then concatenated and further aggregated. The resulting features are at half resolution, so a convolutional and pixel shuffle layer is used to upsample the features to the original resolution, and finally a 2D convolution layer is used to synthesize the final fine intermediate frame.

[0093] This embodiment employs a transformer network structure for feature transfer and refinement. The transformer network uses a many-to-one matching and mapping method, which greatly improves the robustness and fault tolerance of feature matching and reduces the probability of the first-stage error accumulating in the second stage.

[0094] In a preferred embodiment of the present invention, S400 is implemented, where an objective function is set and a two-stage video frame interpolation network is trained. The overall loss function of the network is divided into two parts.

[0095] L = L OC +L O

[0096] This includes the loss function L of the coarse intermediate frame.OC And the loss function L of the final fine frame O Both of the above loss functions are implemented using L1 loss:

[0097] L(O,GT)=||O-GT||1

[0098] This involves calculating the L1 loss between the reconstructed intermediate frames and the ground truth. For reconstructed intermediate frames at different scales, the ground truth is downsampled using bilinear interpolation, and the loss value is calculated accordingly.

[0099] In a training iteration, the entire neural network is first forward-propagated, then the total loss L is calculated. Next, the weight gradients of each layer in the network are calculated using backpropagation based on the total loss L, and the weights are updated. This completes the current iteration, and the next iteration follows. After training, the corresponding two-stage video frame interpolation model is obtained.

[0100] In one specific embodiment of the present invention, given a high frame rate video, the video is frame-by-frame extracted according to the multiple of the frame to be interpolated, resulting in integer frames and true intermediate frames. Image patches are cropped and randomly flipped to construct a dataset for training, validation, and testing; specifically,

[0101] The constructed dataset is implemented using steps S100-S300 in the above embodiments, i.e., a two-stage video frame interpolation network is established. The first stage uses a pure 3D U-shaped network structure for implicit motion estimation and coarse intermediate frame estimation. The second stage uses a transformer-based intermediate frame feature refinement network. A coarse intermediate frame loss function and a final fine intermediate frame loss function are established. The model parameters are continuously optimized using the training set in the constructed dataset, ensuring good performance in both stages. During training, the model's effectiveness on the validation set is verified, and the training parameters are adjusted based on the validation results. The test set from the constructed dataset is input into the trained two-stage network to reconstruct the fine intermediate frames, and the output results are objectively and subjectively evaluated.

[0102] Based on the same inventive concept, other embodiments of the present invention provide a video frame rate enhancement system based on a multi-scale coarse-to-fine transformer network, including a feature extraction module, a first-stage module, a second-stage module, a constraint training module, and an application module. The feature extraction module extracts integer frame features from the input integer frames. The first-stage module uses a coarse intermediate frame estimation network to perform implicit motion estimation on the integer frame features to obtain coarse intermediate frames. The second-stage module uses a feature refinement transformer network to transfer the integer frame features to the position of the coarse intermediate frames for quality enhancement to obtain fine intermediate frames. The constraint training module establishes a coarse intermediate frame loss function and a fine intermediate frame loss function, and trains by adding the two loss functions. The application module inputs four consecutive low frame rate video frames into the trained first-stage network and second-stage network to reconstruct the fine intermediate frames.

[0103] This invention is implemented using Python, specifically the PyTorch deep learning framework, and employs the Adam optimizer. During training, 16 training sample pairs are used in each iteration. The initial learning rate is set to 0.0001. As the slope of the training loss decreases flattens, the learning rate is halved until it falls below 1e. -6 Training stops when the network weights are initialized. The initialization method chosen is the one for ReLU neurons. This invention is trained on the Vimeo-90k public dataset and tested on its test set and the Davis dataset.

[0104] For the evaluation of the results of this embodiment of the invention, since the dataset contains real intermediate frames, two objective quality evaluation metrics with reference are used: PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). The pixel-by-pixel difference and structural similarity between the generated intermediate frames and the real intermediate frames are calculated respectively. Higher PSNR and SSIM indicate that the generated intermediate frames are closer to the real values, and the image quality and the smoothness of the generated video are likely to be higher.

[0105] Table 1. Comparison of objective performance indicators of the embodiments of the present invention and existing frame interpolation methods on the 2x frame interpolation task.

[0106]

[0107] Table 1 shows the objective quality tests performed on the two datasets. On both datasets, the embodiments of the present invention outperform other existing methods in both metrics, indicating that the intermediate frames predicted by the proposed method are closer to the true values. It can be considered that the present invention can extract more accurate and realistic intermediate motion and can effectively synthesize intermediate frames, thereby making the synthesized high frame rate video smoother and with fewer artifacts.

[0108] Figure 5This is a subjective comparison of the effects of the embodiments of the present invention with existing methods. The first column represents the actual intermediate frames, and the second column represents the result of directly superimposing two consecutive integer frames. For the last five columns, the first row represents the predicted intermediate frames, and the second row is a heatmap of the residuals between the predicted and actual intermediate frames (the larger the residual, the redder it appears). Figure 4 It can be seen that the method proposed in this invention can be modified to accurately estimate intermediate motion, and the predicted intermediate frames have more accurate object positions and contours. At the same time, fewer artifacts are caused, all of which demonstrate that this invention has better subjective visual effects.

[0109] The method described in the above embodiments of the present invention utilizes a 3D U-shaped network to implicitly estimate the motion of intermediate frames, thereby improving the accuracy of complex motion scenes. Simultaneously, a transformer structure is employed, utilizing a many-to-one mapping method to enhance the robustness of feature transfer and improve the prediction performance of existing methods.

[0110] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for improving video frame rate based on a multi-scale coarse-to-fine converter network, characterized in that, include: A feature extraction network is used to extract integer frame features from the input integer frames; The first-stage network uses a coarse intermediate frame estimation network to perform implicit motion estimation on the features of the integer frames to obtain coarse intermediate frames; The second-stage network uses a feature refinement transformer network to migrate the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames. Establish the coarse intermediate frame loss function and the fine intermediate frame loss function, and train by adding the two loss functions together; Multiple consecutive low frame rate videos are input into the trained first-stage network and second-stage network to reconstruct fine intermediate frames; The coarse intermediate frame estimation network is a pure 3D convolutional U-shaped network consisting of three scales, each composed of several 3D convolutional layers. The three scales are fine scale, intermediate scale, and coarse scale in order of decreasing resolution. The feature refinement transformer network includes four transformer blocks with different block sizes, varying from large to small, to extract associations at different scales from coarse to fine. The first-stage network employs a coarse intermediate frame estimation network to perform implicit motion estimation on the integer frame features to obtain coarse intermediate frames, including: The integer frame features at the three scales are input into the coarse intermediate frame estimation network; Each integer-scale frame feature is processed using a three-layer network; where... The first layer is a 3D convolutional layer, used to fuse features from multiple integer frames; The second layer is a 3D convolutional layer, which is responsible for feature refinement at the fine scale, for fusing and refining the features output from the first layer at the current scale and the fine scale at the intermediate scale, and for fusing and refining the features output from the first layer at the current scale and the intermediate scale at the coarse scale. The third layer is a 3D convolutional residual block, which consists of two 3D convolutional layers connected by residuals. It is responsible for feature refinement at the coarse scale, and for fusing and refining the features output from the second layer at the current scale and the intermediate scale at the fine scale. At the intermediate scale, it is responsible for fusing and refining the features output from the second layer at the current scale and the coarse scale. The time dimension of the refined features is expanded and merged into the channel dimension, and the time dimension is removed, thus reducing the dimensionality of the features. The expanded and merged features are subjected to dimensionality reduction through three layers of 2D convolution to obtain coarse intermediate frame features, which are then synthesized into coarse intermediate frames, resulting in coarse intermediate frames at three scales. In the coarse intermediate frame estimation network, except for the last 2D convolutional layer which does not use an activation layer, each other convolutional layer uses the LeakyReLU activation function, that is, a non-zero slope is applied to negative values ​​on the basis of the linear rectified ReLU layer. The second-stage network employs a feature refinement transformer network to transfer the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames, including: The coarse intermediate frame and integer frame features at the three scales are used as inputs to the second-stage network; For each scale of coarse intermediate frame, four consecutive transformer blocks are used to migrate the integer frame features of the corresponding scale to the coarse intermediate frame position to enhance the quality of the coarse intermediate frame features. The coarse intermediate frame features, after being processed by the transformer block, are aggregated by a frame synthesis module consisting of three residual densely connected blocks to obtain a fine intermediate frame.

2. The method for improving video frame rate based on multi-scale coarse-to-fine converter networks according to claim 1, characterized in that, The feature extraction network includes three 2D convolutional layers. Each convolutional layer extracts features from the input integer frames and downsamples them to obtain integer frame features at three scales, which serve as the input to the first-stage network.

3. The method for improving video frame rate based on a multi-scale coarse-to-fine converter network according to claim 1, characterized in that, The second layer is a 3D convolutional layer, which downsamples the output features of the first layer at a fine scale and fuses them with the features at an intermediate scale; it downsamples the output features of the first layer at an intermediate scale and fuses them with the features at a coarse scale; the downsampling of the features is accomplished by additional 3D convolution.

4. The method for improving video frame rate based on multi-scale coarse-to-fine converter networks according to claim 1, characterized in that, The third layer is a 3D convolutional residual block, which upsamples the output features of the coarse-scale second layer and fuses them with intermediate-scale features; and upsamples the output features of the intermediate-scale second layer and fuses them with fine-scale features. The upsampling of the features is accomplished by additional 3D deconvolution.

5. The method for improving video frame rate based on a multi-scale coarse-to-fine converter network according to claim 1, characterized in that, The process performed in each of the converter blocks includes: The image features of coarse intermediate frames and integer frames are divided into blocks to obtain several image blocks; Using a self-attention mechanism, the association between the image patches is estimated and matched. Specifically, image patches with coarse intermediate frame features are used as references to perform feature matching with image patches divided by integer frame features to generate weights. The weighted sum of the features of integer frames is calculated based on the weights and used as the refined features for the current position of the coarse intermediate frame, i.e. ,in, The vector to be queried refers to the coarse intermediate frame features; and Both the queried vector and the retrieved value are features of the input integer frame.

6. The method for improving video frame rate based on multi-scale coarse-to-fine converter networks according to claim 1, characterized in that, Establish a comprehensive loss function consisting of the coarse intermediate frame loss function and the fine intermediate frame loss function: ； This includes a loss function for coarse intermediate frames. and the loss function for fine intermediate frames Both loss functions adopt Loss realization: ； That is, calculating the difference between the reconstructed intermediate frames and the true values. Losses, among which For real intermediate frames, The reconstructed intermediate frames output by the network. In For coarse intermediate frames, In For fine intermediate frames; For reconstructed intermediate frames at different scales, the true values ​​are downsampled using bilinear interpolation to calculate the loss value.

7. A video frame rate enhancement system based on a multi-scale coarse-to-fine converter network, characterized in that, include: The feature extraction module extracts integer frame features from the input integer frames; The first stage module uses a coarse intermediate frame estimation network to perform implicit motion estimation on the features of the integer frames to obtain coarse intermediate frames. The second-stage module uses a feature refinement transformer network to migrate the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames. The constraint training module establishes the coarse intermediate frame loss function and the fine intermediate frame loss function, and trains by adding the two loss functions together. The application module inputs multiple consecutive low frame rate video frames into the trained coarse intermediate frame estimation network and the feature refinement transformer network to reconstruct fine intermediate frames. The coarse intermediate frame estimation network is a pure 3D convolutional U-shaped network consisting of three scales, each composed of several 3D convolutional layers. The three scales are fine scale, intermediate scale, and coarse scale in order of decreasing resolution. The feature refinement transformer network includes four transformer blocks with different block sizes, varying from large to small, to extract associations at different scales from coarse to fine. The first-stage module uses a coarse intermediate frame estimation network to perform implicit motion estimation on the integer frame features to obtain coarse intermediate frames, including: The integer frame features at the three scales are input into the coarse intermediate frame estimation network; Each integer-scale frame feature is processed using a three-layer network; where... The first layer is a 3D convolutional layer, used to fuse features from multiple integer frames; The second layer is a 3D convolutional layer, which is responsible for feature refinement at the fine scale, for fusing and refining the features output from the first layer at the current scale and the fine scale at the intermediate scale, and for fusing and refining the features output from the first layer at the current scale and the intermediate scale at the coarse scale. The third layer is a 3D convolutional residual block, which consists of two 3D convolutional layers connected by residuals. It is responsible for feature refinement at the coarse scale, and for fusing and refining the features output from the second layer at the current scale and the intermediate scale at the fine scale. At the intermediate scale, it is responsible for fusing and refining the features output from the second layer at the current scale and the coarse scale. The time dimension of the refined features is expanded and merged into the channel dimension, and the time dimension is removed, thus reducing the dimensionality of the features. The expanded and merged features are subjected to dimensionality reduction through three layers of 2D convolution to obtain coarse intermediate frame features, which are then synthesized into coarse intermediate frames, resulting in coarse intermediate frames at three scales. In the coarse intermediate frame estimation network, except for the last 2D convolutional layer which does not use an activation layer, each other convolutional layer uses the LeakyReLU activation function, that is, a non-zero slope is applied to negative values ​​on the basis of the linear rectified ReLU layer. The second-stage module employs a feature refinement transformer network to migrate the features of the integer frames to the positions of the coarse intermediate frames for quality enhancement, thereby obtaining fine intermediate frames, including: The coarse intermediate frame and integer frame features at the three scales are used as inputs to the feature refinement transformer network; For each scale of coarse intermediate frame, four consecutive transformer blocks are used to migrate the integer frame features of the corresponding scale to the coarse intermediate frame position to enhance the quality of the coarse intermediate frame features. The coarse intermediate frame features, after being processed by the transformer block, are aggregated by a frame synthesis module consisting of three residual densely connected blocks to obtain a fine intermediate frame.

Images