Video super-resolution processing method based on multi-scale bidirectional optical flow connection

Abstract

The invention relates to the technical field of video processing, in particular to a video super-resolution processing method based on multi-scale bidirectional optical flow connection, which is used for processing an input video by integrating a head-tail convolution module, a multi-channel attention mechanism, a feature extraction neural network, a bidirectional optical flow algorithm and a data set amplification algorithm. Firstly, an input feature vector is processed by using a head-tail convolution module, secondly, contribution degrees of different input sources are dynamically evaluated and adjusted by introducing a multi-channel attention mechanism, and unimportant background noise is filtered out; carrying out multi-scale feature fusion through a feature extraction network, and extracting key information; and finally, introducing a bidirectional optical flow algorithm, and improving the quality and time coherence of video super-resolution reconstruction by using time correlation. According to the method, high-quality video reconstruction is realized while the calculation efficiency is ensured.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video super-resolution processing method based on multi-scale bidirectional optical flow connection. Background Art

[0002] With the rapid development of Internet technology, video content has become the core carrier of modern digital entertainment and information dissemination. However, the transmission and storage of high-resolution video are limited by network bandwidth and terminal device performance, especially in mobile communication network environments.

[0003] Although traditional video compression technology and super-resolution methods can alleviate this contradiction to a certain extent, they often sacrifice video quality or increase computational complexity. In recent years, the successful application of deep learning technology, especially convolutional neural network (CNN) in the field of image processing, has provided a new solution for video super-resolution. However, most of the existing video super-resolution technologies fail to fully utilize the temporal correlation between video frames, resulting in the need to improve the detail clarity and temporal coherence of the reconstructed video. Summary of the invention

[0004] The purpose of the present invention is to provide a video super-resolution processing method based on multi-scale bidirectional optical flow connection, aiming to solve the problems of insufficient detail clarity and temporal coherence in existing video processing technology, and to improve the transmission efficiency and visual experience quality of video under low bandwidth conditions.

[0005] To achieve the above object, the present invention provides a video super-resolution processing method based on multi-scale bidirectional optical flow connection, comprising the following steps:

[0006] Step 1: Input the video frame and process the input feature vector using the head-to-tail convolution module. Only the feature vectors at the two ends are convolved, and the features in the middle remain unchanged.

[0007] Step 2: Input the processed feature vector into the multi-input attention mechanism module to perform weighted processing on the high-density information areas in the video frame;

[0008] Step 3: Use the feature extraction network to perform multi-scale feature fusion on the video frames and extract key information;

[0009] Step 4: Using the bidirectional optical flow calculation algorithm, the accuracy of motion estimation is enhanced through bidirectional consistency check of forward and backward optical flows, and the effect of video super-resolution reconstruction is improved by using temporal correlation.

[0010] Optionally, during the execution of step 1, the input feature vector is assumed to be X = [x1, x2, ..., x N], N is the length of the feature vector, and the input feature vector dataset is divided into three subsets: head feature X head =[x1,x2,...,x α ], tail feature X tail =[x N - β+1 , ..., x N-1 , x N ], and the intermediate feature X mid =[x α+1 ,...,x N-β ], where α and β are the lengths of the head and tail features, respectively;

[0011] For the head and tail features, apply the convolution operation: X head ′=f conv (X head ), X tail ′=f conv (X tail ), while for the intermediate features, it remains unchanged: X mid ′=X mid The final output feature vector X′ is composed of these three parts: X′ = [X head ′,X mid ′,X tail ′].

[0012] Optionally, during the execution of step 2, the multi-input attention mechanism module introduces the softmax function as the core operation unit. Each input channel is first converted into a score representing its potential value, and the score is then input into the softmax function to generate a set of probability values. These probability values ​​constitute a weight vector whose sum is 1.

[0013] Optionally, the feature extraction network includes an encoder, a decoder, an improved ResNet structure and an attention mechanism, wherein the feature extraction network includes multiple levels of encoders and decoders, which are respectively responsible for capturing multi-scale features; each encoder module includes a series of downsampling layers for reducing the spatial dimension of the input data; each decoder module uses a series of upsampling layers to restore the compressed feature map to the original spatial size; the improved ResNet structure is used to enhance the information flow in the feature extraction process, and the attention mechanism is responsible for dynamically adjusting the weight distribution of the feature map.

[0014] Optionally, the bidirectional optical flow calculation algorithm calculates the forward optical flow With backward optical flow That is, the pixel displacements from time point t to t+1 and from t+1 back to t are used to construct a comprehensive and coherent motion vector field.

[0015] Optionally, in step 4, in the process of improving the video super-resolution reconstruction by using the temporal correlation, a model based on spatiotemporal interpolation is proposed. The model based on spatiotemporal interpolation combines the current frame I t and the next frame I t+1 The high-resolution frame IHR is reconstructed using the following formula:

[0016]

[0017] Among them, the weight factor By pixel The gradient information in both frames is dynamically calculated, ensuring a balance between smoothness and detail preservation in different regions.

[0018] The present invention provides a video super-resolution processing method based on multi-scale bidirectional optical flow connection, which integrates head-to-tail convolution module, multi-channel attention mechanism, feature extraction neural network, bidirectional optical flow algorithm and data set augmentation algorithm to process input video. Specifically, the head-to-tail convolution module is first used to process the input feature vector, and then the multi-channel attention mechanism is introduced to dynamically evaluate and adjust the contribution of different input sources to filter out unimportant background noise; then multi-scale feature fusion is performed through the feature extraction network to extract key information; finally, the bidirectional optical flow algorithm is introduced to improve the quality and temporal coherence of video super-resolution reconstruction by using time correlation. The present invention achieves high-quality video reconstruction while ensuring computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic flow chart of the steps of a video super-resolution processing method based on multi-scale bidirectional optical flow connection of the present invention.

[0021] Figure 2 It is a schematic diagram of the implementation of the head-to-tail convolution of the present invention.

[0022] Figure 3 It is a schematic diagram of the implementation process of the head-to-tail convolution module of the present invention.

[0023] Figure 4 It is a schematic diagram of the implementation process of the multi-path attention mechanism of the present invention.

[0024] Figure 5It is a schematic diagram of the structural principle of the feature extraction neural network of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0026] See also Figure 1 The present invention provides a video super-resolution processing method based on multi-scale bidirectional optical flow connection, comprising the following steps:

[0027] S1: Input video frame, use the head-tail convolution module to process the input feature vector, only perform convolution operation on the feature vectors at the two ends, and keep the features in the middle unchanged;

[0028] S2: Input the processed feature vector to the multi-input attention mechanism module to perform weighted processing on the high-density information area in the video frame;

[0029] S3: Multi-scale feature fusion of video frames is performed through the feature extraction network to extract key information;

[0030] S4: Using the bidirectional optical flow calculation algorithm, the accuracy of motion estimation is enhanced through bidirectional consistency check of forward and backward optical flows, and the effect of video super-resolution reconstruction is improved by using temporal correlation.

[0031] The present invention integrates head-to-tail convolution module, multi-channel attention mechanism, feature extraction neural network, bidirectional optical flow algorithm and data set augmentation algorithm to process input video, please refer to Figures 2 to 5 , the following is a further explanation of the relevant terms and execution steps:

[0032] In step S1, the head-to-tail convolution module optimizes the allocation of computing resources and reduces redundant calculations by intelligently processing the two ends of the input feature vector. The core idea of ​​the head-to-tail convolution module is to optimize the allocation of computing resources to achieve the goal of significantly reducing the computational complexity without significantly sacrificing the accuracy of the model. This mechanism is achieved by intelligently dividing the input feature vector into three regions: the head, the middle, and the tail, and applying different processing strategies in a targeted manner. In the head-to-tail convolution module, the head and tail of the input feature vector undergo a series of carefully designed convolution operations. These convolutional layers are responsible for capturing the boundary information of the input data, including edge details, texture patterns, and possible local features, which are often key clues in tasks such as image classification and object detection. In this way, the head-to-tail convolution can efficiently extract and utilize this boundary information without having to perform redundant convolution operations on the entire feature map. On the contrary, for the middle part of the feature vector, the head-to-tail convolution chooses to retain its original state, avoiding unnecessary computational overhead. The basis of this strategy is that, in many cases, the features in the middle region tend to have strong translation invariance, that is, they perform similarly at different positions, so directly using the original features can effectively save computing resources while ensuring that the overall performance of the model is not affected.

[0033] Assume that the input feature vector is X = [x1, x2, ..., x N ], where N is the length of the feature vector. First, it is divided into three subsets: head feature X head =[x1,x2,...,x α ], tail feature X tail =[x N-β+1 , ..., x N-1 , x N ], and the intermediate feature X mid =[x α+1 , ..., x N-β ], where α and β are the lengths of the head and tail features, respectively. For the head and tail features, the present invention applies a convolution operation: X head ′=f conv (X head ), X tail ′=f conv (X tail ), while for the intermediate features, it remains unchanged: X mid ′=X mid The final output feature vector X′ is composed of these three parts: X′ = [X head ′,X mid ′,X tail′]. This design not only reflects the pursuit of a balance between computational efficiency and model performance in the field of deep learning, but also demonstrates how to improve the practicality of the model through clever architectural design, especially in resource-constrained or real-time processing scenarios.

[0034] The core of the multi-channel input attention mechanism in step S2 is the ability to dynamically evaluate and adjust the contribution of different input sources, ensuring that the model can focus on the most relevant information while filtering out unimportant background noise. This mechanism effectively implements the weight allocation and importance quantification of multi-source inputs by introducing the softmax function as the core operation unit. In multi-channel input attention, each input channel is first converted into a score representing its potential value, which reflects the relative importance of the channel information to the current task. These scores are then input into the softmax function to generate a set of probability values, which constitute a weight vector with a sum of 1. This weight vector intuitively represents the contribution of each input channel to the final decision, allowing the model to automatically adjust its attention focus according to the importance of the input.

[0035] Specifically, suppose there are C input channels in the present invention, and the preliminary score of each channel can be obtained by a learnable parameter matrix W∈R D×C and a bias vector b∈R C Calculated, where D is the dimension of each input channel. For the input feature vector X∈R D×C , the score s for each channel i It can be expressed as: i =W·x i +b i for i = 1, 2, ..., C; Next, these scores s = [s1, s2, ..., s C ] is input into the softmax function to obtain the weight of each channel The weight vector α obtained in this way is [α1, α2, ..., α C ] will be used to reweight each input channel, ensuring that the model can more effectively utilize key information while ignoring or reducing attention to irrelevant information. The final weighted input X′ can be calculated by the following formula: i ′=α i ·X i for i = 1, 2, ..., C. This design not only improves the flexibility and robustness of the model in processing multivariate information, but also provides a clear path for understanding model decisions, which helps to enhance the interpretability of the model.

[0036] The feature extraction network in step S3 includes the following parts: the encoder and decoder are used to capture multi-scale features, the improved ResNet structure provides powerful representation capabilities, and the attention mechanism dynamically adjusts the weight distribution of feature maps to enhance the extraction of key information of video frames.

[0037] The encoder-decoder architecture uses multiple levels of encoder and decoder modules, which are responsible for capturing multi-scale features. Each encoder module contains a series of downsampling layers (DownSample) to reduce the spatial dimension of the input data so that a large amount of information can be processed more efficiently. Similarly, the decoder module uses a series of upsampling layers (UpSample) to restore the compressed feature map to the original spatial size to obtain a refined output result. This step-by-step transmission of contextual information helps the network better understand the complex scene structure. Secondly, the present invention introduces a modified ResNet structure, which is an optimized residual learning framework that can effectively solve the gradient vanishing problem in deep neural networks. By connecting the input and output of each layer, ResNet allows information to directly skip some intermediate layers, thereby reducing the difficulty of training and improving the convergence speed. The present invention has made improvements on this basis, so that ResNet can better adapt to the needs of video processing tasks. Specifically, the present invention embeds HT blocks into the network to enhance the information flow in the feature extraction process. For the HT block of the i-th layer, its input x i The output y is obtained by the following transformation i =g(x i ,W g )*x i +f(x i ,W f )+b, where g and f are two learnable linear transformation functions, W g and W f is the corresponding weight matrix, and b is the bias term. Finally, the present invention applies an attention mechanism to dynamically adjust the weight distribution of feature maps.

[0038] The attention mechanism is an effective tool that can focus on important areas in a video frame and ignore irrelevant information. In this paper, an additional attention module (marked by a red rectangle in the figure) is used, which receives feature maps from the encoder and decoder and generates an attention weight vector A. This vector is then used to reweight the feature map to emphasize important areas. Specifically, given a feature map F∈R C×H×W , attention weight A∈R H×W , the feature map F′ can be updated by the following formula: in, Represents an element-wise multiplication operation.

[0039] The core idea of ​​the bidirectional optical flow calculation algorithm in step S4 is to calculate the forward optical flow With backward optical flow That is, the pixel displacement from time point t to t+1 and from t+1 back to t respectively constructs a comprehensive and coherent motion vector field. In order to quantify the consistency of bidirectional optical flow, the present invention defines a consistency error index E Consistency , which can help identify and exclude inconsistent optical flow estimates caused by noise, occlusion, or non-rigid motion. The consistency error can be calculated by the following formula: Among them, x → represents the pixel coordinates, and ||·|| refers to the Euclidean norm of the vector. By setting a threshold τ, any pixel exceeding this threshold The values ​​will be marked as unreliable and thus excluded from the algorithm to ensure that the final motion estimation has higher reliability.

[0040] Furthermore, the present invention proposes a model based on spatiotemporal interpolation and uses information fusion of the current frame and the next frame to improve the reconstruction effect. The specific implementation steps and formulas are described below.

[0041] The implementation steps and formulas are as follows:

[0042] 1. Input frame acquisition: input current frame I t and the next frame I t+1 , and use these two frames as the basis for high-resolution frame reconstruction.

[0043] 2. Motion estimation and vector calculation: Obtain the pixel motion vector between the current frame and the next frame through optical flow or other motion estimation algorithms This step ensures that pixel information between different frames can be accurately aligned, which is conducive to improving detail quality by utilizing temporal correlation.

[0044] 3. Weight factor calculation: based on each pixel position in the current frame and the next frame Gradient information of the dynamic calculation weight factor The weight factors are defined as follows:

[0045]

[0046] in, Indicates the gradient magnitude at the current pixel position, which is used to measure the degree of local change. This factor can give the current frame a higher weight in detail-rich areas and tend to smooth interpolation in smooth areas.

[0047] 4. High-resolution frame reconstruction: Apply the weight factor to the interpolation formula to obtain a high-resolution frame by fusing the current frame and the next frame information after motion compensation.

[0048]

[0049] This formulation ensures that motion in the video is adapted while preserving details, and further exploits temporal correlations for reconstruction, resulting in high-resolution frames with higher visual quality.

[0050] The model can dynamically balance details and smoothness, ensuring adaptive interpolation in fast-moving or static areas. At the same time, by using weight factors, motion blur and artifacts can be reduced, achieving super-resolution video reconstruction with better visual effects.

[0051] In addition, in the video super-resolution task, in order to improve the generalization ability of the model and enhance the diversity of training data, the present invention has multiple methods for expanding the super-resolution video dataset. Through a variety of data expansion techniques, the robustness of the model in different environments and conditions is improved, and the super-resolution reconstruction quality of the model is significantly improved.

[0052] 1. Spatial transformation method

[0053] Video frame flipping: This method generates new samples by flipping video frames horizontally or vertically to increase the spatial diversity of training data. In this way, the model can adapt to more diverse spatial structures and maintain sensitivity to image features.

[0054] Frame rotation: Rotate the video frame at a fixed angle (such as 90 degrees, 180 degrees, or 270 degrees) to generate frame samples of different perspectives to enhance the expressiveness of the model under different rotation conditions.

[0055] Frame Scaling: Scaling video frames to simulate changes in perspective and object distance in different scenes, thereby expanding the spatial characteristics of the dataset and improving the adaptability of the model.

[0056] 2. Noise perturbation method

[0057] Gaussian noise injection: Gaussian noise is added to the video frames to simulate the random noise that may occur in the real world. The model trained by this method can maintain stable super-resolution performance in a noisy environment.

[0058] Motion blur enhancement: Introducing a motion-simulated blur effect into the video frames. By simulating blurred images produced by camera motion or rapid object motion, the model is more robust to blurry video scenes.

[0059] 3. Color perturbation method

[0060] Brightness adjustment: Generate training samples under different lighting conditions by randomly increasing or decreasing the brightness of video frames, thereby improving the performance of the model in a variety of lighting environments.

[0061] Contrast adjustment: Adjust the contrast of the video frame to generate sample data with varying contrast. This method enables the model to accurately perform super-resolution reconstruction even when the image clarity changes.

[0062] Color jittering: Introducing random changes in parameters such as hue and saturation in video frames to expand the color diversity of the dataset so that the model has stable performance in different color environments.

[0063] 4. Geometric Transformation Method

[0064] Random cropping: Randomly crop a sub-region in each video frame as a training sample, so that the model pays more attention to the restoration of local details and the detection of small targets, improving the super-resolution effect.

[0065] Affine transformation: including transformation operations such as translation, scaling, rotation and distortion, simulating scene changes from different perspectives so that the model can adapt to multi-angle video images.

[0066] Finally, the output of the neural network is combined with the bicubic interpolation algorithm to obtain the final super-resolution video. Bicubic interpolation uses 16 pixels (4x4 area) around the target pixel for interpolation calculation. The algorithm predicts the pixel value of the target position by constructing a cubic polynomial and considering the brightness or color values ​​of neighboring pixels and their changing trends.

[0067] In the present invention, the output of the neural network and the bicubic interpolation are fused to obtain a super-resolution video, so that the image contains both the high-frequency information generated by the network and the smoothness brought by the interpolation method, thereby obtaining a super-resolution result with a stronger sense of naturalness.

[0068] The output features of the neural network are combined with the bicubic interpolation algorithm, which uses the advanced features of the neural network to enhance image details. At the same time, the image is smoothed with the help of the bicubic interpolation algorithm, ultimately producing clear and delicate super-resolution video, significantly improving the visual quality and viewing experience of the video.

[0069] The multi-channel attention mechanism dynamically evaluates the contribution of different input sources to ensure that the model focuses on key information. The feature extraction neural network combines the encoder and decoder, as well as the improved ResNet structure, to effectively capture multi-scale features. The bidirectional optical flow algorithm uses the consistency of forward and backward optical flow to enhance motion estimation accuracy and improve temporal coherence. The data set augmentation algorithm increases the diversity of training samples and improves the generalization ability of the model.

[0070] In summary, the present invention achieves a significant improvement in the accuracy of motion estimation through the deep integration of bidirectional optical flow consistency check and temporal correlation, while significantly enhancing the visual effect of video super-resolution reconstruction, providing strong support for the development of high-quality video processing technology.

[0071] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A video super-resolution processing method based on multi-scale bidirectional optical flow connection, characterized in that: The following steps are involved: Step 1: Input the video frame and process the input feature vector using the head-to-tail convolution module. Only the feature vectors at the two ends are convolved, and the features in the middle remain unchanged. Step 2: Input the processed feature vector into the multi-input attention mechanism module to perform weighted processing on the high-density information areas in the video frame; Step 3: Use the feature extraction network to perform multi-scale feature fusion on the video frames and extract key information; Step 4: Using the bidirectional optical flow calculation algorithm, the accuracy of motion estimation is enhanced through bidirectional consistency check of forward and backward optical flows, and the effect of video super-resolution reconstruction is improved by using temporal correlation.

2. The video super-resolution processing method based on multi-scale bidirectional optical flow connection according to claim 1, characterized in that: In the execution process of step 1, let the input feature vector be X = [x1, x2, ..., x N ], N is the length of the feature vector, and the input feature vector dataset is divided into three subsets: head feature X head =[x1,x2,...,x α ], tail feature X tail =[x N-β+1 ,...,x N-1 ,x N ], and the intermediate feature X mid =[x α+1 ,...,x N-β ], where α and β are the lengths of the head and tail features, respectively; For the head and tail features, apply the convolution operation: X head ′=f conv (X head ), X tail ′=f conv (X tail ), while for the intermediate features, it remains unchanged: X mid ′=X mid The final output feature vector X′ is composed of these three parts: X′ = [X head ′,X mid ′,X tail ′].

3. The video super-resolution processing method based on multi-scale bidirectional optical flow connection according to claim 2, characterized in that: During the execution of step 2, the multi-input attention mechanism module introduces the softmax function as the core operation unit. Each input channel is first converted into a score representing its potential value, and the score is then input into the softmax function to generate a set of probability values. These probability values ​​constitute a weight vector whose sum is 1.

4. The video super-resolution processing method based on multi-scale bidirectional optical flow connection according to claim 3, characterized in that: The feature extraction network includes an encoder, a decoder, an improved ResNet structure and an attention mechanism. The feature extraction network contains multiple levels of encoders and decoders, which are responsible for capturing multi-scale features respectively; each encoder module contains a series of downsampling layers for reducing the spatial dimension of the input data; each decoder module uses a series of upsampling layers to restore the compressed feature map to the original spatial size; the improved ResNet structure is used to enhance the information flow in the feature extraction process, and the attention mechanism is responsible for dynamically adjusting the weight distribution of the feature map.

5. The video super-resolution processing method based on multi-scale bidirectional optical flow connection according to claim 4, characterized in that: The bidirectional optical flow calculation algorithm calculates the forward optical flow With backward optical flow That is, the pixel displacements from time point t to t+1 and from t+1 back to t are used to construct a comprehensive and coherent motion vector field.

6. The video super-resolution processing method based on multi-scale bidirectional optical flow connection according to claim 5, characterized in that: In step 4, in the process of using temporal correlation to improve video super-resolution reconstruction, a model based on spatiotemporal interpolation is proposed. The model based on spatiotemporal interpolation combines the current frame I t and the next frame I t+1 The high-resolution frame IHR is reconstructed using the following formula: Among them, the weight factor By pixel The gradient information in both frames is dynamically calculated, ensuring a balance between smoothness and detail preservation in different regions.

Images