A video super-resolution method based on bidirectional multi-scale deformable attention network

Abstract

The application discloses a video super-resolution method based on a bidirectional multi-scale deformable attention network, which comprises the following steps: given a low-resolution input frame sequence, shallow features are obtained by using a feature extraction block composed of residual blocks, the features are sent to a DAM alignment module through back propagation, coarse alignment features are output, the shallow features and the coarse alignment features are spliced, the spliced features are sent to a residual block stack for processing to generate fusion features, two adjacent fusion features are input into the DAM through forward propagation to generate fine alignment features, the fine alignment features, the fusion features and the shallow features are fused, and the aggregated features are generated through residual blocks, the aggregated features are recombined through pixels and added to the low-resolution frame after up-sampling, and a high-resolution frame image is output. The deformable alignment module can effectively align and fuse the features by using multi-scale deformable convolution and a multi-scale attention mechanism, and excellent performance can be achieved on large-scale motion videos.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a video super-resolution method based on a bidirectional multi-scale deformable attention network. Background Technology

[0002] Super-resolution (SR) aims to generate high-resolution (HR) images with corresponding low-resolution (LR) images. SR is widely used in medical imaging, video surveillance, aerospace, and many other fields. In recent years, with the popularization of ultra-high-definition display technology, the demand for efficiently converting LR video to HR video has been very strong. Although the well-developed single-image super-resolution (SISR) technology can be directly used to improve the spatial resolution of video frame by frame, its performance has remained unsatisfactory due to the introduction of artifacts and jamming.

[0003] This phenomenon arises from the neglect of temporal dependencies and unused information from adjacent frames. Therefore, video super-resolution (VSR) technology has received considerable attention in recent years. Unlike SISR, in spatial resolution enhancement tasks, it focuses only on the key intrinsic characteristics of a single image, primarily generating HR video frames from its LR video sequences. The key to VSR is how to fully utilize adjacent, highly correlated but potentially misaligned support coordinate systems to reconstruct the reference frame;

[0004] Currently, VSR remains a challenging task because it requires effectively collecting the complementary information needed between multiple misaligned video frames to improve VSR performance. The main challenge of VSR lies in designing accurate alignment and efficient fusion modules for videos with large motion, severe blur, and occlusion. To obtain the required high-quality HR video sequences, alignment methods are used to establish accurate spatial correspondences between frames, and fusion strategies are employed to combine the features of aligned frames for super-resolution reconstruction. For alignment tasks, most early VSR methods estimated the optical flow between the LR reference frame and its neighboring reference frames, achieving explicit alignment of spatial distortion. However, this method is suitable for slow-motion videos but lacks robustness to occluded videos because inaccurate flow estimation degrades reconstruction performance.

[0005] Compared to optical flow-based image-level alignment methods, Deformation Convolution Networks (DCNs) offer an implicit feature-level alignment approach. TDAN first uses deformable convolutions to align features between adjacent frames. Then, EDVR builds upon TDAN by employing coarse-to-fine deformations. Due to the diversity of offsets, deformable alignment is more advantageous than optical flow-based alignment in VSRs. However, deformable convolution alignment is prone to network divergence during training. In recent years, attention mechanisms have been widely applied in the feature fusion stage. For example, Hu et al. obtained the weight coefficients of each channel by learning the nonlinear mapping relationship between them. Woo et al. achieved better results by combining spatial attention and channel attention.

[0006] Zhang et al. proposed the Pyramid Squeeze Attention module, enhancing the model's multi-scale feature representation capabilities. Dosovitskiy et al. achieved better performance in image classification tasks using a self-attention mechanism. In recent years, Transformer has demonstrated powerful long-range relation modeling capabilities in natural language processing and has been increasingly applied to computer vision. This algorithm has achieved good performance in image recognition, object detection, and SR. Compared with convolutional neural networks (CNNs), transformer-based models have a larger receptive field and the ability to model long-term dependencies. The designed dot-product attention structure can extend the receptive field to the entire input. However, modeling global dependencies based on large inputs remains a challenging problem under limited computational resources. The computational complexity of attention modules and the quadratic storage overhead of the input size limit their application in VSR with long sequences and large inputs.

[0007] To address this, a video super-resolution method based on a bidirectional multi-scale deformable attention network is proposed. Summary of the Invention

[0008] The purpose of this invention is to provide a video super-resolution method based on a bidirectional multi-scale deformable attention network to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a video super-resolution method based on a bidirectional multi-scale deformable attention network, specifically including the following steps:

[0010] Step 1: Given a low-resolution input frame sequence, use feature extraction blocks composed of residual blocks to obtain shallow features;

[0011] Step 2: Send these features to the DAM alignment module through backpropagation, calculate and output the coarse alignment features using the formula;

[0012] Step 3: The shallow features and the coarse alignment features are concatenated and fed into the residual block stack for processing to generate fused features;

[0013] Step 4: Using the DAM module, two adjacent fused features are fed into the DAM module through forward propagation, and fine-aligned features are generated by calculating using a formula;

[0014] Step 5: Fuse the fine alignment features, the fused features, and the shallow features, and generate clustered features using the calculation formula of the residual block;

[0015] Step 6: The clustered features are recombined into pixels and added to the upsampled low-resolution frame. The high-resolution frame image is then calculated using a formula.

[0016] Preferably, in step one, a low-resolution input frame sequence is given, which can be denoted as X = {x1, x2, ..., x...} t-1 ,x t ,x t+1 ,…,x T}, where x i ∈R H×W×C (i = 1, 2, ..., T) represents the t-th low-resolution video frame; T, H, W, and C represent the length, height, width, and channels of the video, respectively.

[0017] Preferably, the feature extraction block composed of residual blocks in step one is from x t-N ,…,x t ,…,x t+N 2N+1 features are extracted, and the calculation formula for the extraction is as follows:

[0018] F t+i =F extra (x t+i ), i = -N, ... N

[0019] Wherein: F extra F represents the residual block used to extract shallow features. t+i ∈R H×W×C This represents the shallow features obtained, and C represents the number of channels.

[0020] Preferably, the calculation formula for the output coarse alignment feature in step two is as follows:

[0021]

[0022] In the formula: This indicates a deformable alignment module with backpropagation. This indicates the output coarse alignment feature;

[0023] The DAM module is a deformable alignment module, which includes two types of modules: a multi-scale deformable convolution module (MDCM) and a multi-scale attention module (MAM).

[0024] Preferably, the specific calculation formula for generating the fusion feature by processing the residual block stack in step three is as follows:

[0025]

[0026] In the formula: This represents the fusion feature corresponding to the t+i adjacent frames, [·] represents the concatenation operation, and L Res This represents a residual block.

[0027] Preferably, in step four, obtaining the fine alignment features requires the use of a DAM module to perform forward propagation between the fused features to generate fine alignment features, the calculation formula of which is shown below:

[0028]

[0029] In the formula: This represents the fine alignment features extracted during forward propagation. This indicates a deformable alignment module for forward propagation.

[0030] Preferably, in step five, after obtaining the fine alignment features, the fine alignment features, fused features, and shallow features are fused together, and clustered features are generated through residual blocks. The calculation formula for generating clustered features is as follows:

[0031]

[0032] In the formula: L represents the clustered features generated through residual blocks. Res Represents the residual block. Indicates fine alignment features. F represents the fusion feature. t+i This indicates shallow features.

[0033] Preferably, the calculation formula for analyzing and outputting the high-resolution frame image in step six is ​​as follows:

[0034]

[0035] In the formula: B up Indicates a bicubic upsampling operation, L pc It represents a block composed of convolutions and pixel recombination. This indicates that a high-resolution frame image is output.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] 1. In this invention, the bidirectional multi-scale deformable attention network is more suitable for complex large-scale motion. The designed deformable alignment module can effectively align and fuse features by using multi-scale deformable convolution and multi-scale attention mechanism. Compared with the sliding window method, the bidirectional propagation mechanism uses multi-frame information aggregation and sequence-to-sequence reconstruction to achieve better propagation effect, and can achieve excellent performance on large-scale motion videos.

[0038] 2. In this invention, multi-scale deformable convolution alignment and multi-scale attention together can improve alignment accuracy. Specifically, our multi-scale attention strategy can effectively capture global non-local self-similarity in the space to supplement lost details.

[0039] 3. In this invention, during the multi-scale deformable convolution process, complementary information can be provided for features from different receptive fields to help align the features of each adjacent frame with the features of the reference frame. This multi-scale deformable convolution can align the reference frame and adjacent frames at the feature level without explicit motion estimation, effectively extracting features from a wide range of spatial locations and avoiding divergence during network training.

[0040] 4. In this invention, the multi-scale attention module can take into account the self-similarity at different positions within the frame, aggregate features from local and global perspectives, and effectively utilize intra-frame similar points. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the steps of the present invention;

[0042] Figure 2 This is an overall block diagram of the video super-resolution method based on a bidirectional multi-scale deformable attention network according to the present invention.

[0043] Figure 3 This is a structural diagram of the deformable alignment module of the present invention;

[0044] Figure 4 This is a structural diagram of the multi-scale deformable convolution module of the present invention;

[0045] Figure 5 This is a structural diagram of the multi-scale attention module of the present invention;

[0046] Figure 6 This is a graph showing the experimental comparison results of different processing modules based on the Vimeo-90K dataset in this invention;

[0047] Figure 7 This figure shows the experimental comparison results of different processing modules based on the REDS4 dataset in this invention.

[0048] Figure 8This is a graph showing the data results of a quantitative comparison of different reconstruction methods of the present invention on the Vimeo-90K dataset;

[0049] Figure 9 This is a graph showing the data results of a quantitative comparison of different reconstruction methods of the present invention on the REDS4 dataset. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1:

[0052] Please see Figures 1 to 5 A video super-resolution method based on a bidirectional multi-scale deformable attention network specifically includes the following steps:

[0053] A video super-resolution method based on a bidirectional multi-scale deformable attention network specifically includes the following steps:

[0054] Step 1: Given a low-resolution input frame sequence, use feature extraction blocks composed of residual blocks to obtain shallow features;

[0055] Step 2: Send these features to the DAM alignment module through backpropagation, calculate and output the coarse alignment features using the formula;

[0056] Step 3: The shallow features and the coarse alignment features are concatenated and fed into the residual block stack for processing to generate fused features;

[0057] Step 4: Using the DAM module, two adjacent fused features are fed into the DAM module through forward propagation, and fine-aligned features are generated by calculating using a formula;

[0058] Step 5: Fuse the fine alignment features, the fused features, and the shallow features, and generate clustered features using the calculation formula of the residual block;

[0059] Step 6: The clustered features are recombined into pixels and added to the upsampled low-resolution frame. The high-resolution frame image is then calculated using a formula.

[0060] During operation, low-resolution consecutive frames are first fed into a feature extractor to generate shallow features. Then, two adjacent shallow features are backpropagated and input into the DAM (Digital Aspect Mechanism) to output coarse alignment features. These features are then fed into a residual block along with the concatenated shallow features to generate fused features. Next, two adjacent fused features are forward propagated into the DAM to generate fine alignment features. These three types of features are input into the residual block to obtain aggregated features. Finally, the aggregated features are pixel-wise recombined and added to the upsampled low-resolution frames to obtain the final high-resolution frames.

[0061] Example 2:

[0062] Please see Figure 2-5 This embodiment provides a technical solution based on Embodiment 1: In step one, a low-resolution input frame sequence is given, which can be denoted as X = {x1, x2, ..., x...} t-1 ,x t ,x t+1 ,…,x T}, where x i ∈R H×W×C (i = 1, 2, ..., T) represents the t-th low-resolution video frame; T, H, W, and C represent the length, height, width, and channels of the video, respectively.

[0063] The feature extraction block composed of residual blocks in step one is from x t-N ,…,x t ,…,x t+N 2N+1 features are extracted, and the calculation formula for the extraction is as follows:

[0064] F t+i =F extra (x t+i ), i = -N, ... N

[0065] Wherein: F extra F represents the residual block used to extract shallow features. t+i ∈R H×W×C This represents the shallow features obtained, and C represents the number of channels.

[0066] The calculation formula for the coarse alignment feature output in step two is as follows:

[0067]

[0068] In the formula: This indicates a deformable alignment module with backpropagation. This indicates the output coarse alignment feature.

[0069] The specific calculation formula for generating fused features by processing the residual block stack in step three is shown below:

[0070]

[0071] In the formula: This represents the fusion feature corresponding to the t+i adjacent frames, [·] represents the concatenation operation, and L Res This represents a residual block.

[0072] Step four involves obtaining fine-grained alignment features, which requires the use of the DAM module to perform forward propagation between the fused features and generate fine-grained alignment features. The calculation formula is shown below:

[0073]

[0074] In the formula: This represents the fine alignment features extracted during forward propagation. In step five, after obtaining the fine alignment features, the fine alignment features, fused features, and shallow features are fused together, and clustered features are generated through residual blocks. The calculation formula for generating clustered features is shown below:

[0075]

[0076] In the formula: L represents the clustered features generated through residual blocks. Res Represents the residual block. Indicates fine alignment features. F represents the fusion feature. t+i Indicates shallow features;

[0077] The formula for calculating the high-resolution frame image output in step six is ​​as follows:

[0078]

[0079] In the formula: B up Indicates a bicubic upsampling operation, L pc It represents a block composed of convolutions and pixel recombination. This indicates that a high-resolution frame image is output.

[0080] Example 3:

[0081] Please see Figure 3 This embodiment provides a technical solution based on Embodiment 1. The DAM includes two modules: a multi-scale deformable convolution module (MDCM) and a multi-scale attention module (MAM).

[0082] Considering the offset overflow caused by training instability, adjacent features F are processed before transmission. i+1 and F iLayer normalization is performed to generate the offset, and its calculation formula is shown below:

[0083]

[0084] F i ln =LN(F i )

[0085] Where: LN(.) represents the layer normalization operation.

[0086] We use the convolutional neural network LConv from F i ln and Cascaded prediction offset O along the channel direction i The calculation formula is as follows:

[0087]

[0088] In order to perform motion compensation from the time dimension, the offset O i and F i ln Input MDCM to generate alignment feature F i mdcm The calculation formula is as follows:

[0089]

[0090] In the formula: MDCM(.) represents a multi-scale deformable convolutional module.

[0091] To enhance the features of the reference frame, the original feature F i Add to alignment feature F i mdcm In this process, a refined alignment feature is generated, and its calculation formula is shown below:

[0092]

[0093] To fully utilize intra-frame information for reconstruction, similar features from different spatial locations are leveraged. First, the aligned, refined features F... i mdcm Layer normalization is performed, and then the input is fed into a multi-scale attention module to generate spatial attention features F. i mam :

[0094]

[0095] In the formula: MAM(.) represents a multi-scale attention module.

[0096] Finally, the final alignment features are given as follows:

[0097]

[0098] Example 4:

[0099] Please see Figure 4 This embodiment provides a technical solution based on Embodiment 1. Since features from different receptive fields can provide complementary information during convolution to help align the features of each adjacent frame with the features of the reference frame, a multi-scale deformable convolution module for video super-resolution tasks is designed. Unlike optical flow-based alignment methods, this multi-scale deformable convolution module can align the reference frame and adjacent frames at the feature level without explicit motion estimation. For each frame's feature F... t+i Given a deformable convolution kernel with K sampling positions, i∈[-N,N], p i and w i Let p represent the pre-specified offset and the weight of the i-th position, respectively. The alignment feature at position p can be calculated using the following formula:

[0100]

[0101] In the formula: w i The weights of the convolution kernel, Δp i p i The corresponding offset, M represents the number of elements in the convolution kernel. For a 3×3 kernel, M equals 9, while for a 5×5 kernel, M = 25.

[0102] To overcome train instability caused by offset divergence, a multi-scale deformable convolutional module incorporating layer normalization was designed. This module can effectively extract features from a wide range of spatial locations and avoid divergence during network training. To utilize complementary scale feature information from adjacent frames, normalized features are... and the calculated offset O i Feed into DConv 3×3 Get 3 and DConv 5×5 The specific extraction and calculation formula for obtaining multi-scale aligned features is shown below:

[0103]

[0104]

[0105] In the formula: DConv 3×3 and DConv 5×5 These represent multi-scale deformable convolutional modules equipped with kernel sizes of 3×3 and 5×5, respectively.

[0106] The multi-scale features are then concatenated and passed to the LConov function for further refinement, as shown in the following formula:

[0107]

[0108] In the formula: F i+1 This indicates a refined feature.

[0109] Finally, the alignment refinement feature F generated by MDCM is used. i mdcm The calculation formula is as follows:

[0110]

[0111] This results in the output of a high-resolution frame image;

[0112] Bidirectional multi-scale deformable attention networks are better suited for complex large-scale motion. The designed deformable alignment module can effectively align and fuse features by using multi-scale deformable convolution and multi-scale attention mechanisms. Compared with the sliding window method, the bidirectional propagation mechanism uses multi-frame information aggregation and achieves better propagation effect through sequence-to-sequence reconstruction. Multi-scale deformable convolution alignment together with multi-scale attention can improve alignment accuracy. Specifically, our multi-scale attention strategy can effectively capture global non-local self-similarity in space to supplement lost details.

[0113] Example 5:

[0114] Please see Figure 5-9 This embodiment provides a technical solution based on Embodiment 1. To explore the correlation of intra-frame features, we designed a MAM consisting of a Multi-Scale Visual Aggregation Residual Network (MVARN) and self-attention blocks induced from a Transformer. Specifically, the Transformer is a Visual Transformer image feature extractor, which uses a Visual Transformer model based on contrastive language-image pre-training (CLIP), abbreviated as CLIPViT. The Visual Transformer image feature extractor performs serialization preprocessing on the image when extracting image features. In the MAM module, the MVARN is mainly responsible for extracting multi-scale local correlated feature information, while intra-frame self-attention helps to obtain globally correlated features.

[0115] In our experiments, we conducted tests on two widely used datasets, Vimeo-90K and REDS, to verify the performance of the proposed BDAM method. All experiments were implemented using the PyTorch framework and ran on a computer with an Intel Xeon Silver 4108 @ 1.80GHz 16-core processor, 48GB of RAM, a GTX 2080 Ti graphics card, and Windows 10 operating system.

[0116] We compared our proposed method with state-of-the-art video super-resolution reconstruction methods to demonstrate its effectiveness. First, we trained the designed network and then tested it. During training, each sequence of images was randomly rotated or flipped to augment the data. Simultaneously, we cropped low-resolution video sequences into 64×64 image patches and used two sequences as a batch, inputting them as RGB channel images for training. In the feature extraction part, we used 5 residual blocks with 64 extracted feature channels, while the reconstruction module used 10 residual blocks. In the loss settings, we used the Charbonnier function as the final loss, with ε set to 1×10⁻³.

[0117] We compared our proposed BDAM with several state-of-the-art algorithms, and the quantitative results are as follows: Figure 8 and Figure 9 As shown, where Figure 8 In this context, Y and RGB represent the Y and RGB channels, respectively.

[0118] Models with higher PSNR and SSIM show better performance. The best results are marked in red, and the second-best results in blue. It can be seen that our proposed method achieves better results than the optical flow alignment method TOFlow. Compared to TOFlow, the proposed method improves PSNR by 0.82 dB on the Vimeo-90K dataset. From reconstructed video frames (such as...) Figure 6 and Figure 7 As can be seen, our proposed method can recover clearer square patterns, which shows that multi-scale deformable convolution alignment, together with multi-scale attention, can improve alignment accuracy. In particular, our multi-scale attention can effectively capture global horizontal and local self-similarity in space to supplement lost details.

[0119] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video super-resolution method based on a bidirectional multi-scale deformable attention network, characterized in that: Specifically, the following steps are included: Step 1: Given a low-resolution input frame sequence, use feature extraction blocks composed of residual blocks to obtain shallow features; Step 2: These features are fed into the DAM alignment module through backpropagation. The coarse alignment features are calculated and output using the formula. The DAM contains two modules: a multi-scale deformable convolution module and a multi-scale attention module. Step 3: The shallow features and the coarse alignment features are concatenated and fed into the residual block stack for processing to generate fused features; Step 4: Using the DAM module, two adjacent fused features are fed into the DAM module through forward propagation, and fine-aligned features are generated by calculating using a formula; Step 5: Fuse the fine alignment features, the fused features, and the shallow features, and generate clustered features using the calculation formula of the residual block; Step 6: The clustered features are recombined into pixels and added to the upsampled low-resolution frame. The high-resolution frame image is then calculated using a formula.

2. The video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 1, characterized in that: In step one, a low-resolution input frame sequence is given, which can be denoted as... ,in This represents the t-th low-resolution video frame; T, H, W, and C represent the length, height, width, and number of channels of the video, respectively.

3. The video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 2, characterized in that: The feature extraction block composed of residual blocks in step one is from... 2N+1 features are extracted, and the calculation formula for the extraction is as follows: ； in: This represents the residual block used to extract shallow features. This represents the obtained shallow features. Indicates the number of channels.

4. The video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 3, characterized in that: The calculation formula for the output coarse alignment feature in step two is as follows: ； In the formula: This indicates a deformable alignment module with backpropagation. This indicates the output coarse alignment feature.

5. A video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 4, characterized in that: The specific calculation formula for generating fused features by processing the residual block stack in step three is as follows: ； In the formula: Indicates corresponding to The fusion features of adjacent frames, where [·] represents the splicing operation. This represents a residual block.

6. A video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 5, characterized in that: In step four, obtaining the fine alignment features requires the use of the DAM module to perform forward propagation between the fused features to generate the fine alignment features. The calculation formula is as follows: ； In the formula: This represents the fine alignment features extracted during forward propagation. This indicates a deformable alignment module for forward propagation.

7. A video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 6, characterized in that: In step five, after obtaining the fine alignment features, the fine alignment features, fused features, and shallow features are fused together, and clustered features are generated through residual blocks. The calculation formula for generating clustered features is as follows: ； In the formula: This represents the clustered features generated by the residual blocks. Represents the residual block. This indicates fine alignment features. Indicates fusion features, This indicates shallow features.

8. A video super-resolution method based on a bidirectional multi-scale deformable attention network according to claim 7, characterized in that: The calculation formula for analyzing and outputting the high-resolution frame image in step six is ​​as follows: ； In the formula: This indicates a bicubic upsampling operation. It represents a block composed of convolutions and pixel recombination. This indicates that a high-resolution frame image is output.

Images