Frame recurrent video super-resolution method based on adaptive multi-level attention

Abstract

The application discloses a frame cycle video super-resolution reconstruction technology based on adaptive multi-level attention, and has the characteristics that the technology comprises the following steps: 1) data set preprocessing; 2) establishing a reconstruction model; 3) an adaptive multi-level attention process; 4) a reconstruction network; and 5) model training. The technical scheme can effectively utilize initial low-resolution interframe information and time information reserved after cycling, and can better capture spatial context information, so that more high-frequency information is obtained to enrich image details, and finally the model has better reconstruction effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a frame-looping video super-resolution method based on adaptive multi-level attention. Background Technology

[0002] Super-resolution is a traditional yet dynamic topic in the field of low-resolution imaging, aiming to generate corresponding high-resolution images from low-resolution images. Depending on the input, super-resolution can be divided into single-image super-resolution techniques and video super-resolution techniques. In recent years, with the increasing demands from smartphones, apps, smart HDTVs, and surveillance equipment, the demand for super-resolution technology has grown significantly. With extensive research and positive results in deep learning, single-image super-resolution has made significant progress in recent years. However, compared to single-image super-resolution, which utilizes only information from a single image (i.e., reconstruction is limited to the spatial dimension), video super-resolution incorporates temporal information, namely neighboring frame information, to further improve the quality of super-resolution (SR).

[0003] Since video super-resolution technology relies heavily on the temporal correlation between consecutive video frames to enhance video super-resolution information, it is crucial to make full use of the spatiotemporal information of the video. Deep learning-based video super-resolution technology can be roughly divided into two categories: (1) sliding window-based method; (2) recursive method. The first method usually takes multiple low-resolution images as input to generate a single high-resolution image and uses a sliding window to process long video sequences. This method processes each input frame multiple times during inference, resulting in low feature utilization efficiency and increased cost. The second method is the strategy adopted by most advanced methods. Recursive methods have a better effect on sequence models and can model the long-term contextual information of time series well, but there are still shortcomings. First, it does not make full use of the time information saved after recursion and does not consider whether there is too much redundant information. Second, it is difficult to make better use of the initial low-resolution adjacent frames. Moreover, recursive models are more prone to gradient vanishing, training is unstable, and the receptive field is local.

[0004] Therefore, a new technology is needed to solve the problems mentioned above. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a frame-looping video super-resolution method based on adaptive multi-level attention. This method utilizes adaptive multi-level attention to leverage initial low-resolution inter-frame information and temporal information retained after looping, while capturing spatial context information to obtain more high-frequency information to enrich image details. It can prevent gradient vanishing and training instability during looping, obtain a better spatiotemporal receptive field, learn better features, and improve reconstruction quality.

[0006] The technical solution to achieve the objective of this invention is:

[0007] The frame-looping video super-resolution method based on adaptive multi-level attention differs from existing technologies in that it includes the following steps:

[0008] 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T;

[0009] 2) Establishing the Reconstruction Model: The reconstruction model (Model) is applied to process the training dataset T to obtain the super-resolution image SR. The reconstruction model (Model) consists of an adaptive multi-level attention process AMLANet and a reconstruction process RNet, including:

[0010] 2-1) Take samples I from the training set t ,I t-1 The hidden state time information feature map H from the previous time. t-1 Channel fusion is performed to obtain the input feature f1;

[0011] 2-2) Input feature f1 into a 3*3 convolution kernel for channel dimensionality reduction to obtain feature f2;

[0012] 2-3) Input feature f2 into a convolutional layer to obtain shallow feature f3;

[0013] 2-4) Input the shallow feature f3 into the N adaptive multi-level attention modules AMLAB and output the feature f4;

[0014] 2-5) Input feature f4 into a 3*3 convolution kernel to reduce the number of channels and output feature X3;

[0015] 2-6) The feature X3 is input to the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer. The output is the final super-resolution image SR.

[0016] 3) Adaptive Multi-Level Attention Process: In steps 2) and 4), the adaptive multi-level attention module AMLAB first inputs feature x0 and performs channel reduction through a convolutional operation to obtain feature x1. Then, it passes through a residual multi-level block (RMLB) consisting of multiple cascaded convolutional layers, each with its own output, to obtain feature x2. Next, feature x2 is passed through a channel attention block (CAB) and a spatial attention block (SAB) and then added in parallel with the input feature x2 to obtain feature x3. Each branch is infused with learnable parameters β, α, and μ. Finally, feature x1 and feature x3 are added element-wise and then concatenated with the input feature x0 with the learnable parameter λ added to obtain the final output feature x of the adaptive multi-level attention module. AMLAB The process is shown in the following formula:

[0017] x1 = σ(conv3(x0)),

[0018] x2 = RMLB(x1),

[0019] x3=β*CAB(x2)+μ*x2+α*SAB(x2),

[0020] x AMLAB =concat δ ((x3+x1),λ*x0),

[0021] Where conv3 is a convolution with a 3x3 kernel, σ is the activation function layer, and concat... δ This indicates that the fusion method is channel splicing.

[0022] 4) Reconstruction Process: Using the feature X3 extracted by the adaptive multi-level attention network described in step 2) as the input to the reconstruction process, the input feature X3 is first processed through 4 residual blocks RB to obtain a refined feature map X4. Each residual block has two layers of 3x3 convolutional kernels and one ReLU activation layer. The refined feature map X4 is then processed through convolutional layers and activation function layers σ, Dropout layers, and convolutional layers to obtain the hidden information feature map h. t Reconstructing the output feature map t ; for h t o t The final hidden information feature map H is obtained by concatting and fusion of the channels. t At the same time, for o t Subpixel convolution is performed to obtain the feature map, and the corresponding low-resolution image of the reconstructed frame is obtained by bicubic upsampling. t The final super-resolution image (SR) is obtained by adding elements one by one.

[0023] 5) Model Training: The model in this paper adopts an end-to-end training approach, first using the L1 loss function to accelerate convergence, and then using the L2 loss function for fine-tuning, as shown in the following formula:

[0024] L1 loss function:

[0025] L2 loss function:

[0026] The residual multilevel block (RMLB) mentioned in step 3) is as follows: First, the input feature x1 is cascaded through three convolutional layers conv3 with 3x3 kernels, and an activation function layer σ is added after each convolutional layer. The output of each layer is represented as F. a F b F c Channel concatenation and fusion are performed using `concat`, followed by dimensionality reduction of the number of channels using a 1x1 convolution kernel (`conv1`). The features obtained are then combined with the input features `x1` of the residual multilevel module (RMLB) element-wise to form the residual multilevel module, as shown in the following formula:

[0027] x2 = conv1(concat(F a ,F b ,F c ))+x1.

[0028] The channel attention module (CAB) mentioned in step 3) is as follows: First, the input features are passed through a 3x3 convolutional layer (conv3) and an activation function layer (σ), and a 5x5 convolutional layer (conv5) and an activation function layer, respectively, to obtain features F3 and F5. Then, features F3 and F5 are added pixel by pixel to generate a feature map of size CxHxW, and global average pooling is performed on F. gp Generate Cx1x1 feature map s c Then, the dimensionality is reduced to a Zx1x1 feature map z through a fully connected fully connected fc layer. c Then, a fully connected (FC) interface is used for channel-dimensional expansion. After softmax, channels are split and then multiplied with the corresponding feature layers of different receptive fields. Finally, the feature layers after multiplication are added pixel by pixel to obtain the final output feature layer V, as shown in the following formula:

[0029] F3 = σ(conv3(x2)),

[0030] F5 = σ(conv3(x2)),

[0031] s c =F gp (F3+F5),

[0032] z c=fc(s c ),

[0033] f c =fc(z c ),

[0034] V = split(softmax(f) c ))*F3+split(softmax(f c ))*F5,

[0035] Where x2 represents the output feature of the residual multi-level module.

[0036] The spatial attention module SAB mentioned in step 3) is as follows: First, the input features are subjected to global max pooling (GMP) and global average pooling (GAP) to obtain 1*H*W feature maps F1 and F2S2. Feature maps F1 and F2 are then concatenated by channel to form a 2*H*W feature map. This map is then passed through a multilayer perceptron (MLP), where a sigmoid function is applied, followed by a dot product operation with the input features. This constitutes the spatial attention module. The MLP consists of two 1*1 convolutional layers, one ReLU activation layer, and one 7*7 convolutional layer, as shown in the following formula:

[0037] F1 = GAP(x2),

[0038] F2 = GAP(x2),

[0039] F3 = concat(F1, F2),

[0040] F4 = MLP(F3),

[0041] F5 = Sigmoid(F4),

[0042] F SA =x2⊙F5,

[0043] Where x2 represents the output feature of the residual multi-level module.

[0044] In terms of time, this technical solution can make better use of inter-frame information and time information based on the previous hidden state after the loop; in terms of space, it can make better use of spatial context information, thereby recovering better reconstruction details and thus effectively improving the quality of reconstructed image frames in the video.

[0045] This method employs adaptive multi-level attention, which can effectively utilize initial low-resolution inter-frame information and temporal information retained after looping, and can better capture spatial context information, thereby obtaining more high-frequency information to enrich image details. It can prevent gradient vanishing and training instability during looping, obtain a better spatiotemporal receptive field, learn better features, and improve reconstruction quality. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the method flow in the embodiment;

[0047] Figure 2 This is a schematic diagram of the overall network in the embodiment;

[0048] Figure 3 This is a schematic diagram of the residual multi-level module in the embodiment;

[0049] Figure 4 This is a schematic diagram of the channel attention module in the embodiment;

[0050] Figure 5 This is a schematic diagram of the spatial attention module in the embodiment. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0052] Example:

[0053] Reference Figure 1 , Figure 2 The frame-looping video super-resolution method based on adaptive multi-level attention, unlike existing technologies, includes the following steps:

[0054] 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T;

[0055] 2) Establishing the Reconstruction Model: The reconstruction model (Model) is applied to process the training dataset T to obtain the super-resolution image SR. The reconstruction model (Model) consists of an adaptive multi-level attention process AMLANet and a reconstruction process RNet, including:

[0056] 2-1) Take samples I from the training set t ,I t-1 The hidden state time information feature map H from the previous time. t-1 Channel fusion is performed to obtain the input feature f1;

[0057] 2-2) Input feature f1 into a 3*3 convolution kernel for channel dimensionality reduction to obtain feature f2;

[0058] 2-3) Input feature f2 into a convolutional layer to obtain shallow feature f3;

[0059] 2-4) Input the shallow feature f3 into the N adaptive multi-level attention modules AMLAB and output the feature f4;

[0060] 2-5) Input feature f4 into a 3*3 convolution kernel to reduce the number of channels and output feature X3;

[0061] 2-6) The feature X3 is input to the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer. The output is the final super-resolution image SR.

[0062] 3) Adaptive Multi-Level Attention Process: In steps 2) and 4), the adaptive multi-level attention module AMLAB first inputs feature x0 and performs channel reduction through a convolutional operation to obtain feature x1. Then, it passes through a residual multi-level block (RMLB) consisting of multiple cascaded convolutional layers, each with its own output, to obtain feature x2. Next, feature x2 is passed through a channel attention block (CAB) and a spatial attention block (SAB) and then added in parallel with the input feature x2 to obtain feature x3. Each branch is infused with learnable parameters β, α, and μ. Finally, feature x1 and feature x3 are added element-wise and then concatenated with the input feature x0 with the learnable parameter λ added to obtain the final output feature x of the adaptive multi-level attention module. AMLAB The process is shown in the following formula:

[0063] x1 = σ(conv3(x0)),

[0064] x2 = RMLB(x1),

[0065] x3=β*CAB(x2)+μ*x2+α*SAB(x2),

[0066] x AMLAB =concat δ ((x3+x1),λ*x0),

[0067] Where conv3 is a convolution with a 3x3 kernel, σ is the activation function layer, and concat... δ This indicates that the fusion method is channel splicing.

[0068] 4) Reconstruction Process: The feature X3 extracted by the adaptive multi-layer attention network described in step 2) is used as the input to the reconstruction process. First, the input feature X3 is passed through 4 residual blocks RB to obtain a refined feature map X4. Each residual block has two layers of 3x3 convolutional kernels and one ReLU activation layer. The refined feature map X4 is then subjected to convolutional layers and activation function layers σ, Dropout layers, and convolutional layers to obtain the hidden information feature map h. t Reconstructing the output feature map t ; for h t o t The final hidden information feature map H is obtained by concatting and fusion of the channels. t At the same time, for o t Subpixel convolution is performed to obtain the feature map, and the corresponding low-resolution image of the reconstructed frame is obtained by bicubic upsampling. t The final super-resolution image (SR) is obtained by adding elements one by one.

[0069] 5) Model Training: The model in this paper adopts an end-to-end training approach, first using the L1 loss function to accelerate convergence, and then using the L2 loss function for fine-tuning, as shown in the following formula:

[0070] L1 loss function:

[0071] L2 loss function:

[0072] Reference Figure 3 The residual multilevel RMLB block includes the following steps:

[0073] The residual multilevel block (RMLB) mentioned in step 3) is as follows: First, the input feature x1 is cascaded through three convolutional layers conv3 with 3x3 kernels, and an activation function layer σ is added after each convolutional layer. The output of each layer is represented as F. a F b F c Channel concatenation and fusion are performed using `concat`, followed by dimensionality reduction of the number of channels using a 1x1 convolution kernel (`conv1`). The features obtained are then combined with the input features `x1` of the residual multilevel module (RMLB) element-wise to form the residual multilevel module, as shown in the following formula:

[0074] x2 = conv1(concat(F a ,F b ,F c ))+x1.

[0075] Reference Figure 4 The Channel Attention Module (CAB) includes the following steps:

[0076] The channel attention module (CAB) mentioned in step 3) is as follows: First, the input features are passed through a 3x3 convolutional layer (conv3) and an activation function layer (σ), and a 5x5 convolutional layer (conv5) and an activation function layer, respectively, to obtain features F3 and F5. Then, features F3 and F5 are added pixel by pixel to generate a feature map of size CxHxW, and global average pooling is performed on F. gp Generate Cx1x1 feature map s c Then, the dimensionality is reduced to a Zx1x1 feature map z through a fully connected fully connected fc layer. c Then, a fully connected (FC) interface is used for channel-dimensional expansion. After softmax, channels are split and then multiplied with the corresponding feature layers of different receptive fields. Finally, the feature layers after multiplication are added pixel by pixel to obtain the final output feature layer V, as shown in the following formula:

[0077] F3 = σ(conv3(x2)),

[0078] F5 = σ(conv3(x2)),

[0079] s c =F gp (F3+F5),

[0080] z c =fc(s c ),

[0081] f c =fc(z c ),

[0082] V = split(softmax(f) c ))*F3+split(softmax(f c ))*F5,

[0083] Where x2 represents the output feature of the residual multi-level module.

[0084] Reference Figure 5 The Spatial Attention Module (SAB) includes the following steps:

[0085] The spatial attention module SAB mentioned in step 3) is as follows: First, the input features are subjected to global max pooling (GMP) and global average pooling (GAP) to obtain 1*H*W feature maps F1 and F2S2. Feature maps F1 and F2 are then concatenated by channel to form a 2*H*W feature map. This map is then passed through a multilayer perceptron (MLP), where a sigmoid function is applied, followed by a dot product operation with the input features. This constitutes the spatial attention module. The MLP consists of two 1*1 convolutional layers, one ReLU activation layer, and one 7*7 convolutional layer, as shown in the following formula:

[0086] F1 = GAP(x2),

[0087] F2 = GAP(x2),

[0088] F3 = concat(F1, F2),

[0089] F4 = MLP(F3),

[0090] F5 = Sigmoid(F4),

[0091] F SA =x2⊙F5,

[0092] Where x2 represents the output feature of the residual multi-level module.

Claims

1. A frame-looping video super-resolution method based on adaptive multi-level attention, characterized in that, Includes the following steps: 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T; 2) Establishing the Reconstruction Model: The reconstruction model (Model) is applied to process the training dataset T to obtain the super-resolution image SR. The reconstruction model (Model) consists of an adaptive multi-level attention process AMLANet and a reconstruction process RNet, including: 2-1) Take samples I from the training set t ,I t-1 The hidden state time information feature map H from the previous time. t-1 Channel fusion is performed to obtain the input feature f1; 2-2) Input feature f1 into a 3*3 convolution kernel for channel dimensionality reduction to obtain feature f2; 2-3) Input feature f2 into a convolutional layer to obtain shallow feature f3; 2-4) Input the shallow feature f3 into the N adaptive multi-level attention modules AMLAB and output the feature f4; 2-5) Input feature f4 into a 3*3 convolution kernel to reduce the number of channels and output feature X3; 2-6) The feature X3 is input to the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer. The output is the final super-resolution image SR. 3) Adaptive Multi-Level Attention Process: In steps 2) and 4), the adaptive multi-level attention module AMLAB first inputs feature x0 and performs channel reduction through a convolutional layer to obtain feature x1. Then, it passes through a residual multi-level block (RMLB) consisting of multiple cascaded convolutional layers, each with its own output, to obtain feature x2. Next, feature x2 is passed through the channel attention module (CAB) and the spatial attention module (SAB) respectively, and then added in parallel with the input feature x2 to obtain feature x3. Each branch is given learnable parameters β, α, and μ. Finally, feature x1 and feature x3 are added element-wise and then concatenated with the input feature x0 with the added learnable parameter λ to obtain the final output feature x of the adaptive multi-level attention module. AMLAB The process is shown in the following formula: x1 = σ(conv3(x0)), x2 = RMLB(x1), x3=β*CAB(x2)+μ*x2+α*SAB(x2), x AMLAB =concat δ ((x3+x1),λ*x0), Where conv3 is a convolution with a 3x3 kernel, σ is the activation function layer, and concat... δ This indicates that the fusion method is channel splicing. 4) Reconstruction Process: The features X3 extracted by AMLANet in the adaptive multi-level attention process described in step 2) are used as input to the reconstruction process. First, the input features X3 are passed through 4 residual blocks RB to obtain a refined feature map X4. Each residual block has two layers of 3x3 convolutional kernels and one ReLU activation layer. The refined feature map X4 is then subjected to convolutional layers and activation function layers σ, Dropout layers, and convolutional layers to obtain the hidden information feature map h. t Reconstructing the output feature map t ; for h t o t The final hidden information feature map H is obtained by concatting and fusion of the channels. t At the same time, for o t Subpixel convolution is performed to obtain the feature map, and the corresponding low-resolution image of the reconstructed frame is obtained by bicubic upsampling. t The final super-resolution image (SR) is obtained by adding elements one by one. 5) Model Training: The model in this paper adopts an end-to-end training approach, first using the L1 loss function to accelerate convergence, and then using the L2 loss function for fine-tuning, as shown in the following formula: L1 loss function: L2 loss function:

2. The frame-looping video super-resolution method based on adaptive multi-level attention according to claim 1, characterized in that, The residual multilevel block (RMLB) mentioned in step 3) is as follows: First, the input feature x1 is cascaded through three convolutional layers conv3 with 3x3 kernels, and an activation function layer σ is added after each convolutional layer. The output of each layer is represented as F. a F b F c Channel concatenation and fusion are performed using `concat`, followed by dimensionality reduction of the number of channels using a 1x1 convolution kernel (`conv1`). The features obtained are then combined with the input features `x1` of the residual multilevel module (RMLB) element-wise to form the residual multilevel module, as shown in the following formula: x2=conv1(concat(F a ,F b ,F c ))+x1。 3. The frame-looping video super-resolution method based on adaptive multi-level attention according to claim 1, characterized in that, The channel attention module (CAB) mentioned in step 3) is as follows: First, the input features are passed through a 3x3 convolutional layer (conv3) and an activation function layer (σ), and a 5x5 convolutional layer (conv5) and an activation function layer, respectively, to obtain features F3 and F5. Then, features F3 and F5 are added pixel by pixel to generate a feature map of size CxHxW, and global average pooling is performed on F. gp Generate Cx1x1 feature map s c Then, the dimensionality is reduced to a Zx1x1 feature map z through a fully connected fully connected fc layer. c Then, a fully connected (FC) interface is used for channel-dimensional expansion. After softmax, channels are split and then multiplied with the corresponding feature layers of different receptive fields. Finally, the feature layers after multiplication are added pixel by pixel to obtain the final output feature layer V, as shown in the following formula: F3 = σ(conv3(x2)), F5 = σ(conv3(x2)), s c =F gp (F3+F5), from c =fc(s c ), f c =fc(z c ), V=split(softmax(f c ))*F3+split(softmax(f c ))*F5, Where x2 represents the output feature of the residual multi-level module.

4. The frame-looping video super-resolution method based on adaptive multi-level attention according to claim 1, characterized in that, The spatial attention module SAB mentioned in step 3) is as follows: First, the input features are subjected to global max pooling (GMP) and global average pooling (GAP) to obtain 1*H*W feature maps F1 and F2S2. Feature maps F1 and F2 are then concatenated by channel to form a 2*H*W feature map. This map is then passed through a multilayer perceptron (MLP), where a sigmoid function is applied, followed by a dot product operation with the input features. This constitutes the spatial attention module. The MLP consists of two 1*1 convolutional layers, one ReLU activation layer, and one 7*7 convolutional layer, as shown in the following formula: F1 = GAP(x2), F2 = GAP(x2), F3 = concat(F1, F2), F4 = MLP(F3), F5 = Sigmoid(F4), F SA =x2⊙F5, Where x2 represents the output feature of the residual multi-level module.

Images