Super-resolution reconstruction method for enhancing self-attention based on space channel

Abstract

The invention discloses a super-resolution reconstruction method for enhancing self-attention based on a space channel. The super-resolution reconstruction method is specifically implemented according to the following steps: step 1, constructing a super-resolution reconstruction model for enhancing self-attention based on the space channel; 2, training the model constructed in the step 1; and step 3, testing the model trained in the step 2. According to the method, the ubiquitous problems of huge network model parameter quantity, high calculation complexity, insufficient channel information utilization, non-ideal reconstruction effect and the like in the existing reconstruction method are solved, texture details of the reconstructed image are richer and more vivid, the reconstruction quality is remarkably improved, and the method has wide prospects and important application values in practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and relates to a super-resolution reconstruction method based on spatial channel enhanced self-attention. Background Art

[0002] Single image super-resolution reconstruction technology is an image processing method that restores high-resolution images by computer processing low-resolution images or image sequences. In recent years, this technology has attracted widespread attention in the field of image processing. Successfully reconstructing high-resolution images with rich details and high quality has become a research focus, which has a profound impact on the progress of technologies such as real-time video surveillance, medical image processing, face recognition and remote sensing imaging. With the rise of machine learning and deep learning technologies, learning-based strategies have gradually received attention. Among them, convolutional neural network (CNN) technology can achieve high-quality image reconstruction and has become the mainstream method of super-resolution technology. In recent years, the Transformer model has achieved remarkable results in the fields of natural language processing and computer vision. The Transformer model has gradually been introduced into single image super-resolution reconstruction technology with its powerful global modeling ability. Compared with traditional convolutional neural networks, Transformer can more effectively capture long-range dependencies and complex texture information in images, thereby improving the quality of reconstructed images. Existing reconstruction methods commonly have problems such as large number of network model parameters, high computational complexity, insufficient utilization of channel information, and unsatisfactory reconstruction effects. Therefore, how to combine the single image super-resolution reconstruction technology of the Transformer model to reduce dependence on a large number of parameters and make it have advantages in computational cost and performance has become a research focus in the field of image processing. Summary of the invention

[0003] The purpose of the present invention is to provide a super-resolution reconstruction method based on spatial channel enhanced self-attention, which can improve the quality of reconstructed images.

[0004] The technical solution adopted by the present invention is a super-resolution reconstruction method based on spatial channel enhanced self-attention, which is specifically implemented in the following steps:

[0005] Step 1: construct a super-resolution reconstruction model based on spatial channel enhanced self-attention;

[0006] Step 2, training the model constructed in step 1;

[0007] Step 3: Test the model trained in step 2.

[0008] The present invention is also characterized in that:

[0009] Step 1 is implemented as follows:

[0010] Step 1.1, construct a shallow feature preliminary extraction module;

[0011] Step 1.2, construct deep feature extraction and fusion module;

[0012] Step 1.3, build the upsampling module.

[0013] Step 1.1 is implemented as follows:

[0014] Step 1.1.1, input a low-resolution image containing 3 channels;

[0015] In step 1.1.2, a standard 3×3 convolution operation is performed on the input features, and the number of output channels is adjusted to 60.

[0016] Step 1.2: Constructing the deep feature extraction and fusion module is implemented in the following steps:

[0017] Step 1.2.1, build a convolutional pixel attention block (CPAB);

[0018] F0=H CPAB (F sf ) (1)

[0019] Where F0 represents the output depth feature, F sf is the shallow feature of the input.

[0020] Step 1.2.2, construct 12 residual fusion modules (RFM);

[0021] F i =H RFM (F i-1 ) (2)

[0022] Where F i represents the deep features of the i-th RFM (1,…,12), F df It is the deep feature after concatenation along the channel dimension.

[0023] Step 1.2.3, construct an efficient convolution block.

[0024] In step 1.2.4, a residual connection operation is performed on the output features of step 1.2.3 and the output features of step 1.1.

[0025] Step 1.2.2 is implemented as follows:

[0026] Step 1.2.2.1, build enhanced self-attention block

[0027] First, the input features are layer normalized; secondly, two paths are used to extract features from the input features respectively; in path 1, a spatial self-attention block based on Swin Transformer is constructed, and then an enhanced spatial attention is constructed to calculate the spatial attention map of the output features, and the output features are multiplied by the channel attention map of the other path; in path 2, a channel self-attention block based on Transformer is constructed, and then a contrast-aware channel attention is constructed to calculate the channel attention map of the output features, and the output features are multiplied by the spatial attention map of the other path; then the output features of path 1 are added to the output features of path 2, and then a 1×1 convolution operation is performed; finally, the output features are residually connected with the input features of the module; where, given the input X∈R H×W×C , assuming the window size is h×w, then X can be divided into multiple windows, and the attention calculation of the i-th window of SSB can be expressed as:

[0028]

[0029] Q i n =X i W n q ,K i n =X i W n k ,V i n =X i W n v (4)

[0030]

[0031] Among them, X i represents the input of the i-th window, n represents the n-th attention head (n=1,…,N), is the weight of the linear layer, which is used to generate Query, Key, and Value respectively, d = C / N, and B is the relative position encoding; given an input X∈R (H×W)×C , the attention calculation of CSB can be expressed as:

[0032]

[0033] in, are the weights of the linear layer, which are used to generate Query, Key, and Value respectively. n represents the nth attention head (n=1,…,N). γ is used to adaptively scale the features of each attention head.

[0034] Step 1.2.2.2, build a gated feedforward network

[0035] First, perform layer normalization on the input features; second, perform 1×1 convolution and GeLU nonlinear activation on the features; then, perform 3×3 convolution on the features, calculate the product with the original features, and then perform 1×1 convolution; finally, perform residual connection on the output features and the input features of the module;

[0036] Step 1.2.2.3, construct an efficient convolution block.

[0037] Step 1.2.3 is implemented as follows:

[0038] Step 1.2.3.1, perform a 3×3 convolution operation on the input features, reduce the number of channels, and perform a LeakyReLU nonlinear activation operation;

[0039] Step 1.2.3.2, perform a 1×1 convolution operation on the input features and perform a LeakyReLU nonlinear activation operation;

[0040] In step 1.2.3.3, perform a 3×3 convolution operation on the input features and expand the number of channels.

[0041] In step 1.3, a 3×3 convolution operation is performed on the input features, and the number of output features is the product of the square of the scaling factor and 3. The features are then rearranged pixel by pixel to generate an upsampled reconstructed image.

[0042] In step 2, the following steps are implemented:

[0043] Step 2.1, using 800 images from DIV2K and 2650 images from Flickr2K as training datasets, randomly rotating and horizontally flipping the images in the training datasets for data augmentation;

[0044] Step 2.2, downsampling using the bicubic method to generate low-resolution images at 2x, 3x, and 4x scaling factors respectively;

[0045] Step 2.3, crop the original image into a 64×64 block as the RGB image for model input;

[0046] Step 2.4, set the training hyperparameters and train the model.

[0047] In step 2.4, ADAM is used as the optimizer for training, and the initial learning rate is set to 2×10 -4 , a multi-step learning rate reduction strategy is adopted, and the L1 loss function is used to optimize the model 5×10 5 Iterations:

[0048]

[0049] Where N is the number of input low-resolution images LR, θ is the training parameter set of the model, and |·| is the absolute value operator. In addition, in the training of models with scaling factors of 3 and 4, the trained model with a scaling factor of 2 is used as the initial weight.

[0050] Step 3 is implemented as follows:

[0051] Step 3.1, use Set5, Set14, BSD100, Urban100 and Manga109 as test datasets;

[0052] Step 3.2, load the trained model and generate a super-resolution reconstructed image;

[0053] In step 3.3, the peak signal-to-noise ratio, structural similarity, LPIPS, and perceptual index are calculated to evaluate the quality of the reconstructed image.

[0054] The beneficial effects of the present invention are:

[0055] The super-resolution reconstruction method based on spatial channel enhanced self-attention in the present invention solves the problems commonly existing in existing reconstruction methods, such as large number of network model parameters, high computational complexity, insufficient utilization of channel information, and unsatisfactory reconstruction effect. It makes the texture details of the reconstructed image richer and more realistic, and the reconstruction quality is significantly improved. It has broad prospects and important application value in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is an overall framework diagram of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0057] Figure 2 It is a framework diagram of the convolution pixel attention block in step 1 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0058] Figure 3 It is a framework diagram of the enhanced self-attention block in step 1 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0059] Figure 4 It is a framework diagram of the gated feedforward network in step 1 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0060] Figure 5 It is the original high-definition image No. 0158 of the DIV2K data set in step 2 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0061] Figure 6 It is a low-resolution image obtained by reducing the resolution of image No. 0158 of the DIV2K data set by 4 times by using bicubic downsampling in step 2 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention;

[0062] Figure 7 It is the super-resolution reconstruction result of the comic image in the Set14 dataset at a 4-fold magnification factor obtained in step 3 of the super-resolution reconstruction method based on spatial channel enhanced self-attention of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Example 1

[0065] The present invention is based on a super-resolution reconstruction method based on spatial channel enhanced self-attention. The overall framework of the model is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0066] Step 1: construct a super-resolution reconstruction model based on spatial channel enhanced self-attention;

[0067] Step 1 is implemented as follows:

[0068] Step 1.1, construct a shallow feature preliminary extraction module;

[0069] Step 1.2, construct deep feature extraction and fusion module;

[0070] Step 1.3, build upsampling module;

[0071] Step 1.3 is implemented as follows:

[0072] Perform a 3×3 convolution operation on the input features, and the output feature number is the product of the square of the scaling factor and 3. Then rearrange the features pixels to generate an upsampled reconstructed image.

[0073] Step 2: train the model constructed in step 1;

[0074] In step 2, the following steps are implemented:

[0075] Step 2.1, using 800 images from DIV2K and 2650 images from Flickr2K as training datasets, randomly rotating and horizontally flipping the images in the training datasets for data augmentation;

[0076] Step 2.2, the input original dataset DIV2K image 0158 is as follows Figure 5As shown, the bicubic method is used for downsampling to generate low-resolution images at 2x, 3x, and 4x scaling factors, respectively. Figure 6 That is, the image with the resolution reduced by 4 times;

[0077] Step 2.3, crop the original image into a 64×64 block as the RGB image for model input;

[0078] Step 2.4, set the training hyperparameters and train the model;

[0079] In step 2.4, ADAM is used as the optimizer for training, and the initial learning rate is set to 2×10 -4 , a multi-step learning rate reduction strategy is adopted, and the L1 loss function is used to optimize the model 5×10 5 Iterations:

[0080]

[0081] Where N is the number of input low-resolution images LR, θ is the training parameter set of the model, and |·| is the absolute value operator. In addition, in the training of models with scaling factors of 3 and 4, the trained model with scaling factor of 2 is used as the initial weight.

[0082] Step 3: Test the model trained in step 2;

[0083] Step 3 is implemented as follows:

[0084] Step 3.1, use Set5, Set14, BSD100, Urban100 and Manga109 as test datasets;

[0085] Step 3.2, load the trained model and generate super-resolution reconstructed images. The super-resolution reconstruction results of the comic image in the Set14 dataset at a 4x magnification factor are as follows: Figure 7 As shown;

[0086] In step 3.3, the peak signal-to-noise ratio, structural similarity, LPIPS, and perceptual index are calculated to evaluate the quality of the reconstructed image.

[0087] Example 2

[0088] The present invention is based on a super-resolution reconstruction method of spatial channel enhanced self-attention.

[0089] In step 1.1, a shallow feature preliminary extraction module is constructed, which is implemented in the following steps:

[0090] Step 1.1.1, input a low-resolution image containing 3 channels (RGB);

[0091] Step 1.1.2, perform a 3×3 standard convolution operation on the input features and adjust the number of output channels to 60;

[0092] Step 1.2: Build a deep feature extraction and fusion module. The specific steps are as follows:

[0093] Step 1.2.1, build a convolutional pixel attention block, the module structure is as follows Figure 2 As shown, it is expressed as

[0094] F0=H CPAB (F sf ) (1)

[0095] In the formula, F0 represents the output depth feature, F sf is the shallow feature of the input;

[0096] Step 1.2.2, construct 12 residual fusion modules (RFM), expressed as:

[0097] F i =H RFM (F i-1 ) (2)

[0098] In the formula, F i represents the deep features of the i-th RFM (1,…,12), F df is the deep feature after connection according to the channel dimension;

[0099] Step 1.2.2 is implemented according to the following steps:

[0100] Step 1.2.2.1, build the enhanced self-attention block (ESB), the module structure is as follows Figure 3 As shown, the specific implementation steps are as follows:

[0101] First, the input features are layer normalized; secondly, two paths are used to extract features from the input features; in path 1, a spatial self-attention block (SSB) based on Swin Transformer is first constructed, and then an enhanced spatial attention (ESA) is constructed to calculate the spatial attention map of the output features, and the output features are multiplied by the channel attention map of the other path; in path 2, a channel self-attention block (CSB) based on Transformer is constructed, and then a contrast-aware channel attention (CCA) is constructed to calculate the channel attention map of the output features, and the output features are multiplied by the spatial attention map of the other path; then the output features of path 1 are added to the output features of path 2, and then a 1×1 convolution operation is performed; finally, the output features are residually connected with the input features of the module. Where, given the input X∈R H×W×C, assuming the window size is h×w, then X can be divided into multiple windows, and the attention calculation of the i-th window of SSB can be expressed as:

[0102]

[0103] Q i n =X i W n q ,K i n =X i W n k ,V i n =X i W n v (4)

[0104]

[0105] Among them, X i represents the input of the i-th window, n represents the n-th attention head (n=1,…,N), is the weight of the linear layer, which is used to generate Query, Key, and Value respectively, d = C / N, and B is the relative position encoding; given an input X∈R (H×W)×C , the attention calculation of CSB can be expressed as:

[0106]

[0107]

[0108] in, are the weights of the linear layer, which are used to generate Query, Key, and Value respectively. n represents the nth attention head (n=1,…,N). γ is used to adaptively scale the features of each attention head.

[0109] Step 1.2.2.2, construct a gated feedforward network (GFN), the module structure is as follows Figure 4 As shown, the specific implementation is as follows:

[0110] First, perform layer normalization on the input features; second, perform 1×1 convolution and GeLU nonlinear activation on the features; then, perform 3×3 convolution on the features, calculate the product with the original features, and then perform 1×1 convolution; finally, perform residual connection on the output features and the input features of the module;

[0111] Step 1.2.2.3, construct an efficient convolution block (ECB);

[0112] Step 1.2.3, build an efficient convolution block, and implement it according to the following steps

[0113] Step 1.2.3.1, perform a 3×3 convolution operation on the input features, reduce the number of channels, and perform a LeakyReLU nonlinear activation operation;

[0114] Step 1.2.3.2, perform a 1×1 convolution operation on the input features and perform a LeakyReLU nonlinear activation operation;

[0115] Step 1.2.3.3, perform a 3×3 convolution operation on the input features and expand the number of channels;

[0116] In step 1.2.4, a residual connection operation is performed on the output features of step 1.2.3 and the output features of step 1.1.

[0117] The experimental results are shown in Table 1 and Table 2. The results show that the network model SCESN trained by the method proposed in the present invention achieves higher PSNR and SSIM values, while lower LPIPS and PI values, with the best comprehensive perceptual quality and the most outstanding restored image effect.

[0118] Table 1. Comparison of the results of the method of the present invention with those of the mainstream methods LPIPS and PI, where the bold ones are the best

[0119]

[0120] Table 2. Comparison of PSNR and SSIM results between the proposed method and mainstream methods, where the bold ones are the best and the underlined ones are the suboptimal

[0121]

[0122]

[0123] Example 3

[0124] The present invention is based on a super-resolution reconstruction method of spatial channel enhanced self-attention, aiming to solve the problem of insufficient utilization of global context and channel features by existing methods. This method adopts self-attention and window self-attention as basic units, enhances the ability to capture global context, and has the ability to extract long-distance features. At the same time, a spatial channel enhancement block is introduced to interact and enhance spatial and channel information, so that the model can better understand the image structure and content, thereby improving reconstruction accuracy and image clarity. A large number of experiments have shown that compared with recent lightweight methods, this method exhibits highly competitive performance. In super-resolution tasks, this method not only achieves significant improvements in reconstruction accuracy, but also shows excellent results in image clarity and detail restoration.

Claims

1. Super-resolution reconstruction method based on spatial channel enhanced self-attention, It is characterized in that Please follow the steps below to implement: Step 1: construct a super-resolution reconstruction model based on spatial channel enhanced self-attention; Step 2: train the model constructed in step 1; Step 3: Test the model trained in step 2.

2. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 1, It is characterized in that The step 1 is specifically implemented according to the following steps: Step 1.1, construct a shallow feature preliminary extraction module; Step 1.2, construct deep feature extraction and fusion module; Step 1.3, build the upsampling module.

3. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 2, It is characterized in that The step 1.1 is specifically implemented according to the following steps: Step 1.1.1, input a low-resolution image containing 3 channels; In step 1.1.2, a standard 3×3 convolution operation is performed on the input features, and the number of output channels is adjusted to 60.

4. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 2, It is characterized in that The step 1.2 is specifically implemented according to the following steps: Step 1.2.1, construct a convolutional pixel attention block, expressed as: F 0 =H CPAB (F sf ) (1) In the formula, F 0 Represents the output deep features, F sf is the shallow feature of the input; Step 1.2.2, construct 12 residual fusion modules, expressed as: F i =H RFM (F i-1 ) (2) In the formula, F i represents the deep features of the i-th RFM (1,…,12), F df is the deep feature after connection according to the channel dimension; Step 1.2.3, construct an efficient convolution block; In step 1.2.4, a residual connection operation is performed on the output features of step 1.2.3 and the output features of step 1.

1.

5. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 4, It is characterized in that The step 1.2.2 is specifically implemented according to the following steps: Step 1.2.2.1, build enhanced self-attention block First, the input features are layer normalized; secondly, two paths are used to extract features from the input features respectively; in path 1, a spatial self-attention block based on Swin Transformer is constructed, and then an enhanced spatial attention is constructed to calculate the spatial attention map of the output features, and the output features are multiplied by the channel attention map of the other path; in path 2, a channel self-attention block based on Transformer is constructed, and then a contrast-aware channel attention is constructed to calculate the channel attention map of the output features, and the output features are multiplied by the spatial attention map of the other path; then the output features of path 1 are added to the output features of path 2, and then a 1×1 convolution operation is performed; finally, the output features are residually connected with the input features of the module; where, given the input X∈R H×W×C , assuming the window size is h×w, then X can be divided into multiple windows, and the attention calculation of the i-th window of SSB can be expressed as: Q i n =X i W n q ,K i n =X i W n k ,V i n =X i W n v (4) Among them, X i represents the input of the i-th window, n represents the n-th attention head (n=1,…,N), is the weight of the linear layer, which is used to generate Query, Key, and Value respectively, d = C / N, and B is the relative position encoding; given an input X∈R (H ×W)×C , the attention calculation of CSB can be expressed as: in, are the weights of the linear layer, which are used to generate Query, Key, and Value respectively. n represents the nth attention head (n=1,…,N). γ is used to adaptively scale the features of each attention head. Step 1.2.2.2, build a gated feedforward network First, perform layer normalization on the input features; second, perform 1×1 convolution and GeLU nonlinear activation on the features; then, perform 3×3 convolution on the features, calculate the product with the original features, and then perform 1×1 convolution; finally, perform residual connection on the output features and the input features of the module; Step 1.2.2.3, build an efficient convolutional block Perform a 3×3 convolution operation on the input features, reduce the number of channels, and perform a LeakyReLU nonlinear activation operation; then perform a 1×1 convolution operation on the input features, and perform a LeakyReLU nonlinear activation operation; then perform a 3×3 convolution operation on the input features, and expand the number of channels.

6. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 4, It is characterized in that The step 1.2.3 is specifically implemented according to the following steps: Step 1.2.3.1, perform a 3×3 convolution operation on the input features, reduce the number of channels, and perform a LeakyReLU nonlinear activation operation; Step 1.2.3.2, perform a 1×1 convolution operation on the input features and perform a LeakyReLU nonlinear activation operation; In step 1.2.3.3, perform a 3×3 convolution operation on the input features and expand the number of channels.

7. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 2, It is characterized in that In step 1.3, a 3×3 convolution operation is performed on the input features, and the output feature number is the product of the square of the scaling factor and 3. The features are then rearranged in pixels to generate an upsampled reconstructed image.

8. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 1, It is characterized in that The step 2 is specifically implemented according to the following steps: Step 2.1, using 800 images from DIV2K and 2650 images from Flickr2K as training datasets, randomly rotating and horizontally flipping the images in the training datasets for data augmentation; Step 2.2, downsampling using the bicubic method to generate low-resolution images at 2x, 3x, and 4x scaling factors, respectively; Step 2.3, crop the original image into a 64×64 block as the RGB image for model input; Step 2.4, set the training hyperparameters and train the model.

9. A super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 8, It is characterized in that In step 2.4, ADAM is used as the optimizer for training, and the initial learning rate is set to 2×10 -4 , using a multi-step learning rate reduction strategy, using L 1 The loss function optimizes the model 5×10 5 Iterations: Where N is the number of input low-resolution images LR, θ is the training parameter set of the model, and |·| is the absolute value operator. In addition, in the training of models with scaling factors of 3 and 4, the trained model with a scaling factor of 2 is used as the initial weight.

10. The super-resolution reconstruction method based on spatial channel enhanced self-attention according to claim 1, It is characterized in that The step 3 is specifically implemented according to the following steps: Step 3.1, use Set5, Set14, BSD100, Urban100 and Manga109 as test datasets; Step 3.2, load the trained model and generate a super-resolution reconstructed image; In step 3.3, the peak signal-to-noise ratio, structural similarity, LPIPS, and perceptual index are calculated to evaluate the quality of the reconstructed image.

Images