Image super-resolution method for improving SwinIR global modeling

By introducing enhanced global information module and self-calibration convolution operation in SwinIR, the global perception range of image super-resolution network is expanded, and the traditional convolutional network is solved inadequate simulation of remote dependencies and limited pixel information in image super-resolution Problem, achieving higher quality image reconstruction and performance improvement.

CN119067855APending Publication Date: 2024-12-03ANHUI UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411082005.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional convolutional neural networks have fixed size limitations in image super-resolution, resulting in insufficient simulation remote dependency capabilities and high computational complexity. SwinIR uses limited pixel information during super-resolution, resulting in artifacts and reconstructed image boundaries. Performance improvement is limited.

Method used

Adopting an improved SwinIR global modeling method, the enhanced global information module and self-calibrated convolution enhancement module are introduced into the deep feature extraction network to increase the attention of MLP channel and self-calibrated convolution operation. , expand the global perception range and pixel information utilization of the network.

Benefits of technology

The utilization of pixel information during image super-resolution is improved, the quality of reconstruction is improved, the interference of human subjective factors is reduced, the PSNR and SSIM indicators are significantly improved, and the calculation complexity and number of parameters are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067855A_ABST
    Figure CN119067855A_ABST
Patent Text Reader

Abstract

The invention relates to the field of image super-resolution, and discloses an image super-resolution method for improving SwinIR global modeling, and the method comprises the following steps: 1, carrying out the shallow feature extraction; 2, transmitting the shallow feature F0 as input to a deep feature extraction network HDF (.) to obtain a deep feature FDF; 3, fusing the shallow and deep features by using residual connection, and taking the fused shallow and deep features as the input of an up-sampling reconstruction module to obtain the final output of the SR network; according to the method, an MLP-based channel attention mechanism is additionally added in a SwinIR displacement window attention mechanism, and different from a traditional channel attention method, an MLP structure is adopted, so that all picture information comes from the picture, interference of human subjective factors is reduced to the maximum extent, and the quality of a reconstructed image is improved; self-calibration convolution is used for replacing traditional convolution in SwinIR, and the network uses more pixel information in the super-resolution process through the self-calibration convolution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution, and in particular to an image super-resolution method with improved SwinIR global modeling. Background Art

[0002] Image super-resolution (SR) is an important task in the field of computer vision and has wide applications in remote sensing, surveillance, medical imaging, and unmanned vehicles. The main goal of SR is to extract complex details from low-resolution images and reconstruct them into high-fidelity images.

[0003] Traditional convolutional neural networks (CNNs), such as the pioneering work of SRCNN, have revolutionized the field of SR. SRCNN applied deep learning methods to the field of image super-resolution for the first time and proposed an image super-resolution method based on convolutional neural networks. This pioneering work laid the foundation for subsequent deep learning super-resolution research.

[0004] The emergence of the Transformer model, especially ViT (Vision Transformer), has sparked a revolution in the field of computer vision. ViT effectively captures long-distance dependencies through its self-attention mechanism, overcoming the limitations of traditional convolutional networks. This has spawned a number of excellent networks based on Transformer for image super-resolution.

[0005] Attribution Map (LAM) is an image super-resolution analysis tool that visualizes the range of pixels utilized during the super-resolution process, with a particular focus on which input pixels contribute most to the final output during the super-resolution process.

[0006] In the above, we found the following shortcomings: 1) Due to the inherent ill-posed nature of SR and in the process of exploring the application of deep neural networks in image super-resolution, although these networks have shown strong learning and mapping capabilities, their inherent complexity and non-intuitiveness also make some behavior patterns difficult to accurately predict; 2) Since the fixed size of traditional convolutional kernels limits their ability to simulate long-range dependencies, many convolution-based SR architectures attempt to improve performance by increasing the depth or width of the network, which inevitably increases the complexity and computational cost of the model; 3) The Transformer model has high computational complexity and requires the input image to be divided into patches of fixed size, which may cause artifacts at the boundaries of super-resolution images; 4) As Figure 3 As shown in Figure 3, by using attribution graph analysis on SwinIR, it was found that the pixel information used by SwinIR in the super-resolution process is very limited.

[0007] Based on this, an image super-resolution method based on improved SwinIR global modeling is proposed here to solve the above problems. Summary of the invention

[0008] In order to overcome the above technical problems, the object of the present invention is to provide an image super-resolution method for improving SwinIR global modeling. The object of the present invention can be achieved by the following technical solutions:

[0009] An image super-resolution method for improving SwinIR global modeling includes the following steps:

[0010] Step 1: Shallow feature extraction; there is a 3X3 traditional convolution to map the input feature dimension to a higher dimension. Specifically, given a low-resolution image input I LR ∈R H×W×Cin , first use the shallow feature extraction network H SF (·) to extract shallow features F0∈R H×W×C , which is defined as F0=H SF (I LR ), where C in represents the low-resolution input channel, and C represents the channel after being mapped by the shallow feature extraction network.

[0011] Step 2: Transfer the shallow feature F0 as input to the deep feature extraction network H DF (·), and obtain the deep feature F DF , that is, F DF =H DF (F0),

[0012] Step 3: Use residual connection to fuse shallow and deep features and use them as the input of upsampling reconstruction module to get the final output of SR network. The process can be described as: SR =H upsample (F0+F DF ), where H upsample (·) represents the upsampling reconstruction module, which is composed of convolution and pixel shuffle to fuse the shallow and deep feature information obtained, and upsample to achieve the reconstruction effect; Step 2 includes the following steps:

[0013] Step 2.1: H DF (·) That is, the deep feature extraction network contains N enhanced global information modules (Enhance Global Information Module, EGIM) and a self-calibrated convolution enhancement module (Self-Calibrated Enhance Block, SCEB), and the process can be described as:

[0014] F i =EGIM i (F i-1)i=1,2,3,...,N

[0015] F DF =SCEB(F N )

[0016] EGIM i (·) represents the i-th EGIM block, which has the ability to enhance the global information induction and make the entire training process more stable. i A self-calibrated convolution enhancement module SCEB is added to the end of the block, and this design is also consistent with SwinIR.

[0017] Step 2.2: For the Enhanced Global Information Module (EGIM), an MLP channel attention is added on the basis of the traditional SwinIR shift window attention.

[0018] Step 2.3: For the Self-Calibrated Enhance Block (SCEB), the global perception range of SwinIR is extended by improving its convolution operation. Specifically, residual connections are added to the traditional convolutional layers of SwinIR to make the entire training smoother. In addition, the self-calibrated convolutional network calculations are performed in parallel, so that the network can utilize as much pixel information involved in the super-resolution process as possible.

[0019] Beneficial effects of the present invention:

[0020] 1. In the displacement window attention mechanism of SwinIR, the present invention adds an MLP-based channel attention mechanism, which can introduce additional channel information for window attention, so that more pixel information can be used in the super-resolution process. Different from the traditional channel attention method, we adopt the MLP structure so that all image information comes from the image itself, which minimizes the interference of human subjective factors and improves the quality of the reconstructed image.

[0021] 2. Self-calibration convolution is used to replace the traditional convolution in SwinIR. Self-calibration convolution is a technology used to enhance the performance of convolutional neural networks. The basic idea is to use the self-calibration mechanism to enable the convolution operation to better capture and integrate the information in the feature map. Specifically, the self-calibration convolution divides the input feature map into two parts, performs a self-calibration operation on one part, and then recombine the result with the other part to form the final feature map output. The self-calibration convolution enables the network to use more pixel information in the super-resolution process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with the accompanying drawings.

[0023] Figure 1 A flow chart of the image super-resolution network method provided by the present invention;

[0024] Figure 2 is a network structure diagram of the present invention;

[0025] Figure 3 It is a rendering of the visualization tool, which represents SwinIR from left to right; the MLP channel attention network structure is added to SwinIR; the SCEB network structure is added to SwinIR; and the improved network structure rendering of the present invention.

[0026] Figure 4 This is a diagram of the super-resolution method of the improved network of the present invention and the comparison of sota performance. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 - Figure 4 As shown, an image super-resolution method for improving SwinIR global modeling includes the following steps:

[0029] Step 1: Shallow feature extraction; there is a 3X3 traditional convolution to map the input feature dimension to a higher dimension. Specifically, given a low-resolution image input I LR ∈R H×W×Cin , we first use a shallow feature extraction network H SF (·) to extract shallow features F0∈R H×W×C , which is defined as F0=H SF (I LR ), where C in Represents the low-resolution input channel, and C represents the channel after mapping by the shallow feature extraction network; shallow feature extraction maps the input dimension to a higher dimension, which helps to capture information such as texture, edge, and structure in the image, thereby providing richer input features for the early Transformer model, which has a significant positive impact. In addition, the early convolution operations help stabilize and optimize the performance of the network.

[0030] Step 2: Transfer the shallow feature F0 as input to the deep feature extraction network H DF(·), and obtain the deep feature F DF , that is, F DF =H DF (F0),

[0031] Step 3: Use the residual cascade to fuse the shallow and deep features and use them as the input of the upsampling reconstruction module to obtain the final output of the SR network. The process can be described as: SR =H upsample (F0+F DF ), where H upsample (·) represents the upsampling reconstruction module, which is composed of convolution and pixel shuffle to fuse the acquired feature information and upsample to achieve the reconstruction effect.

[0032] Step 2 includes the following steps:

[0033] Step 2.1: H DF (·) That is, the deep feature extraction network contains N enhanced global information modules (Enhance Global Information Module, EGIM) and a self-calibrated convolution enhancement module (Self-Calibrated Enhance Block, SCEB). The process can be described as:

[0034] F i =EGIM i (F i-1 )i=1,2,3,...,N

[0035] F DF =SCEB(F N )

[0036] EGIM i (·) represents the i-th EGIM block, which has the ability to enhance the global information induction and make the entire training process more stable. i A self-calibrated convolution enhancement module SCEB is added to the end of the block, and this design is also consistent with SwinIR.

[0037] Step 2.2: For the Enhance Global Information Module (EGIM), an MLP channel attention is added on the basis of the traditional SwinIR shift window attention; the unique MLP structure of this method effectively eliminates redundant information by ensuring that all information comes directly from the image itself without being affected by any external artificial factors. Therefore, it produces a more discriminative and robust feature representation. In addition, the channel attention can add additional channel information to the window shift attention, so that the mixed attention has a more robust global information generalization ability, activating more pixel information involved in the super-resolution process.

[0038] Step 2.3: For the Self-Calibrated Enhance Block (SCEB), it expands the global perception range of SwinIR by improving its convolution operation. Specifically, residual connections are added to the traditional convolution layer of SwinIR to make the entire training smoother. In addition, the self-calibrated convolutional network calculations are performed in parallel, so that the network can capture as much pixel information involved in the super-resolution process as possible.

[0039] The present invention provides an embodiment, such as Figure 1 As shown, it is a common flow chart of image super-resolution network method, including:

[0040] Input, shallow feature extraction, deep feature extraction, upsampling and output. In the field of image super-resolution, the input is usually photos from public datasets, such as Set5, Set14, Div2K, Urban100 and Manga109. Shallow feature extraction usually uses a 3×3 convolution kernel to map the dimension of the input image to a higher feature dimension. Deep feature extraction is the most critical part of super-resolution network design. The main difference in network architecture is reflected in this part. The upsampling process usually uses a combination of convolution and pixel-shuffle to ultimately generate a high-resolution output.

[0041] The present invention improves the classic SwinIR super-resolution network so that it can involve more pixel ranges in the image super-resolution process to achieve better performance and visual effects, specifically including:

[0042] The SwinIR model has been praised for its breakthrough performance. It introduces an innovative sliding window attention mechanism. This architecture significantly reduces computational overhead while solving the problem of limited information exchange between patches. Therefore, SwinIR has become a benchmark model for various super-resolution tasks. However, by using the super-resolution process visualization tool LAM, it is found that the pixel range involved in the super-resolution process of SwinIR is very limited, such as Figure 3 The baseline in .

[0043] Therefore, the present invention uses MLP-mixer and self-calibrated convolution (SCEB) to expand and optimize SwinIR, so that the improved network can expand the pixel range used in the super-resolution process, more specifically as follows:

[0044] The MLP-mixer module of the present invention consists of two main components, such as Figure 2 (c) shows: MLP feature extraction (MLPFE) module and channel attention multi-layer perceptron (MLPCA). The MLPFE module extracts features through two multi-layer perceptrons and activation functions. This method effectively eliminates redundant information and enhances feature representation by ensuring that all information comes directly from the image itself without any external influence. Therefore, it generates more discriminative and robust feature representations, thereby improving model performance and generalization ability, while significantly reducing computational complexity and the number of parameters. In the MLPCA module, the features extracted by MLPFE are first subjected to global average pooling to obtain a global feature representation for each channel; subsequently, a series of fully connected layers, activation functions, and Sigmoid functions further process the features obtained from global pooling to learn the weights of each channel; this process allows the model to adaptively select and weight features from different channels, thereby more effectively capturing key information in the image; finally, the learned weights are multiplied with the original input of each pixel for feature fusion, so that the fused pixels can capture the global information features guided by channel attention. For the input feature F in ∈R H×W×C , firstly passes through the multi-layer perceptron feature extraction module (MLPFE) to obtain the intermediate feature F0∈R H×W×C Then, the channel attention weight W∈R is obtained through the multi-layer perceptron channel attention module (MLPCA) H×W×C Finally, the input feature F in Multiply by weight W to get the final output F out ∈R H×W×C , the calculation process of the MLP-mixer module can be defined as:

[0045] F0=MLPFE(F in )W=MLPCA(F0)

[0046] F out =F in ×W

[0047] pass Figure 3It can be clearly seen from the W / O SCEB in the figure that the MLP-mixer module proposed in the present invention significantly improves the pixel range involved in the super-resolution process. In addition, the present invention also conducts a quantitative analysis on the public dataset Urban100. The results show that after adding the MLP-mixer module, the PSNR and SSIM indicators are significantly higher than SwinIR.

[0048] In addition, the present invention uses self-calibrated convolution (SCEB) to replace all traditional 3X3 convolutions in the deep feature extraction network in SwinIR. Figure 2 As shown in (d), the present invention expands the global perception range of SwinIR by improving its convolution operation. Specifically, we add residual connections to the original convolution layer, which helps the stability of network training, and perform self-calibration convolution network calculations in parallel so that the network can capture as much pixel information involved in the super-resolution process as possible. Specifically, the output feature X of RSMCB is first convolved in parallel with the convolution activation module CAM(·) and the self-calibration attention module SEA(·), and then the outputs of the two are convolved by 1×1 convolution to obtain the final feature output X out , the output at this time contains the global information obtained from the self-calibrated attention module. The whole process can be defined as

[0049] X CAM =CAM(X)+XX SEA =SEA(X)+X

[0050] X out =conv(cat(X CAM ,X SEA ))

[0051] CAM(·) consists of three convolution functions and two activation functions, X CAM represents the feature output of the convolutional activation module (CAM), X SEA represents the feature output of the self-calibrated attention module (SEA), cat represents the concatenation operation, and X out Represents the final output feature. The core of SEA is the SCA module, whose structure is as follows: Figure 2 As shown in (e), the input feature X in ∈R B×C×H×W , first divided into two equal parts according to the number of channels, X1∈R B×C / 2×H×W and X2∈R B×C / 2×H×W, this division ensures that each part contains the same number of channels, thus providing a balanced feature representation for subsequent operations. Subsequently, convolution operations are applied to X1 and X2 respectively to reduce the number of parameters and improve computational efficiency, and the feature representation is optimized by the activation function. After that, the Sigmoid function is applied to X1 to generate the self-attention weight, which is then multiplied with X1 to produce a feature representation modulated by the self-attention weight. This weight modulation allows the network to adaptively adjust the importance of different channels, thereby enhancing the focus on salient features. Finally, X1 and X2 are combined by element-by-element addition to obtain an aggregate feature representation, and then additional convolution operations are performed to further enhance the feature representation to obtain the final output X out ∈RB×C×H×W, the convolution operation not only integrates the self-calibration features, but also captures the cross-channel context information. The process can be described as:

[0052] X in =X1+X2 X 1out =Sigmod(X1)×X1

[0053] X out =X 1out +X

[0054] Of course, the SCA module adaptively adjusts the weights of the intermediate features, enhances the model's ability to represent the features of the input data, and enables the network to effectively understand and utilize feature information. The SCA module can utilize a wider range of pixels in the super-resolution process, thereby significantly improving the visual quality. This mechanism is very valuable for capturing key information in the data and enhancing the model's ability to represent and process input data features.

[0055] Similarly, through Figure 3 It can be clearly seen from the W / O MLP-mixer in the figure that the self-calibrated convolution (SCEB) module proposed in the present invention significantly improves the pixel range involved in the super-resolution process; in addition, the present invention also conducts a quantitative analysis on the public dataset Urban100, and the results show that after adding the SCEB module, the PSNR and SSIM indicators are significantly higher than SwinIR.

[0056] Now let's review the structure of the network of the present invention. For an input photo, a shallow feature extraction is first performed through a 3×3 convolution kernel, and then the feature map extracted from the shallow feature is used as the input of the deep feature map.

[0057] Deep feature extraction will first perform layer normalization, and then execute the sliding window attention and MLP-mixer channel attention in parallel. After that, the two branches are merged together, and layer normalization and MLP mapping are performed again. We call this structure a Swin Transformer module with MLP channel attention cascade (Residual Swin Transformerwith MLP Channel Attention Block, RSMCAB). A series of RSMCABs and the final self-calibrated convolution (SCEB) are combined into an enhanced global information module (Enhance Global Information Module, EGIM). A series of EGIMs constitute the main body of the deep features. In the final EGIM, a self-calibrated convolution is added to summarize information.

[0058] The obtained deep features and shallow features are used as input to the upsampling module, and upsampling is performed using pixel-shuffle to finally obtain the output. This is the whole process of the super-resolution network proposed by the present invention.

[0059] like Figure 4 As shown, our method is compared with several state-of-the-art (SOTA) models, including EDSR, RDN, RCAN, NLRN, RNAN, SAN, OISR-RK3, IGNN, HAN, NLSN, SwinIR and ENLCN. Our model performs well in various mainstream test sets, especially in ×3 and ×4 tasks, and also performs well in ×2 tasks, which highlights the superiority of our network. It is worth noting that our training dataset is DIV2K, so the comparison with the SOTA model is based on the results trained on DIV2K, where the best performance is marked in red and the second best performance is marked in blue. We propose that the improved performance is significantly higher than other SOTA-level super-resolution networks.

[0060] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An image super-resolution method based on improved SwinIR global modeling, characterized in that: The following steps are involved: Step 1: Shallow feature extraction; there is a 3X3 traditional convolution to map the input feature dimension to a higher dimension. Specifically, given a low-resolution image input I LR ∈R H×W×Cin , first use the shallow feature extraction network H SF (·) to extract shallow features F0∈R H×W×C , which is defined as F0=H SF (I LR ), where C in represents the low-resolution input channel, and C represents the channel after being mapped by the shallow feature extraction network. Step 2: Transfer the shallow feature F0 as input to the deep feature extraction network H DF (·), and obtain the deep feature F DF , that is, F DF =H DF (F0), Step 3: Use residual connection to fuse shallow and deep features and use them as the input of upsampling reconstruction module to get the final output of SR network. The process can be described as: SR =H upsample (F0+F DF ), where H upsample (·) represents the upsampling reconstruction module, which is composed of convolution and pixel shuffle to fuse the acquired shallow and deep feature information and upsample to achieve the reconstruction effect.

2. The image super-resolution method according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: H DF (·) That is, the deep feature extraction network contains N enhanced global information modules (Enhance Global Information Module, EGIM) and a self-calibrated convolution enhancement module (Self-Calibrated EnhanceBlock, SCEB), and the process can be described as: F i =EGIM i (F i-1 )i=1,2,3,...,N F DF =SCEB(F N ) EGIM i (·) represents the i-th EGIM block, which has the ability to enhance the global information induction and make the entire training process more stable. i A self-calibrated convolution enhancement module SCEB is added to the end of the block, and this design is also consistent with SwinIR. Step 2.2: For the Enhanced Global Information Module (EGIM), an MLP channel attention is added on the basis of the traditional SwinIR shift window attention. Step 2.3: For the Self-Calibrated Enhance Block (SCEB), the global perception range of SwinIR is extended by improving its convolution operation. Specifically, residual connections are added to the traditional convolutional layers of SwinIR to make the entire training smoother. In addition, the self-calibrated convolutional network calculations are performed in parallel, so that the network can utilize as much pixel information involved in the super-resolution process as possible.

Citation Information

Cited By

  • Scanning electron microscope image super-resolution reconstruction method based on space-frequency double-domain collaboration

    CN122472985A