Single image super-resolution reconstruction method based on channel fusion self-attention mechanism

Abstract

The application provides a single image super-resolution reconstruction method based on a channel fusion self-attention mechanism, and a light image super-resolution network based on the channel fusion self-attention mechanism, which comprises a shallow feature extraction module, a channel fusion self-attention module, a dense feature fusion module and an image reconstruction module. The channel fusion self-attention is a new linear self-attention method. The traditional self-attention mechanism generates new pixel features by weighting and summing similar features in adjacent positions in the image space, while the channel fusion self-attention generates new channel features by fusing image region features in different channels. In addition, the image super-resolution network of the application reduces the calculation complexity of the network by the channel fusion self-attention and the light convolution module, so that the network has sufficient lightness and flexibility.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a single-image super-resolution reconstruction method based on a channel fusion self-attention mechanism. Background Technology

[0002] Images, as one of the primary mediums for humans to acquire information, contain a wealth of digital information and have crucial applications in fields such as medicine, remote sensing, and surveillance. However, due to factors such as digital image acquisition equipment and environment, the acquired output images often suffer from insufficient resolution, blurriness, and missing details, rendering them unusable directly. Therefore, to address these issues in the image acquisition process, super-resolution (SR) image reconstruction technology has emerged. Under existing hardware conditions, obtaining high-resolution images at low cost through software-based super-resolution reconstruction technology has significant research value in the field of image processing.

[0003] Image super-resolution (SR) reconstruction is a low-level computer vision task that reconstructs high-resolution (HR) images from low-resolution (LR) images. SR has wide applications and has attracted considerable attention in both academia and industry. In recent years, convolutional neural networks (CNNs) have demonstrated enormous potential in SR tasks, and numerous CNN-based SR models have achieved significant results and possess high practical value.

[0004] Dong et al. first proposed SRCNN[7], which learns an end-to-end mapping from LR images to HR images through a CNN containing only three convolutional layers. Then, VDSR[8] and DRCN[9] further improved SR performance by learning larger networks through residual learning and recursive learning, respectively. By adopting residual learning and recursive learning strategies, DRRN

[11] achieved better performance with fewer parameters. MemNet

[12] was proposed to address the long-term dependency problem by mining persistent memory. In these methods, the original LR image is bicubic upsampled to the scale of the HR image before being fed into the network. To improve SR speed, most new SR models take the original LR image as input and improve spatial resolution through deconvolution or subpixel convolution at the end of the network

[14] . Unlike other SR methods, LapSRN

[10] reconstructs SR images by progressively increasing the image resolution and predicting subband residuals of the HR image. SRResNet

[15] and EDSR

[16] proposed SR models by stacking a series of residual blocks to improve SR performance. In particular, EDSR achieves performance improvements by modifying the residual blocks by removing the batch normalization (BN) layer.

[0005] To improve the performance of image processing tasks and restore high-frequency details, the aforementioned convolutional neural network-based image reconstruction (SR) models often employ very deep network structures, leading to a large consumption of parameters and computational resources, making them unsuitable for real-world scenarios. Therefore, designing lightweight networks that balance model size and reconstruction performance has become a key research area in image processing and is of great significance for future development.

[0006] Hui et al. proposed IDN

[13] to progressively extract long and short path features and extract more useful information for SR reconstruction. Based on IDN, IMDN

[19] proposed multiple distillation and contrast-aware channel attention mechanisms and won the AIM 2019 image super-resolution challenge. Liu et al. proposed RFDN

[20] , which introduced feature distillation connections and shallow residual blocks for fast SR with fewer parameters than IMDN.

[0007] Inspired by the human visual system’s ability to automatically focus on important regions, attention mechanisms are designed to concentrate on the most informative parts of the input signal. Recently, some works have introduced attention mechanisms

[23] into SR tasks. Zhang et al. proposed RCAN

[24] to focus on the most important channels by introducing a channel attention mechanism into a simplified residual block. Magid et al. proposed DFSA

[25] to predict attention maps of features in the frequency domain using a matrix multispectral channel attention mechanism. Liu et al. proposed an enhanced spatial attention (ESA) module

[20] to effectively utilize local spatial information with fewer parameters. In addition, nonlocal attention mechanisms designed to capture long-range spatial information have been studied. Methods such as NLRN

[26] , RNAN

[27] , CSNLN

[28] , and ENLCN

[29] have introduced nonlocal attention to achieve performance improvements. Recently, models such as [30-32] have introduced self-attention-based Transformers to further improve SR performance. Self-attention mechanisms are designed to encode long-range dependencies and capture global interactions and can be regarded as a special case of nonlocal attention mechanisms. In particular, Liang et al. proposed SwinIR

[30] based on Swin Transformer

[33] to achieve excellent SR performance.

[0008] In addition, collaborations have used a variety of attention mechanisms to improve SR results. Dai et al. proposed SAN

[34] to refine features using nonlocal attention and second-order channel attention. Niu et al. proposed HAN

[35] , which not only learns the channel and spatial correlations of features in each layer by using channel attention and spatial attention, but also introduces layer attention to explore the correlations between layers.

[0009] The references related to this invention include:

[0010] [1] Radu Timofte, Eirikur Agustsson, Luc Van Gool, MingHsuan Yang, and LeiZhang. “Ntire 2017 Challenge on Single Image Super-Resolution: Methods andResults,” in Proceedings of the IEEE Conference on Computer Vision and PatternRecognition Workshops, pages 114–125, 2017.

[0011] [2] B.Marco, R.Aline, G.Christine, and A.Marieline, “Low-ComplexitySingle-Image Super-Resolution Based on Nonnegative Neighbor Embedding,” inBritish Machine Vision Conference (BMVC), pp.1-10, 2012.

[0012] [3] R.Zeyde, M.Elad, and M.Protter, “On Single Image Scale-Up UsingSparse-Representations,” in International Conference on Curves and Surfaces, pp.711-730, 2012.

[0013] [4] P.Arbelaez, M.Maire, C.Fowlkes, and J.Malik, “Contour Detection andHierarchical Image Segmentation,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol.33, no.5, pp.898-916, 2011.

[0014] [5]J.Huang,A.Singh,and N.Ahuja,“Single Image Super-resolution fromTransformed Self-exemplars,”in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition(CVPR),pp.5197-5206,2015.

[0015] [6]Y.Matsui,K.Ito,Y.Aramaki,A.Fujimoto,T.Ogawa,T.Yamasaki,andK.Aizawa,“Sketch-based Manga Retrieval using Manga109 Dataset,”MultimediaTools and Applications,vol.76,no.20,pp.21811-21838,2017.

[0016] [7]C.Dong,C.Chen,K.He,and X.Tang,“Image Super-Resolution Using DeepConvolutional Networks,”IEEE Transactions on Pattern Analysis and MachineIntelligence,vol.38,no.2,pp.295-307,2016.

[0017] [8 J.Kim,J.K.Lee,and K.M.Lee,“Accurate Image Super-Resolution UsingVery Deep Convolutional Networks,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition(CVPR),pp.1646-1654,2016.

[0018] [9]J.Kim,J.K.Lee,and K.M.Lee,“Deeply-Recursive Convolutional Networkfor Image Super-Resolution,”in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition(CVPR),pp.1637-1645,2016.

[0019]

[10] W.Lai,J.Huang,N.Ahuja,and M.Yang,“Deep Laplacian Pyramid Networksfor Fast and Accurate Super-Resolution,”in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition(CVPR),pp.5835-5843,2017.

[0020]

[11] Y.Tai,J.Yang and X.Liu,“Image Super-Resolution via Deep RecursiveResidual Network,”in Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition(CVPR),pp.2790-2798,2017.

[0021]

[12] Y.Tai,J.Yang,X.Liu and C.Xu,“MemNet:A Persistent Memory Networkfor Image Restoration,”in IEEE International Conference on Computer Vision(ICCV),pp.4549-4557,2017.

[0022]

[13] Z.Hui,X.Wang,and X.Gao,“Fast and Accurate Single Image Super-Resolutionvia Information Distillation Network,”in Proceedings of the IEEEConference onComputer Vision and Pattern Recognition(CVPR),pp.723-731,2018.

[0023]

[14] C.Ledig,Z.Wang,W.Shi,L.Theis,F.Huszar,J.Caballero,A.Cunningham,A.Acosta,A.Aitken,and A.Tejani,“Photo-Realistic Single Image Super-ResolutionUsinga Generative Adversarial Network,”in Proceedings of the IEEE ConferenceonComputer Vision and Pattern Recognition(CVPR),pp.105-114,2017.

[0024]

[15] T.Tong,G.Li,X.Liu,and Q.Gao,“Image Super-Resolution Using DenseSkipConnections,”in IEEE International Conference on Computer Vision(ICCV),pp.4809-4817,2017.

[0025]

[16] B.Lim,S.Son,H.Kim,S.Nah,and K.M.Lee,“Enhanced Deep ResidualNetworksfor Single Image Super-Resolution,”in Workshop of the IEEE ConferenceonComputer Vision and Pattern Recognition(CVPRW),pp.1132-1140,2017.

[0026]

[17] Chao Dong,Chen Change Loy,and Xiaoou Tang.“Accelerating theSuper-Resolution Convolutional Neural Network,”in Proceedings of theEuropeanConference on Computer Vision(ECCV),pages 391–407,2016.

[0027]

[18] Wenbo Li,Kun Zhou,Lu Qi,Nianjuan Jiang,Jiangbo Lu,and Jiaya Jia.“Lapar:Linearly-Assembled Pixel-Adaptive Regression Network for SingleImageSuper-Resolution and Beyond,”Advances in Neural Information ProcessingSystems,33:20343–20355,2020.

[0028]

[19] Zheng Hui,Xinbo Gao,Yunchu Yang,and Xiumei Wang.“LightweightImageSuper-Resolution with Information Multidistillation Network,”inProceedings of theACM International Conference on Multimedia,pages 2024–2032,2019.

[0029]

[20] Jie Liu,Jie Tang,and Gangshan Wu.“Residual Feature DistillationNetwork forLightweight Image Super-Resolution,”in Proceedings of the EuropeanConference onComputer Vision(ECCV),pages 41–55,2020.

[0030]

[21] Luo,X.,Xie,Y.,Zhang,Y.,Qu,Y.,Li,C.,Fu,Y.:Latticenet:Towardslightweightimage super-resolution with lattice block.In:Proceedings of theEuropeanConference on Computer Vision,pp.272–289(2020)

[0031]

[22] Fang,J.,Lin,H.,Chen,X.,Zeng,K.:A hybrid network of cnn andtransformer for lightweight image super-resolution.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition Workshops,pp.1103–1112(2022)

[0032]

[23] Qin,D.,Gu,X.:Single-image super-resolution with multilevelresidual attention network.Neural Computing and Applications 32(19),15615–15628(2020)

[0033]

[24] Zhang,Y.,Li,K.,Li,K.,Wang,L.,Zhong,B.,Fu,Y.:Image super-resolution using very deep residual channel attention networks.In:Proceedingsof the European Conference on Computer Vision,pp.286–301(2018)

[0034]

[25] Magid,S.A.,Zhang,Y.,Wei,D.,Jang,W.-D.,Lin,Z.,Fu,Y.,Pfister,H.:Dynamic high-pass filtering and multi-spectral attention for image super-resolution.In:Proceedings of the IEEE Conference on Computer Vision andPattern Recognition,pp.4288–4297(2021)

[0035]

[26] Liu,D.,Wen,B.,Fan,Y.,Loy,C.C.,Huang,T.S.:Non-local recurrentnetwork for image restoration.In:Proceedings of the Advances in NeuralInformation Processing Systems(2018)

[0036]

[27] Zhang,Y.,Li,K.,Li,K.,Zhong,B.,Fu,Y.:Residual non-local attentionnetworks for image restoration.arXiv preprint arXiv:1903.10082(2019)

[0037]

[28] Mei,Y.,Fan,Y.,Zhou,Y.,Huang,L.,Huang,T.S.,Shi,H.:Image super-resolution with cross-scale non-local attention and exhaustive self-exemplarsmining.In:Proceedings of the IEEE Conference on Computer Vision and PatternRecognition,pp.5690–5699(2020)

[0038]

[29] Xia,B.,Hang,Y.,Tian,Y.,Yang,W.,Liao,Q.,Zhou,J.:Efficient non-local contrastive attention for image super-resolution.arXiv preprint arXiv:2201.03794(2022)

[0039]

[30] Liang,J.,Cao,J.,Sun,G.,Zhang,K.,Van Gool,L.,Timofte,R.:Swinir:Image restoration using swin transformer.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,pp.1833–1844(2021)

[0040]

[31] Chen,H.,Wang,Y.,Guo,T.,Xu,C.,Deng,Y.,Liu,Z.,Ma,S.,Xu,C.,Xu,C.,Gao,W.:Pre-trained image processing transformer.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,pp.12299–12310(2021)

[0041]

[32] Li,W.,Lu,X.,Lu,J.,Zhang,X.,Jia,J.:On efficient transformer andimage pre-training for low-level vision.arXiv preprint arXiv:2112.10175(2021)

[0042]

[33] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10012–10022 (2021)

[0043]

[34] Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order Attention Network for Single Image Super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

[0044]

[35] Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., Shen, H.: Single Image Super-resolution via a Holistic Attention Network. In: Proceedings of the European Conference on Computer Vision, pp. 191–207 (2020)

[0045]

[36] Zhang X, Zeng H, Guo S, et al. Efficient Long-Range Attention Network for Image Super-resolution[J]. arXiv preprint arXiv:2203.06697, 2022. Summary of the Invention

[0046] The purpose of this invention is to overcome the shortcomings of existing deep learning-based single-image super-resolution reconstruction methods. Considering the high computational complexity of self-attention, a lightweight network model is constructed by designing a channel fusion self-attention mechanism. Convolutional layers extract shallow features from the reconstructed image, and then several channel fusion self-attention modules (CMSAB) are used to analyze these shallow features, extracting hierarchical features. These generated hierarchical features are concatenated and fused, added to the residual features, and finally passed through an upsampling module to obtain the reconstructed image. Based on this, this invention proposes a single-image super-resolution reconstruction method based on a lightweight image super-resolution network with a channel fusion self-attention mechanism.

[0047] In existing technologies, almost all networks are deep networks consisting of multiple convolutional layers or convolutional network modules chained together. The lightweight image super-resolution network CMSAN of this invention, with its channel fusion self-attention mechanism, is a lightweight super-resolution model constructed using channel fusion self-attention and lightweight convolution. CMSAN consists of four parts: a shallow feature extraction module, a channel fusion self-attention module (CMSAB), a dense feature fusion module, and an upsampling module. Compared to other network structures, CMSAN can efficiently collect deep features from each layer through the channel fusion self-attention module, resulting in higher resolution images with clearer details. This approach also significantly improves the network's reconstruction speed.

[0048] This invention presents a lightweight image super-resolution network based on a channel fusion self-attention mechanism. The network consists of four parts: a shallow feature extraction module, a channel fusion self-attention module, a dense feature fusion module, and an image reconstruction module. Channel fusion self-attention is a novel linear self-attention method proposed in this invention. Traditional self-attention mechanisms generate new pixel features by weighted summation of similar features in adjacent locations in the image space, while channel fusion self-attention regenerates new channel features by fusing image region features from different channels. Furthermore, the image super-resolution network of this invention reduces computational complexity through channel fusion self-attention and a lightweight convolutional module, making it sufficiently lightweight and flexible. Extensive experimental results demonstrate that the image super-resolution network of this invention outperforms existing state-of-the-art methods in terms of super-resolution image reconstruction quality, model complexity, and inference speed.

[0049] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0050] A single-image super-resolution reconstruction method based on channel fusion self-attention mechanism is characterized by using a CMSAN network as the deep network for single-image super-resolution reconstruction, directly inputting a low-resolution image Y into the CMSAN network to reconstruct a high-resolution image.

[0051] Furthermore, the CMSAN network includes a shallow feature extraction module, a multi-channel fusion self-attention module (CMSAB), a dense feature fusion module, and an upsampling module. It uses convolutional layers to extract shallow features from the low-resolution image Y, then uses the CMSAB module to analyze these shallow features, extracting hierarchical features. The generated hierarchical features are concatenated and fused, and then added to the residual features. Finally, the upsampling module yields the reconstructed high-resolution image.

[0052] Furthermore, the Channel Fusion Self-Attention Module (CMSAB) structure is defined by channel fusion self-attention and lightweight convolution;

[0053] As per the instruction manual Figure 1 As shown, traditional self-attention calculates the similarity between pixels, while channel fusion self-attention calculates the correlation between channels. This invention significantly improves model efficiency by cleverly maintaining local attention and windowing mechanisms in the channel dimension rather than the pixel space. Channel fusion self-attention can be expressed by the following formula:

[0054] Q(Softmax(K T )V)

[0055] More specifically, the present invention first calculates Softmax(K) T ) Obtain the importance of each pixel in each channel, then multiply by V on the right to obtain the inter-channel relationships. Softmax(K) T The dimension of V is dxd, and its computational complexity is O(Nd). 2 The final matrix is ​​Softmax(K). T Multiplying V by Q yields the updated features, and its computational complexity is also O(Nd). 2 It should be noted that the Softmax function is added separately to K. T The pixel space is used to compute pixel attention weights between different channels, which also helps with training convergence. Since d << N, the computational complexity of channel fusion self-attention is O(Nd). 2 This is far less than the time complexity of traditional self-attention, which is O(N). 2 d), for detailed calculation steps, please refer to [reference needed]. Figure 2 .

[0056] Channel fusion self-attention significantly improves model efficiency by calculating attention across different channel dimensions. The convolutional part uses a lightweight module: two 1x1 convolutions and one 3x3 depthwise convolution. Two Batch Normalization (BN) layers are introduced to replace the Layer Normalization (LN) layers in the standard self-attention module, improving model speed. Compared to other network structures, the channel fusion self-attention module efficiently collects multi-layer features, generating more effective and compact high-level features, resulting in higher-resolution images with clearer details.

[0057] Furthermore, when a set of training data is provided Where I is the number of image patches in the training database, Y i and X i If the images are low-resolution and high-resolution patches respectively, then the loss function is expressed as:

[0058]

[0059] in It is a reconstructed high-resolution image.

[0060] Compared with existing technologies, this invention and its preferred embodiment consider the high computational complexity of existing self-attention models and construct a lightweight network model through the design of a channel fusion self-attention module. Convolutional layers extract shallow features of the reconstructed image, and then several channel fusion self-attention modules (CMSAB) are used to extract deep features, finally reconstructing a clearer, higher-resolution image. Attached Figure Description

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0062] Figure 1 This is a schematic diagram comparing the linear self-attention of the lightweight network (CMSAN) with the channel fusion self-attention mechanism in this embodiment of the invention with the original self-attention.

[0063] Figure 2 This is a schematic diagram of the structure of a lightweight network (CMSAN) with a channel fusion self-attention mechanism in an embodiment of the present invention;

[0064] Figure 3 This is a comparison chart of the reconstruction results of a single-image super-resolution reconstruction method using a lightweight network (CMSAN) with a channel fusion self-attention mechanism in this embodiment of the invention, and other methods. Detailed Implementation

[0065] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below, along with accompanying drawings, for detailed explanation:

[0066] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0067] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0068] The purpose of this invention is to overcome the shortcomings of existing deep learning-based single-image super-resolution reconstruction methods. Due to the high computational complexity of self-attention, a lightweight network model is constructed through the design of a channel fusion self-attention module. Convolutional layers extract shallow features from the reconstructed image, and then several channel fusion self-attention modules (CMSAB) are used to analyze these shallow features, extract hierarchical features, concatenate and fuse these generated hierarchical features, add them to the residual features, and finally, an upsampling module is used to obtain the reconstructed image. Based on this, this invention proposes a lightweight network-based single-image super-resolution reconstruction method based on a channel fusion self-attention mechanism.

[0069] This invention presents a single-image super-resolution reconstruction method based on a lightweight network (CMSAN) with a channel fusion self-attention mechanism. This method differs from the original Transformer's self-attention calculation in pixel space. Linear self-attention is calculated in the channel dimension, i.e., the weights of different pixels between each channel are calculated. Then, this invention designs a channel fusion self-attention and lightweight convolutional SR model, proposing a lightweight SR model, namely the Lightweight Image Super-Resolution Network (CMSAN) with a channel fusion self-attention mechanism. Specifically, CMSAN consists of four parts: a shallow feature extraction module, a channel fusion self-attention module (CMSAB), a dense feature fusion module, and an upsampling module. First, in the shallow feature extraction module, only one convolutional layer is used to extract shallow features containing low-frequency information; then, six CMSABs are used to extract hierarchical features. Each CMSAB contains a channel fusion self-attention module and a lightweight convolutional module. The channel fusion self-attention calculates the original O(n) value by optimizing the channel dimension. 2The complexity is transformed into O(n) (linear complexity). The lightweight convolutional module contains two 1x1 convolutions and a 3x3 depthwise convolution. Then, these hierarchical features generated by CMSAB are concatenated and fused layer by layer to obtain residual features, and shallow features are added to obtain deep features. Finally, the upsampling module generates the SR result. The CMSAN of this invention can extract more effective SR features and obtain better SR results.

[0070] A single-image super-resolution reconstruction method based on a channel fusion self-attention mechanism (CMSAN) uses CMSAN as the deep network for single-image super-resolution reconstruction. First, a low-resolution image Y is directly input into the CMSAN to reconstruct a high-resolution image. CMSAN contains only a shallow feature extraction module, several channel fusion self-attention modules, a dense feature fusion module, and an upsampling module. The shallow feature extraction module extracts shallow features from the low-resolution image Y. Then, the channel fusion self-attention modules analyze these shallow features to extract hierarchical features. These generated hierarchical features are concatenated and fused, and added to the residual features to obtain deep features. Finally, the upsampling module yields the reconstructed high-resolution image.

[0071] Based on the above principles, the method of the present invention is specifically implemented as follows:

[0072] In this invention, a low-resolution image Y is directly input into a lightweight network (CMSAN) with a channel fusion self-attention mechanism, and finally a reconstructed high-resolution image is obtained.

[0073] CMSAN comprises a shallow feature extraction module, several channel fusion self-attention modules, a dense feature fusion module, and an upsampling module. The shallow feature extraction module extracts shallow features from the low-resolution image Y. Then, the channel fusion self-attention modules analyze these shallow features to extract hierarchical features. These generated hierarchical features are concatenated and fused, and then added to the residual features. Finally, the upsampling module yields the reconstructed high-resolution image.

[0074] When a set of training data is provided Where L is the number of image patches in the training database, Y i and X i If the images are low-resolution and high-resolution patches respectively, then the loss function can be expressed as:

[0075]

[0076] in It is a high-resolution image reconstructed by a single-image super-resolution reconstruction method based on a lightweight network with a channel fusion self-attention mechanism.

[0077] The following experiments were conducted based on the method of this invention:

[0078] In this embodiment of the lightweight network (CMSAN) with channel fusion self-attention mechanism, DIV2K is utilized. [1] CMSAN was trained using 800 training images from the dataset. Data augmentation was achieved through rotations of 90°, 180°, and 270°, and horizontal flipping. For testing, this invention used five benchmark datasets: Set5. [2] Set14 [3] B100 [4] Urban100 [5] and Manga109 [6] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to measure the quality of SR images. For each training mini-batch, 16 64×64 patches are randomly cropped from LR images as input. The CMSAN of this invention is trained using the Adam optimizer, β1 = 0.9, β2 = 0.999. There are a total of 1000 training epochs. The learning rate is initialized to 5e-4 and halved every 200 epochs.

[0079] Table 1

[0080]

[0081]

[0082] Table 2

[0083]

[0084] To demonstrate the effectiveness of the method of this invention, experiments were conducted to compare it with existing representative deep learning-based single-image super-resolution reconstruction methods. These methods include SRCNN. [7] FSRCNN

[17] IDN

[13] LAPAR-A

[18] IMDN

[19] and RFDN

[20] LatticeNet

[21] HNCT

[22] SwinIR

[30] and ELAN

[36] .

[0085] In the experiment of single-image super-resolution reconstruction, the quantitative performance comparison between the existing deep learning-based single-image super-resolution reconstruction methods and the method of the present invention is shown in Table 1. The evaluation indexes are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), and the values ​​are the mean values ​​of the reconstruction results of the test dataset. The unit of PSNR is dB.

[0086] As shown in Table 1, the experimental data demonstrate that the present invention (CMSAN) outperforms other super-resolution reconstruction methods, with PSNR and SSIM being optimal at 3x and 4x respectively.

[0087] Different reconstruction methods were tested on a single GeForce RTX 3090, and their inference times are shown in Table 2. Although the inference speed of the model CMSAN of this invention is slower than that of CNN-based methods, it is essentially on the same order of magnitude. Both CMSAN and SwinIR employ self-attention strategies, but CMSAN achieves a significant improvement in runtime efficiency due to the efficient design of the model in this invention, with its best inference speed being nearly 8 times faster than SwinIR.

[0088] In terms of visual reconstruction, the super-resolution reconstruction effects of various comparison methods and the method of this invention are compared, for example... Figure 3 As shown. By Figure 3 It can be seen that the single-image super-resolution reconstruction method based on the channel fusion self-attention mechanism of the present invention can reconstruct more accurate detail information.

[0089] As can be seen, by using a lightweight network module with a channel fusion self-attention mechanism, CMSAN in this invention greatly reduces the computational complexity of the self-attention module, thereby improving the efficiency of super-resolution reconstruction and enhancing the quality of the reconstructed image.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of single-image super-resolution reconstruction methods based on channel fusion self-attention mechanisms under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A single-image super-resolution reconstruction method based on channel fusion self-attention mechanism, characterized in that: Using the CMSAN network as a deep network for single-image super-resolution reconstruction, low-resolution images are reconstructed. The image was directly input into the CMSAN network and reconstructed into a high-resolution image. ; The CMSAN network includes a shallow feature extraction module, a multi-channel fusion self-attention module (CMSAB), a dense feature fusion module, and an upsampling module. It uses convolutional layers to extract low-resolution images. The shallow features are then analyzed using the CMSAB module to extract hierarchical features. These hierarchical features are then concatenated and fused, and added to the residual features. Finally, an upsampling module is used to obtain the reconstructed high-resolution image. ; The Channel Fusion Self-Attention Module (CMSAB) structure is defined by channel fusion self-attention and lightweight convolution; Channel fusion self-attention improves model efficiency by maintaining the Transformer's local attention and window mechanism movement in the channel dimension rather than the pixel space; the convolution part uses two 1x1 convolutions and one 3x3 depthwise convolution; two BN layers are introduced to replace the LN layer of self-attention to improve the model's running speed.

2. The single-image super-resolution reconstruction method based on channel fusion self-attention mechanism according to claim 1, characterized in that: When a set of training data is provided ,in The number of image patches in the training database, and If the images are low-resolution and high-resolution patches respectively, then the loss function is expressed as: in It is a reconstructed high-resolution image.

Images