Efficient image super-resolution method based on multi-branch hybrid Transform-MBMT

Abstract

The invention relates to an efficient image super-resolution method based on a multi-branch hybrid Transformer (Multi-Branch Mixer Transformer, MBMT), in the method, a unique module of a multi-branch Token Mixer (Multi-Branch Token Mixer, MBTM) is introduced, and the module aims to replace the depth by increasing the network width, so that the learning ability and the generalization ability of the model are enhanced. The method is characterized in that the method is composed of three branches: a self-attention mechanism module, a depth convolution (Depth-Width Conversion, DWConv) and an adaptive token mixer (ATM), and each branch is responsible for capturing a long-term dependency relationship, extracting local features and expanding a global information space. According to the method, an adaptive space and a channel attention mechanism are combined, so that sufficient interaction among information extracted from the three branches is ensured, and the feature representation capability of MBTM is enhanced. Benefited from the high efficiency and the shallower network structure of the MBTM, the method provided by the invention reduces the complexity while significantly improving the SR model performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution, and in particular to an efficient image super-resolution method based on a multi-branch hybrid Transformer-MBMT. Background Art

[0002] Single-Image Super-Resolution (SISR) aims to upscale low-resolution (LR) images to generate high-resolution (HR) images. Deep learning methods have made significant progress in the field of single image super-resolution (SISR), among which Transformer-based models often outperform CNN-based methods in performance. However, due to the self-attention mechanism in Transformer, it is still challenging to achieve lightweight models compared with CNN-based methods.

[0003] The success of Transformer in natural language processing has paved the way for its application in computer vision, achieving performance levels comparable to or even better than traditional convolutional neural networks (CNNs) in image processing tasks. [17 ,18 ,33] Applying Transformer in super-resolution tasks can take advantage of its advantages in capturing global information, which helps to understand the structure and content of the image to generate high-quality high-resolution (HR) images. [19-22] TTSR

[34] A texture transformer is introduced to compute the correlation between a low-resolution (LR) image and a reference (Ref) image. By using hard and soft attention modules to learn the joint features between LR and Ref, it effectively transfers the texture information of Ref into the HR image.

[26] Using Swin Transformer

[33] SWinIR is a highly efficient architecture for single image super-resolution (SISR) tasks. It uses the self-attention mechanism of Transformer to capture long-range dependencies in images and enhances feature extraction and processing efficiency through a hierarchical structure. Compared with traditional convolutional neural networks, SWinIR performs well in various super-resolution tasks. ESRT

[35] It is a lightweight architecture that combines CNN and Transformer. The model uses a feature segmentation module to split a long sequence into multiple subsequences, and then performs attention operations on these subsequences to reduce GPU memory consumption. To solve the problem of limited receptive field in SWinIR, Haram et al.

[23] The N-Gram

[36] context was introduced into SWin, and NGSWin was proposed. NGSWin expands the visible area and restores degraded pixels through sliding window self-attention interactions. LBNet

[24] introduced a recursive mechanism in Transformer, which enables the model to effectively learn global information without significantly increasing GPU memory consumption and model parameters. DAT

[22] adopts a strategy of alternating spatial and channel self-attention within the Transformer module. This method can perform feature aggregation in spatial and channel dimensions, enhance the image representation ability of the model, and significantly improve the performance of image super-resolution.

[0004] In the visual Transformer model, input features are converted into a sequence of tokens, and each token interacts with other tokens through a self-attention mechanism. This interaction process is often referred to as token mixing [17,18]. However, the self-attention mechanism causes the computational complexity to grow quadratically with the length of the sequence, resulting in increased cost, slower training and inference speeds, and the need for significantly increased computing resources. Some studies have shown [37,38] that using only MLP can achieve performance comparable to that of the Transformer, suggesting that the self-attention module can be replaced by a basic token mixer. Poolformer

[39] is a Transformer-based model that replaces the self-attention module in the Transformer with a simple spatial pooling operation as the basic token mixer. This approach enables the model to achieve performance comparable to that of the Transformer and similar MLP models. Huang et al.

[40] The token is converted to the frequency domain using Fourier transform and adaptive filtering is performed on the converted token to achieve lightweight token mixing.

[29] By recombining tokens, adaptive global information integration is achieved, achieving excellent performance under limited computational cost.

[0005] In the field of CNN-based single image super-resolution (SISR), restoration of high-resolution (HR) images usually relies on complex network architectures and a large number of parameters. However, these architectures usually require a lot of computational resources, limiting their practicality in resource-constrained environments. In order to achieve a balance between model complexity and performance, an increasing number of lightweight image super-resolution methods have been studied in recent years. [2,8-14] SRCNN[1] uses a simple convolutional architecture to achieve end-to-end SISR, highlighting the pioneering role of convolutional neural networks in super-resolution tasks. [2]By optimizing the network architecture of SRCNN, the inference speed is significantly improved while maintaining excellent super-resolution performance.

[14] The innovative use of sub-pixel convolutional layers to rearrange low-resolution (LR) features into high-resolution images effectively solves the limitations of traditional upsampling methods related to pixel-by-pixel operations. This method performs well in both visual fidelity and computational efficiency.

[30] The advent of the super-resolution algorithm has inspired the development of many lightweight SR models with residual structures, which has promoted significant progress in the field of super-resolution. [8 ,10,11,12,31,32]. In recent years, Transformer-based SR networks have gradually become a trend, and their performance is comparable to or even better than traditional CNN methods. [22 ,23,25,27].

[0006] The success of Convolutional Neural Network (CNN) in the field of super-resolution (SR) is attributed to its powerful feature extraction capability and end-to-end learning framework. Researchers have made significant progress by training CNN models to learn the mapping from LR to HR images. [1-7] Although CNN models perform well in generating high-quality SR images, the depth and complexity of these networks result in large model sizes and high computational requirements, which can be an obstacle when deployed in resource-constrained environments. To address these challenges, researchers are actively exploring the design of lightweight super-resolution architectures to reduce model size and computational cost while maintaining performance. [8-16] .

[0007] The receptive field of CNN-based SR models is mainly limited by the size and depth of the convolution kernel. These limitations restrict the model's ability to capture long-term image dependencies and may cause image distortion or blur, especially in larger images or complex scenes. To address this challenge, researchers have begun to explore the use of emerging architectures such as Transformer in super-resolution tasks. [17-22]. Unlike CNN, Transformer is not limited by the receptive field and can effectively capture global information and long-term dependencies, which is expected to improve the performance and effect of SR models. Although Transformer has made significant progress in the field of computer vision, its training and inference require a lot of computing resources, which is mainly attributed to the self-attention (SA) mechanism

[17] , whose computational complexity grows quadratically with the increase of token length. In addition, Transformer has difficulty in processing local information, which limits the performance of the model to a certain extent. For these reasons, designing lightweight Transformer-based SR models remains a challenging topic in this field [23-27]. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes an efficient image super-resolution method based on a multi-branch hybrid Transformer (MBMT), which is a lightweight SR network that significantly improves the performance of the SR model while reducing the complexity.

[0009] The technical solution adopted by the present invention is:

[0010] An efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT, the goal of this method is to learn to LR To high resolution image I HR The end-to-end mapping function F(·) is:

[0011]

[0012] Among them, θ represents the learnable parameter; and the overall structure of MBMT is divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer.

[0013] Feature expansion layer: It is mainly used to increase the channel dimension of the input image, thereby introducing more features and context information to enhance the performance of the model: Its structure is shown in formula (2):

[0014] F l =Conv3(I LR ) (2)

[0015] Among them, Conv3(·) represents a 3×3 convolution kernel, F l Represents the low-level features extracted by the expansion layer.

[0016] Feature extraction layer: In order to extract the low-level features F from the input l Further extract complex and meaningful context features, Fl is input into the feature extraction layer to generate high-level features F h , as shown in formula (3):

[0017]

[0018] As shown in formula (3), the feature extraction layer consists of the feature extraction function F ext (·) and residual connections; where represents the i-th MBTM and LN(·) represents layer normalization.

[0019] Image reconstruction layer: In the image reconstruction layer, high-level features F h Enlarge it to restore the high-resolution image; as shown in formula (4):

[0020]

[0021] Among them, UP s (·) represents sub-pixel convolution

[14] , and s is the magnification factor. During the training process, the L1 loss function is used as the cost function, and its optimization objective can be calculated as:

[0022]

[0023] The MBTM adopts the general macro architecture of MetaFormer, which is expressed as follows:

[0024]

[0025] Among them, F in , F out and F hidden They represent input features, output features and hidden layer features respectively; Τ(·) represents a multi-branch token mixer, and MLP(·) represents a multi-layer perceptron.

[0026] The multi-branch token mixer consists of three branches, expressed as follows:

[0027] T(F in )=Τ SW (F in )+Τ DW (F in )+Τ ATM (F in ) (7)

[0028] Among them, SW (·) represents the shift window Swin self-attention interaction branch, Τ DW (·) represents the 3×3 depthwise convolutional interaction branch, Τ ATM(·) represents the adaptive token mixer interaction branch.

[0029] In order to ensure the complete integration of local and global information extracted by different modules, interactions are performed between features of different branches:

[0030]

[0031] Where · represents the Hadamard product, SA(·) and CA(·) represent spatial attention and channel attention respectively, SW(·) represents the shifted window Swin self-attention interaction mixer, ATM(·) represents the adaptive token mixer, and DW(·) represents the deep convolutional layer for extracting local features:

[0032]

[0033] Among them, GELU(·) represents the Gaussian error linear unit activation function, DWConv3(·) represents a 3×3 deep convolutional layer, and BN(·) represents batch normalization.

[0034] The calculation steps of the adaptive token mixer ATM(·) are as follows:

[0035]

[0036] Among them, ATM W (·) and ATM H (·) denotes the adaptive token mixer along the horizontal and vertical directions, respectively, while P(·) represents the linear mapping layer; F H 、F W and F I Represent the recombined features along the vertical and horizontal directions and the original features respectively; then, using the learned weights [λ W ,λ H ,λ I ] Mix these three sets of features into fusion feature F fusion ; By ATM W (·) as an example, the calculation process is as follows:

[0037]

[0038] The variables i, j, c, and o represent the feature height, width, channel index, and the corresponding offset of the query feature value, respectively; O∈R H×W×C Represents the query offset matrix.

[0039] The multi-branch token mixer includes a feed-forward layer SGFN, and the SGFN adopts a self-gated activation function (Swish) structure.

[0040] Unlike other traditional Transformer architectures, the MBMT proposed in the present invention introduces a unique module called Multi-Branch Token Mixer (MBTM), which adopts a token mixing method different from SA. MBTM represents a more complex token mixing method that aims to enhance the learning and generalization capabilities of the model by increasing the network width instead of the depth. It consists of three branches: a self-attention mechanism module, a depth-wise convolution (DWConv)

[28] , and an adaptive token mixer (ATM)

[29] , each of which is responsible for capturing long-term dependencies, extracting local features, and expanding the global information space. Inspired by DAT

[22] , the present invention combines adaptive spatial and channel attention mechanisms to ensure that there is sufficient interaction between the information extracted from the three branches, thereby enhancing the feature representation capability of MBTM. Thanks to the high efficiency and shallower network structure of MBTM, the MBMT of the present invention significantly improves the performance of the SR model while reducing the complexity, such as Figure 1 The main contributions of the present invention are as follows:

[0041] 1. We propose a multi-branch token mixer that aims to capture long-term dependencies while extracting local information. In addition, we use spatial and channel attention mechanisms to interact between features at different levels, which greatly improves the super-resolution performance of the model.

[0042] 2. ATM is introduced in MBTM to achieve richer global information interaction while avoiding the excessive computational cost associated with the self-attention mechanism.

[0043] 3. Extensive experiments are conducted to demonstrate that the proposed MBMT achieves excellent image restoration quality while maintaining low complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the performance and parameter comparison between the present invention and other models on the BSD100×4 data set;

[0045] Figure 2 It is a structural schematic diagram of the MBMT model of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the multi-branch token mixer of the present invention;

[0047] Figure 4 (a) is a schematic diagram of the adaptive token mixer (ATM) structure; (b) is a schematic diagram of the ATM in the horizontal direction (ATM W );

[0048] Figure 5 Schematic diagram of qualitative comparison results of the present invention with the most advanced 4x super-resolution method on Urban100 / Manga109 dataset;

[0049] Figure 6 Comparison charts of performance and complexity analysis of different models: (a) Comparison of running time between different models; (b) Comparison of MACs between different models; (c) Pie chart of FLOPs distribution of different modules in MBMT; running time, FLOPs and MACs are all calculated on the Manga109 dataset;

[0050] Figure 7 Schematic diagram of attribution analysis for the Transformer-based super-resolution (SR) model, where the first row: GT image with wireframe; rows 2-4: show the pixel areas that contribute to the super-resolution of the wireframe area. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and experiments.

[0052] 1. Detailed description of the present invention: The following will describe the overall structure of the present invention ( Figure 2 ), its core modules Multi-Branch Token Mixer (MBTM) and Adaptive Token Mixer (ATM) introduce the present invention in detail.

[0053] 1. Overall network architecture: The overall architecture of the present invention is as follows Figure 2 As shown, the goal of MBMT is to learn LR To high resolution image I HR The end-to-end mapping function F(·) is

[0054]

[0055] Among them, θ represents the learnable parameter. The overall structure of MBMT can be divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer.

[0056] Feature expansion layer: The expansion layer is mainly used to increase the channel dimension of the input image, thereby introducing more features and context information to enhance the performance of the model.

[0057] F l =Conv3(I LR ) (2)

[0058] Among them, Conv3(·) represents a 3×3 convolution kernel, F lRepresents the low-level features extracted by the expansion layer.

[0059] Feature extraction layer: In order to extract the low-level features F from the input l Further extract complex and meaningful context features, F l is input into the feature extraction layer to generate high-level features F h :

[0060]

[0061] As shown in the above formula, the feature extraction layer consists of the feature extraction function F ext (·) and residual connection. Among them, represents the i-th MBTM and LN(·) represents layer normalization.

[0062] Image reconstruction layer: Finally, in the image reconstruction layer, the high-level features F h Upscale to restore the high-resolution image.

[0063]

[0064] Among them, UP s (·) represents sub-pixel convolution

[14] , and s is the magnification factor. During the training process, the L1 loss function

[41] is used as the cost function, and its optimization objective can be calculated as:

[0065]

[0066] 2. The multi-branch token mixer MBTM is the core component of the MBMT model. MBTM adopts the general macro architecture of MetaFormer

[39] , which can be expressed as:

[0067]

[0068] Among them, F in , F out and F hidden They represent input features, output features, and hidden layer features, respectively. Τ(·) represents a multi-branch token mixer, and MLP(·) represents a multi-layer perceptron.

[0069] In the field of computer vision, token mixers enable information (tokens) between different spatial locations in an image to interact in order to extract global features. Previous studies [39,40] have explored different token mixing methods, among which the self-attention mechanism is the most commonly used one. However, since stacking self-attention modules in deeper networks may lead to decreased inference efficiency, this paper proposes a more efficient token mixing method: a multi-branch token mixer. Figure 3As shown, this module consists of three branches, as follows:

[0070] T(F in )=Τ SW (F in )+Τ DW (F in )+Τ ATM (F in ) (7)

[0071] Among them, SW (·) represents the shift window Swin self-attention interaction branch, Τ DW (·) represents the 3×3 depthwise convolutional interaction branch, Τ ATM (·) represents the adaptive token mixer interaction branch.

[0072] In order to ensure the complete integration of local and global information extracted by different modules, interactions are performed between features of different branches:

[0073]

[0074] where · represents the Hadamard product, SA(·) and CA(·) represent spatial attention and channel attention respectively, SW(·) represents the shifted window Swin self-attention mixer

[33] , ATM(·) represents the adaptive token mixer

[29] , and DW(·) represents the deep convolutional layer used to extract local features:

[0075]

[0076] Here, GELU(·) represents the Gaussian error linear unit activation function

[42] , and DWConv3(·) represents a 3×3 deep convolutional layer.

[43] , BN(·) stands for batch normalization. The implementation details of ATM are introduced in the next section.

[0077] Multilayer Perceptron: In the architecture of the MetaFormer model, a feed-forward layer is usually attached after the token mixer. The feed-forward layer usually uses linear transformation and nonlinear activation function to enhance the nonlinear learning ability of the model. Influenced by DAT network

[22] Inspired by , we introduce SGFN as the feed-forward layer. SGFN uses the self-gated activation function (Swish)

[44] The structure of SGFN is shown in Figure 2, and deep convolution is used to reduce channel redundancy, thereby improving the reasoning efficiency of the model. Figure 3 shown.

[0078] 3. Adaptive token mixer. In the token mixer module, the present invention enriches the feature information by increasing the width of the model and enhances the performance of the entire network. In view of the fact that the self-attention token mixing method will increase the network load, the present invention adopts a more efficient mixing method, namely the adaptive token mixer. Calculate ATM(·)

[29] The steps are as follows:

[0079]

[0080] In the above equation, ATM W (·) and ATM H (·) denotes the adaptive token mixer along the horizontal and vertical directions, respectively, while P(·) represents the linear mapping layer. H 、F W and F I denote the recombined features along the vertical and horizontal directions and the original features respectively. Then, using the learned weights [λ W ,λ H ,λ I ] Mix these three sets of features into fusion feature F fusion . By ATM W (·) as an example, the calculation process is as follows:

[0081]

[0082] The variables i, j, c, and o represent the feature height, width, channel index, and the corresponding offset of the query feature value, respectively. H×W×C Represents the query offset matrix.

[0083] 2. Experimental verification of the present invention

[0084] 1. Experimental setup

[0085] Dataset and evaluation metrics: Similar to most supervised single image super-resolution (SISR) models, we use the DF2K dataset in our experiments. [47,48] To train our model. The dataset obtains the corresponding low-resolution images through bicubic downscaling. There are 3550 images in total, of which 3450 are used for training and 100 are used for verification. We use the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM)

[49] As a baseline, we evaluate the performance of our model and compare it with other models on five standard benchmark datasets: Set5

[50] , Set14

[51] ,BSD100

[52] ,Urban100

[53] , Manga109

[54] .

[0086] Implementation details: The MBMT proposed in the present invention contains 8 multi-branch token mixers (MBTMs). In each MBTM, the number of self-attention heads, channel dimensions, and dimensionality expansion factors are set to 6, 64, and 2, respectively. In ATMB, the interval of the offset generation layer is set to 1, and the dimensionality expansion factor is 2. The present invention randomly crops the image to 192×192 as the model input, and the batch size is 16. In order to enhance the performance of the model, horizontal flipping and random rotation (90°, 180°, 270°) are applied for data augmentation. The dataset is expanded to 100 times the original size. During the model training process, the initial learning rate is set to 1×10 -3 , a total of 2×10 6 The model of the present invention is implemented using the PyTorch framework and trained on an NVIDIA GeForce RTX 2070SUPER GPU.

[0087] 2. Benchmark Results

[0088] As shown in Table 1, in this experiment, MBMT is compared with the current state-of-the-art lightweight super-resolution (SR) methods. This comparative analysis covers model parameters as well as ×2 and ×4 results on five benchmark test sets. The results show that MBMT performs well in terms of model efficiency and multi-scale super-resolution capabilities. Compared with SWinIR-NG

[23] and SPIN

[25] In comparison, MBMT achieves an average PSNR improvement of 0.04 and 0.01 on the ×4 super-resolution test set. Although Transformer-based models generally outperform CNN-based SR models, they are often accompanied by larger model sizes. In contrast, the model of the present invention achieves excellent SR performance while maintaining a smaller model size. Figure 5 The visual comparison results of the proposed MBMT and other models on the ×4 test set are shown. This figure shows the superior ability of MBMT in recovering details and textures compared with other methods.

[0089] Table 1 Quantitative comparison results with the current state-of-the-art methods on public benchmark datasets. The best and second-best results are in bold and Underline - indicates that the item is not provided in the original paper.

[0090]

[0091] 2.1 Model Complexity Comparison

[0092] Comparative experiments are conducted on different models, including comparison of model parameters, running time and multiplication-addition operations (MACs). In order to ensure the fairness of the comparison, the present invention focuses on the super-resolution (SR) method that provides a pre-trained model. Figure 6 (a) shows the comparison of parameters and execution time of different SR models on the Manga109 dataset. It can be clearly seen that Transformer-based models are usually slower than CNN-based SR models. It is worth noting that among the Transformer-based models, MBMT shows certain advantages. Figure 6 The computational complexity analysis in (b) shows that the MACs of the MBMT model remain at a similar level to the CNN-based SR model. However, the proposed model excels in super-resolution (SR) quality, highlighting the effectiveness of the proposed method in balancing efficiency and excellent performance under resource-constrained conditions.

[0093] 2.2 Attribution Analysis

[0094] The present invention adopts the local attribution map (LAM)

[55] An attribution analysis is performed on three Transformer-based models: SWinIR, SWinIR-NG, and MBMT. Figure 7 As shown, the black areas in the heat map represent specific pixel areas that the network focuses on during the super-resolution process, and these pixels are crucial for generating the GT image within the wireframe. Figure 7 It can be clearly seen that the MBMT of the present invention has a significantly larger receptive field than SWinIR and SWinIR-NG. This observation shows that MBMT can utilize a wider range of pixels for image reconstruction, thereby improving the performance of super-resolution.

[0095] 3. Ablation experiment

[0096] Table 2 Ablation study of the performance of the multi-branch token mixer. All models are trained from scratch and use the same hyperparameters. FLOPs are calculated on 1280×720 GT images.

[0097]

[0098] 3.1 Effectiveness of multi-branch token mixer: In order to analyze the effectiveness of the multi-branch token mixer structure, this experiment conducted ablation experiments on each branch separately, and verified the model using the high-resolution dataset DIV2K100. Table 2 shows that the SWin+DWConv+ATM multi-branch structure achieved the highest super-resolution (SR) performance. Among the three branches, SWin has the greatest impact on SR, followed by ATM, while DWConv contributes the least. Although SWin has fewer parameters, its FLOPs is higher due to the quadratic computational complexity caused by the self-attention mechanism, which affects the efficiency of the model. Although ATM has more parameters, its computational load is mainly concentrated on the calculation of the offset, so the FLOPs are lower. As Figure 6 As shown in (c), SWin, ATM, and DWConv account for 24.08%, 8.07%, and 16.37% of the total FLOPs of the model, respectively. Combined with their impact on SR performance, this shows that the ATM branch plays an important role in achieving lightweight SR.

[0099] Table 3 Ablation study of the adaptive token mixer on model performance. All models are trained from scratch and use the same hyperparameters. MACs are calculated on 1280×720 GT images.

[0100]

[0101] 3.2 Effectiveness of Adaptive Token Mixer: Table 3 shows the results of ablation experiments on ATM, evaluating the impact of original features and adaptive token mixers along the horizontal and vertical directions on the ×4 super-resolution performance. Overall, the impact of these three factors is roughly equal. Although the parameters and model complexity of the adaptive token mixers along the horizontal and vertical directions are the same, the impact of the horizontal direction on the SR performance is slightly greater than that of the vertical direction. In contrast, the original features have a slight advantage in model size, model complexity, and SR performance. Combined with the conclusions in Table 2, it can be inferred that ATM plays a significant role in achieving a lightweight model structure.

[0102] Conclusion

[0103] The lightweight super-resolution model MBMT proposed in the present invention realizes the extraction of global and local features at the same time through a multi-branch structure design. This innovative design significantly reduces the network depth while slightly increasing the network width, laying the foundation for the lightweight super-resolution Transformer. A large number of experimental results show that MBMT has achieved SOTA performance in terms of super-resolution quality and model complexity on multiple benchmark datasets. However, considering that the self-attention branch still has a greater impact on the model performance than other branches, designing a more efficient token mixer remains an important challenge.

[0104] References

[0105] [1]Dong C, Loy CC, He K, et al.Image Super-Resolution Using DeepConvolutional

[0106] Networks[J].IEEE Trans Pattern Anal Mach Intell, 2016,

[0107] 38(2):295-307.DOI:10.1109 / TPAMI.2015.2439281.

[0108] [2]Dong C, Loy CC, Tang X.Accelerating the Super-ResolutionConvolutional Neural

[0109] Network[J].Springer,Cham,2016.DOI:10.1007 / 978-3-319-46475-6_25.

[0110] [3]Kim J, Lee JK, Lee K M.Accurate Image Super-Resolution Using VeryDeep

[0111] Convolutional Networks[J].IEEE,2016.DOI:10.1109 / CVPR.2016.182.

[0112] [4]Ledig C, Theis L, Huszar F, et al.Photo-Realistic Single Image Super-Resolution Using

[0113] a Generative Adversarial Network[J].IEEE Computer Society,

[0114] 2016.DOI:10.1109 / CVPR.2017.19.

[0115] [5]Wang X,Yu K,Wu S,et al.Esrgan:Enhanced super-resolution generativeadversarial networks[C].Proceedings of the European conference on computervision(ECCV)

[0116] workshops.2018:0-0.

[0117] [6]Tai Y,Yang J,Liu X,et al.MemNet:A Persistent Memory Network forImage Restoration[C] / / IEEE International Conference on Computer Vision.IEEEComputer Society,2017.DOI:10.1109 / ICCV.2017.486.

[0118] [7]Lim B,Son S,Kim H,et al.Enhanced Deep Residual Networks for SingleImage Super-Resolution[J].IEEE,2017.DOI:10.1109 / CVPRW.2017.151.

[0119] [8]Hui Z,Gao X,Yang Y,et al.Lightweight image super-resolution withinformation multi-distillation network[C] / / Proceedings of the 27th acminternational conference on multimedia.2019:2024-2032.

[0120] [9]Ahn N,Kang B,Sohn K A.Fast,Accurate,and Lightweight Super-Resolution with Cascading Residual Network[J].2018.DOI:10.48550 / arXiv.1803.08664.

[0121]

[10] Li Z,Liu Y,Chen X,et al.Blueprint separable residual network forefficient image super-resolution[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition.2022:833-843.

[0122]

[11] Kong F,Li M,Liu S,et al.Residual local feature network forefficient super-resolution[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2022:766-776.

[0123]

[12] Liu J,Tang J,Wu G.Residual feature distillation network forlightweight image super-resolution[C] / / Computer vision–ECCV 2020 workshops:Glasgow,UK,August

[0124] 23–28,2020,proceedings,part III 16.Springer International Publishing,2020:41-55.

[13] Lai W S,Huang J B,Ahuja N,et al.Deep laplacian pyramidnetworks for fast and accurate super-resolution[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2017:624-632.

[0125]

[14] Shi W,Caballero J,Huszár F,et al.Real-time single image and videosuper-resolution using an efficient sub-pixel convolutional neural network[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2016:1874-1883.

[0126]

[15] Tai Y,Yang J,Liu X.Image super-resolution via deep recursiveresidual network[C] / / Proceedings of the IEEE conference on computer visionand pattern recognition.2017:3147-3155.

[0127]

[16] Kim J,Lee J K,Lee K M.Deeply-recursive convolutional network forimage super-resolution[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2016:1637-1645.

[0128]

[17] Vaswani A,Shazeer N,Parmar N,et al.Attention Is All You Need[J].arXiv,

[0129] 2017.DOI:10.48550 / arXiv.1706.03762.

[0130]

[18] Dosovitskiy A.An image is worth 16x16 words:Transformers forimage recognition at scale[J].arxiv preprint arxiv:2010.11929,2020.

[0131]

[19] Wang Y,Liu Y,Zhao S,et al.CAMixerSR:Only Details Need More"

[0132] Attention"[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2024:25837-25846.

[0133]

[20] Cao J,Liang J,Zhang K,et al.Reference-based image super-resolution with deformable attention transformer[C] / / European conference oncomputer vision.

[0134] Cham:Springer Nature Switzerland,2022:325-342.

[0135]

[21] Zhou Y,Li Z,Guo C L,et al.Srformer:Permuted self-attention forsingle image super-resolution[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:12780-12791.

[0136]

[22] Chen Z,Zhang Y,Gu J,et al.Dual aggregation transformer for imagesuper-resolution[C] / / Proceedings of the IEEE / CVF international conference oncomputer vision.2023:12312-12321.

[0137]

[23] Choi H,Lee J,Yang J.N-gram in swin transformers for efficientlightweight image super-resolution[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition.2023:2071-2081.

[0138]

[24] Gao G,Wang Z,Li J,et al.Lightweight bimodal network for single-image super-resolution via symmetric CNN and recursive transformer[J].arXivpreprint arXiv:2204.13286,2022.

[0139]

[25] Zhang A,Ren W,Liu Y,et al.Lightweight image super-resolution withsuperpixel token interaction[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:12728-12737.

[0140]

[26] Liang J,Cao J,Sun G,et al.Swinir:Image restoration using swintransformer[C] / / Proceedings of the IEEE / CVF international conference oncomputer vision.2021:1833-1844.

[0141]

[27] Wang H,Chen X,Ni B,et al.Omni aggregation networks forlightweight image super-resolution[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2023:22378-22387.

[0142]

[28] Howard A G.Mobilenets:Efficient convolutional neural networks formobile vision applications[J].arXiv preprint arXiv:1704.04861,2017.

[0143]

[29] Wei G,Zhang Z,Lan C,et al.Activemlp:An mlp-like architecture withactive token mixer[J].arXiv preprint arXiv:2203.06108,2022.

[0144]

[30] He K,Zhang X,Ren S,et al.Deep residual learning for imagerecognition[C] / / Proceedings of the IEEE conference on computer vision andpattern recognition.2016:770-778.

[0145]

[31] Zhang Y,Tian Y,Kong Y,et al.Residual dense network for imagesuper-resolution[C] / / Proceedings of the IEEE conference on computer visionand pattern recognition.2018:2472-2481.

[0146]

[32] Khan F S,Khan S.Ntire 2022 challenge on efficient super-resolution:Methods and results[C] / / Conference on Computer Vision and PatternRecognition Workshops

[0147] (CVPRW).2022:1061-1101.

[0148]

[33] Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.

[0149]

[34] Yang F,Yang H,Fu J,et al.Learning texture transformer network forimage super-resolution[C] / / Proceedings ofthe IEEE / CVF conference on computervision and pattern recognition.2020:5791-5800.

[0150]

[35] Lu Z,Li J,Liu H,et al.Transformer for single image super-resolution[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:457-466.

[0152]

[36] Li H,Cai D,Xu J,et al.$N$-gram Is Back:Residual Learning ofNeural Text Generation with$n$-gram Language Model[J].arXiv preprint arXiv:2210.14431,2022.

[0153]

[37] Tolstikhin I O,Houlsby N,Kolesnikov A,et al.Mlp-mixer:An all-mlparchitecture for vision[J].Advances in neural information processing systems,2021,34:24261-24272.

[38] Touvron H,Bojanowski P,Caron M,et al.Resmlp:Feedforward networks for image classification with data-efficient training[J].IEEE transactions on pattern analysis and machine intelligence,2022,45(4):5314-5321.

[0154]

[39] Yu W,Luo M,Zhou P,et al.Metaformer is actually what you need forvision[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:10819-10829.

[0155]

[40] Huang Z,Zhang Z,Lan C,et al.Adaptive frequency filters asefficient global token mixers[C] / / Proceedings ofthe IEEE / CVF InternationalConference on Computer Vision.2023:6049-6059.

[0157]

[41] Zhao H,Gallo O,Frosio I,et al.Loss functions for imagerestoration with neural networks[J].IEEE Transactions on computationalimaging,2016,3(1):47-57.

[0158]

[42] Hendrycks D,Gimpel K.Gaussian error linear units(gelus)[J].arXivpreprint arXiv:1606.08415,2016.

[0159]

[43] Howard A G.Mobilenets:Efficient convolutional neural networks formobile vision applications[J].arXiv preprint arXiv:1704.04861,2017.

[0160]

[44] Ramachandran P,Zoph B,Le Q V.Swish:a Self-Gated ActivationFunction[J].

[0161] 2017.DOI:10.48550 / arXiv.1710.05941.

[0162]

[45] Hui Z,Wang X,Gao X.Fast and accurate single image super-resolution via information distillation network[C] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2018:723-731.

[0163]

[46] Zhang X,Zeng H,Guo S,et al.Efficient long-range attention networkfor image super-resolution[C] / / European conference on computer vision.Cham:Springer Nature Switzerland,2022:649-667.

[0164]

[47] Agustsson E,Timofte R.Ntire 2017 challenge on single image super-resolution:

[0165] Dataset and study[C] / / Proceedings of the IEEE conference on computervision and pattern recognition workshops.2017:126-135.

[0166]

[48] Lim B,Son S,Kim H,et al.Enhanced deep residual networks forsingle image super-resolution[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition workshops.2017:136-144.

[0167]

[49] Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:fromerror visibility to structural similarity[J].IEEE transactions on imageprocessing,2004,13(4):600-612.

[50] Bevilacqua M,Roumy A,Guillemot C,et al.Low-complexity single-image super-resolution based on nonnegative neighborembedding[J].2012.

[0168]

[51] Zeyde R,Elad M,Protter M.On single image scale-up using sparse-representations[C] / / Curves and Surfaces:7th International Conference,Avignon,France,June 24-30,2010,Revised Selected Papers 7.Springer Berlin Heidelberg,2012:711-730.

[0169]

[52] Martin D,Fowlkes C,Tal D,et al.A database of human segmentednatural images and its application to evaluating segmentation algorithms andmeasuring ecological statistics[C] / / Proceedings eighth IEEE internationalconference on computer vision.

[0170] ICCV 2001.IEEE,2001,2:416-423.

[0171]

[53] Huang J B,Singh A,Ahuja N.Single image super-resolution fromtransformed self-exemplars[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2015:5197-5206.

[0172]

[54] Matsui Y,Ito K,Aramaki Y,et al.Sketch-based manga retrieval usingmanga109

[0173] dataset[J].Multimedia tools and applications,2017,76:21811-21838.

[0174]

[55] Gu J,Dong C.Interpreting super-resolution networks with localattribution maps[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2021:9199-9208。

Claims

1. An efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT, characterized in that: The goal of this method is to learn LR To high resolution image I HR The end-to-end mapping function F(·) is: Among them, θ represents the learnable parameter; and the overall structure of MBMT is divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer; Feature expansion layer: It is mainly used to increase the channel dimension of the input image, thereby introducing more features and context information to enhance the performance of the model: Its structure is shown in formula (2): F l =Conv3(I LR ) (2) Among them, Conv3(·) represents a 3×3 convolution kernel, F l Represents the low-level features extracted by the expansion layer; Feature extraction layer: In order to extract the low-level features F from the input l Further extract complex and meaningful context features, F l is input into the feature extraction layer to generate high-level features F h , as shown in formula (3): As shown in formula (3), the feature extraction layer consists of the feature extraction function F ext (·) and residual connections; where represents the i-th MBTM, LN(·) represents layer normalization; Image reconstruction layer: In the image reconstruction layer, high-level features F h Enlarge it to restore the high-resolution image; as shown in formula (4): Among them, UP s (·) represents sub-pixel convolution [14] , s is the magnification factor; during the training process, the L1 loss function is used as the cost function, and its optimization goal can be calculated as:

2. The efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT according to claim 1, characterized in that: The MBTM adopts the general macro architecture of MetaFormer, which is expressed as follows: Among them, F in , F out and F hidden They represent input features, output features and hidden layer features respectively; Τ(·) represents a multi-branch token mixer, and MLP(·) represents a multi-layer perceptron.

3. The efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT according to claim 2, characterized in that: The multi-branch token mixer consists of three branches, expressed as follows: T(F in )=T SW (F in )+T DW (F in )+T ATM (F in ) (7) Among them, SW (·) represents the shift window Swin self-attention interaction branch, Τ DW (·) represents the 3×3 depthwise convolutional interaction branch, Τ ATM (·) represents the adaptive token mixer interaction branch; In order to ensure the complete integration of local and global information extracted by different modules, interactions are performed between features of different branches: Where · represents the Hadamard product, SA(·) and CA(·) represent spatial attention and channel attention respectively, SW(·) represents the shifted window Swin self-attention interaction mixer, ATM(·) represents the adaptive token mixer, and DW(·) represents the deep convolutional layer for extracting local features: Among them, GELU(·) represents the Gaussian error linear unit activation function, DWConv3(·) represents a 3×3 deep convolutional layer, and BN(·) represents batch normalization.

4. The efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT according to claim 3, characterized in that: The calculation steps of the adaptive token mixer ATM(·) are as follows: Among them, ATM W (·) and ATM H (·) denotes the adaptive token mixer along the horizontal and vertical directions, respectively, while P(·) represents the linear mapping layer; F H 、F W and F I Represent the recombined features along the vertical and horizontal directions and the original features respectively; then, using the learned weights [λ W ,λ H ,λ I ] Mix these three sets of features into fusion feature F fusion ; By ATM W (·) as an example, the calculation process is as follows: The variables i, j, c, and o represent the feature height, width, channel index, and the corresponding offset of the query feature value, respectively; O∈R H×W×C Represents the query offset matrix.

5. The efficient image super-resolution method based on multi-branch hybrid Transformer-MBMT according to claim 3, characterized in that: The multi-branch token mixer includes a feed-forward layer SGFN, and the SGFN adopts the structure of the self-gated activation function Swish.

Images