Lightweight hybrid Transform-Mama network-TransMama for super-resolution

Abstract

The invention discloses a lightweight hybrid Transform-Mama network-TransMama for super resolution, which is a lightweight hybrid Transform-Mama network, and the calculation cost brought by a self-attention mechanism is reduced by alternately using a Transform module and a Mama module. In order to balance extraction of local and global information, a Transform aggregation block (TAB) and a Mama aggregation block (MAB) are designed to enhance feature representation. In addition, a re-parameterization space gating feedforward network (RepSGFN) is provided, and the feature extraction capability of the model is further improved. A large number of experiments show that the most advanced performance (SOTA) is achieved in models of the same scale, and meanwhile an efficient model architecture is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution, and in particular to a lightweight hybrid Transformer-Mamba network (TranMamba) for super-resolution. Background Art

[0002] Single-Image Super-Resolution (SISR) aims to reconstruct high-resolution (HR) images with richer details by refining and enhancing low-resolution (LR) images. Thanks to the powerful nonlinear mapping ability and adaptive optimization characteristics of Convolutional Neural Network (CNN), CNN-based super-resolution methods show strong robustness in learning complex image data relationships [1-3]. This advantage has promoted the development of CNN-based super-resolution methods, and researchers have continuously proposed new network architectures and optimization strategies to further improve image reconstruction quality and efficiency [4-6]. Although these methods significantly improve image quality, the fixed receptive field of the convolution kernel limits their ability to capture long-range dependencies. To compensate for this deficiency, more network layers are usually stacked, which leads to increased computational costs in deep networks.

[0003] In order to better capture the global information in the image, researchers have gradually turned their attention to the Transformer model [7-9], which can model long-distance dependencies through the self-attention mechanism. In order to ensure that both local and global features can be effectively captured, recent studies have explored hybrid models that combine Transformers and CNN architectures. This combination takes advantage of the advantages of CNN in extracting local features, while leveraging the ability of Transformers to model long-distance dependencies to achieve a more comprehensive feature representation [10-12]. Although these methods have made significant progress in image restoration quality, the quadratic complexity of the self-attention mechanism remains the main bottleneck for designing lightweight models. The Mamba model [13-15] is a powerful alternative to the Transformer that can effectively model long-distance dependencies with linear complexity, thereby alleviating the high computational cost of the self-attention mechanism. However, existing studies have shown that its performance is still inferior to that of Transformers [16,17], so further exploration and optimization of the Mamba architecture is still necessary.

[0004] With the continuous development of deep learning, CNN-based super-resolution (SR) methods have gradually become mainstream. However, due to the fixed size of the convolution kernel, traditional CNNs often find it difficult to achieve a global receptive field. In contrast, the outstanding performance of the Transformer model in natural language processing (NLP) has inspired its application in the field of computer vision

[19] . The success of the Transformer is mainly attributed to its self-attention mechanism, which can overcome the limitations of the local receptive field and effectively capture global information and long-distance dependencies. In image restoration tasks such as super-resolution and denoising, the Transformer also demonstrates its advantages. The self-attention mechanism enables the Transformer to transcend the limitations of traditional methods and accurately capture global information, thereby achieving more accurate high-resolution image reconstruction in these tasks [9,10,12 ,20] .SwinIR [7] By introducing a moving window mechanism

[19] , overcoming the limitation of fixed windows and making full use of the advantages of Transformer. This method significantly improves the performance of image restoration tasks without significantly increasing the computational cost. Lu et al.

[11] designed a lightweight hybrid model architecture ESRT, which effectively reduced the computational cost and GPU memory usage by combining the advantages of lightweight CNN and Transformer networks. DAT

[12] achieved the latest state-of-the-art (SOTA) performance in super-resolution tasks by using adaptive interaction modules to fuse local and global features and alternating between spatial and channel attention modules. NGSwin [8] introduced N-Gram context into Swin Transformer, enhancing the model's ability to capture the relationship between adjacent windows, thereby enhancing SwinIR's performance in capturing global information and reconstructing image details. CAMixerSR

[10] introduced a content-aware model that combines the advantages of convolution in processing simple context with the ability of deformable window attention to process sparse textures, achieving dynamic feature extraction. This method performs well in super-resolution tasks and significantly improves the model's ability to increase image resolution.

[0005] State space model (SSM) is a mathematical model used to describe dynamic systems and is widely used in fields such as control theory and signal processing. Although SSM has strong memory and the ability to capture long-range dependencies in time series data, its application in deep learning is limited by certain theoretical and computational complexities [21,22]. Linear state space layer (LSSL) provides a unified framework that combines the advantages of recurrent neural networks (RNNs), convolutional neural networks (CNNs), and continuous time models (CTMs), so that SSM can be potentially applied to deep learning by considering these models as special cases of LSSL [21,23]. Structured state space sequence model (S4) reparameterizes the structured state matrix into low-rank and standard parts. This decomposition reduces the computational and memory requirements of LSSL and solves its numerical instability problem, making it feasible to apply SSM in deep learning

[23] . In order to achieve efficient parallel computing of SSM, S5

[24] introduced multi-input and multi-output SSM and achieved excellent performance in long sequence modeling tasks by utilizing efficient parallel scanning operations. In order to solve the efficiency problem of Transformer in processing long sequence tasks, Gu et al.

[13] designed the Mamba model based on S4 and other SSM. Mamba achieves efficient computing and solves the long-range dependency problem by introducing a selection mechanism and hardware-aware parallel algorithm. In order to adapt the Mamba model to visual tasks, recent research has focused on visual state space models and has made significant progress [15,17]. Although Mamba performs well in processing long-range dependencies, its performance still lags behind the Transformer model. However, due to Mamba's advantage in computational efficiency, the exploration of hybrid models combining Transformer and Mamba has become an important research direction [16,17]. Summary of the invention

[0006] The present invention provides a lightweight hybrid Transformer-Mamba network for super-resolution - TranMamba, which achieves the latest state-of-the-art (SOTA) in single image super-resolution tasks while maintaining an efficient model architecture.

[0007] The technical solution adopted by the present invention is:

[0008] A lightweight hybrid Transformer-Mamba network for super-resolution, TranMamba, is designed to learn an end-to-end mapping function F(·) that transforms a low-resolution image I LR Zoom in to high resolution image

[0009]

[0010] Wherein, θ represents a learnable parameter, and the overall structure of the network is divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer;

[0011] Feature extension layer: The extension layer is mainly used to enhance the representation ability of the model. By increasing the channel dimension of the input image, it can capture more feature information and provide richer input data for subsequent feature extraction. The feature extension layer is formulated as:

[0012] F s =Conv3(I LR ) (2)

[0013] Among them, Conv3(·) represents a 3×3 convolution kernel, F s Represents the shallow features extracted by the feature expansion layer;

[0014] Feature extraction layer: In order to extract the shallow features F from the input s These features are then used to generate deep features F through the feature extraction layer. d ; The feature extraction layer is formulated as:

[0015]

[0016] As shown in the above equation, the feature extraction layer consists of the feature extraction function F ext (·) and residual connections; where represents the i-th hybrid Transformer-Mamba block, HTMB, and LN(·) represents LayerNorm

[25] , σ r (·) represents the ReLU activation function, and ConvB(·) represents a convolutional block containing three convolutional layers and two activation functions;

[0017] Reconstruction layer: In the image reconstruction layer, deep features F d Upsample to restore the high-resolution image:

[0018]

[0019] Among them, UP s (·) represents sub-pixel convolution [3], s is the magnification factor; during training, the L1 loss function is used.

[27] As a cost function, its optimization objective can be calculated as:

[0020]

[0021] Where N is the batch size, and These are low-resolution and high-resolution images respectively.

[0022] The HTMB module is formed by alternating the Transformer aggregation module TAB and the Mamba aggregation module MAB, and the MAB module is built based on the VSSM;

[0023] The structural formula of HTMB is:

[0024] F HTMB (·)=F MAB (F TAB (·)) (6)

[0025] Among them, F TAB (·) indicates Transformer Aggregation Block, TAB, Transformer Aggregation Block, F MAB (·) represents the Mamba aggregation block MAB, Mamba Aggregation Block; both modules adopt the MetaFormer architecture

[29] ; these two modules are unified by F AB (·) indicates that it is expressed as:

[0026]

[0027] Among them, F in and F h Represent the input features and hidden layer features respectively; F TM represents the Token mixer, and RepSGFN represents the reparameterized spatially gated feedforward network;

[0028] The Token mixer F TM A dual-branch architecture is adopted in the g (·) and a local branch F l (·), these two branches work together to extract comprehensive features; F TM The structural formula of is defined as follows:

[0029] F TM (·)=F g (·)·CA(F l )+F l (·)·SA(F g ) (8)

[0030] Where SA(·) and CA(·) represent the spatial attention module and channel attention module respectively; F l(·) is a re-parameterized convolution block; the global branch F g (·) has two different forms:

[0031]

[0032] Among them, F VSSM represents the visual state space module, F Swin Represents the self-attention module in Swin Transformer.

[0033] The visual state space module F VSSM It adopts the multi-layer perceptron MLP module structure and combines it with the gating mechanism. VSSM The architecture of consists of two branches, denoted as F1(·) and F2(·), as follows:

[0034]

[0035] Where DWConv(·) represents a 3×3 deep convolutional layer; SSM(·) represents a two-dimensional selective scanning module (SS2D); σ s (·) denotes SiLU (Swish) activation function.

[0036] The re-parameterized spatial gated feedforward network is to add a re-parameterized convolution block to the spatial gated feedforward network SGFN; the formula is as follows:

[0037]

[0038] Among them, σ g (·) represents the GeLU activation function; Split(·) represents the hidden feature F h Divide into two equal parts, namely F h1 and F h2 ; F RepConv (·) represents a reparameterized convolution operation.

[0039] TranMamba, proposed in this paper, is a lightweight hybrid Transformer-Mamba architecture for single image super-resolution (SISR). TranMamba effectively balances model efficiency and the ability to capture local and global information. The image reconstruction effect benefits from the following three key design innovations. First, in order to strike a balance between extracting global information and computational efficiency, TranMamba adopts a strategy of alternating between Transformer and Mamba modules. This strategy performs well in natural language processing (NLP) tasks while reducing the computational complexity brought by the self-attention mechanism and maintaining good performance

[16] . Second, inspired by the adaptive interaction module

[12] , designed the Transformer Aggregation Block (TAB) and the Mamba Aggregation Block (MAB) to enhance the local feature extraction capability of TranMamba. Specifically, while using the self-attention module and the state-space module to capture long-distance dependencies, the present invention introduces a reparameterized convolution block (RepConv) to focus on the extraction of local features.

[18] Finally, a reparameterized spatial-gate feed-forward network (RepSGFN) was designed based on SGFN to further enhance the model’s feature representation capabilities. With these innovative designs, TranMamba has reached the state-of-the-art (SOTA) in performance, such as Figure 1 As shown, compared with other state-of-the-art (SOTA) methods, the present invention achieves a higher PSNR value with fewer parameters. The effects of the present invention are summarized as follows:

[0040] 1. The TranMamba network proposed in this paper is a lightweight hybrid Transformer-Mamba architecture for single image super-resolution (SISR). TranMamba uses Transformer and Mamba alternately.

[0041] module, reducing the computational complexity associated with the self-attention mechanism without compromising performance.

[0042] 2. The present invention designs a hybrid Transformer-Mamba Block (HTMB), which consists of a Transformer aggregation block (TAB) and a Mamba aggregation block (MAB) to enhance the local feature extraction capability of TranMamba. In addition, a reparameterized spatial gated feedforward network (RepSGFN) is developed by introducing a reparameterized convolution (RepConv), which improves the feature representation capability of the model.

[0043] 3. The present invention has conducted sufficient experiments, and the results show that TranMamba achieves the latest technical level (SOTA) in single image super-resolution tasks while maintaining an efficient model architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of the comparison between PSNR and parameter quantity on the Manga109×4 dataset of the present invention;

[0045] Figure 2 Schematic diagram of the structure of the hybrid Transformer-Mamba network (TranMamba) of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the hybrid Transfomer-Mamba module of the present invention;

[0047] Figure 4 Middle: (a) is a schematic diagram of the architecture of the visual state space module (VSSM); (b) is a schematic diagram of the two-dimensional selective scanning (SS2D);

[0048] Figure 5 It is a schematic diagram of qualitative comparison between the present invention and the most advanced 4x super-resolution method on Urban100 dataset;

[0049] Figure 6 Comparative analysis of the performance and complexity of different models: (a) comparison of average execution time of different models; (b) comparison of MACs of different models; (c) comparison of memory consumption of different models; These calculations of running time, MACs and memory consumption are all based on the Manga109 dataset (GT image size: 816×1164). DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and experiments.

[0051] First, detailed description of the present invention:

[0052] The present invention will be described in detail below from the aspects of the network structure of the present invention, the specific structure of the hybrid Transformer-Mamba module (HTMB), the structure of the visual state space module (VSSM) and the reparameterized spatial gated feedforward network (RepSGFN) integrated in the model, etc.

[0053] 1. Network structure

[0054] Figure 2 (a) shows a lightweight hybrid Transformer-Mamba network for super-resolution, TranMamba, which aims to learn an end-to-end mapping function F(·) to transform a low-resolution image I LR Zoom in to high resolution image

[0055]

[0056] Among them, θ represents the learnable parameter. The overall structure of TranMamba can be divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer.

[0057] Feature extension layer: The extension layer is mainly used to enhance the representation ability of the model by increasing the channel dimension of the input image, enabling it to capture more feature information and provide richer input data for subsequent feature extraction.

[0058] F s =Conv3(I LR ) (2)

[0059] Among them, Conv3(·) represents a 3×3 convolution kernel, F s Represents shallow features extracted by the feature expansion layer.

[0060] Feature extraction layer: In order to extract the shallow features F from the input s These features are then used to generate deep features F through the feature extraction layer. d :

[0061]

[0062] As shown in the above equation, the feature extraction layer consists of the feature extraction function F ext (·) and residual connection. Among them, represents the i-th hybrid Transformer-Mamba block, Hybrid Transformer-Mamba Block, HTMB, LN(·) represents LayerNorm

[25] , σ r(·) represents the ReLU activation function

[26] , and ConvB(·) represents a convolutional block containing three convolutional layers and two activation functions.

[0063] Reconstruction layer: In the image reconstruction layer, deep features F d Upsample to restore the high-resolution image:

[0064]

[0065] Among them, UP s (·) represents sub-pixel convolution [3] , s is the magnification factor. During the training process, the L1 loss function is used

[27] As a cost function, its optimization objective can be calculated as:

[0066]

[0067] Where N is the batch size, and These are low-resolution and high-resolution images respectively.

[0068] 2. Hybrid Transfomer-Mamba Module

[0069] Although Mamba

[13] The performance of the module is slightly lower than that of the Transformer module

[16] , but it maintains a high efficiency and can effectively handle long-distance dependencies. Considering the impact of the quadratic computational complexity of self-attention in the Transformer on the model efficiency, the present invention designs a MAB based on VSSM [15, 28]. This module is used alternately with TAB to form HTMB. This design can take advantage of Mamba's efficiency and ability to handle long-distance relationships, while also giving full play to the Transformer's powerful ability in feature extraction. Figure 2 As shown in (b), the structure of HTMB can be formulated as:

[0070] F HTMB (·)=F MAB (F TAB (·)) (6)

[0071] Among them, F TAB (·) represents the Transformer aggregation module, F MAB (·) indicates the Mamba aggregation module. Figure 3 (a) and Figure 3 As shown in (b), both modules use the MetaFormer architecture

[29] . For convenience, these two modules are uniformly referred to as F AB (·) indicates that it is expressed as:

[0072]

[0073] Among them, F in and F h Represent the input features and hidden layer features respectively. TM Represents a Token mixer [29,30] , while RepSGFN represents the reparameterized spatially gated feedforward network.

[0074] In order to effectively capture long-range dependencies and local features, DAT

[12] Inspired by this, the present invention is in the Token mixer F TM A dual-branch architecture is adopted in the g (·) and a local branch F l (·), these two branches work together to extract comprehensive features. TM The structure can be formally defined as follows:

[0075] F TM (·)=F g (·)·CA(F l )+F l (·)·SA(F g ) (8)

[0076] Among them, SA(·) and CA(·) represent the spatial attention module and channel attention module respectively.

[12] . F l (·) is a re-parameterized convolutional block [18 ,31 ,32] ,like Figure 3 (d) shows the global branch F g (·) has two different forms:

[0077]

[0078] Among them, F VSSM represents the visual state space module, F Swin Represents the self-attention module in Swin Transformer

[19] .

[0079] 3. Visual State Space Model

[0080] Theory: A state-space model (SSM) is a mathematical framework for representing the dynamic behavior of a system in terms of state variables. These equations are mathematically expressed as follows:

[0081]

[0082] The discretized matrix and The calculation is as follows:

[0083]

[0084] Where k represents the length of the sequence and Δ represents the sampling interval.

[0085] Based on the recursive relationship of the state space model, the output y k It can be expressed in convolution form:

[0086]

[0087] Among them, the symbol * represents the convolution operation, and the SSM convolution kernel K is expressed as:

[0088]

[0089] Visual State Space Model VSSM: VSSM inherits the network structure characteristics of Mamba and adopts a multi-layer perceptron (MLP) module structure

[33] , and combined with a gating mechanism [34,35] .like Figure 4 As shown in (a), F VSSM The architecture of consists of two branches, denoted as F1(·) and F2(·), which are described in detail as follows:

[0090]

[0091] Among them, DWConv(·) represents a 3×3 deep convolutional layer

[36] ; SSM(·) represents the two-dimensional selective scanning module (SS2D); σ s (·) represents SiLU (Swish) activation function [35,37] .

[0092] Since SSM was originally designed for NLP tasks, many studies [15,17,28] When the input feature F in When adapted to visual tasks, the 2D feature map is flattened into a 1D sequence before being input into the VSSM. However, this flattening process reduces the model's ability to capture spatial structural information. To address this limitation, VMamba

[28] The proposed SS2D adopts a cross-scanning strategy to improve the performance of Mamba in visual tasks. The present invention also adopts this method in model design. Figure 4 As shown in (b), SS2D arranges pixels in four directions (up, down, left, and right) in forward and reverse directions to generate four sequences with different orders. Each sequence is based on the selective state space model (s6)

[13] The processing is done and the outputs are merged to get the final result.

[0093] 4. Reparameterized Spatial Gated Feedforward Network

[0094] Feed-Forward Network (FFN)

[38] It is a key component in the Transformer model. It makes up for the shortcomings of the self-attention mechanism in nonlinear fitting ability by adding a linear transformation layer, thus enhancing the representation ability of the model.

[39] Spatial Gated Feedforward Network (SGFN)

[12] This is further improved by introducing a gating mechanism that compresses channel redundancy while selectively focusing on different spatial locations, similar in function to the attention mechanism.

[13] In order to enhance the spatial information extraction capability and improve the model's ability to capture local features, the present invention adds a re-parameterized convolution block to SGFN. This addition provides a more flexible network structure, thereby improving the model's ability to process spatial information and better extract important local features. The detailed process is as follows:

[0095]

[0096] Among them, σ g (·) represents the GeLU activation function; Split(·) represents the hidden feature F h Divide into two equal parts, namely F h1 and F h2 ; F RepConv (·) represents a reparameterized convolution operation.

[0097] like Figure 3 As shown in (d), the re-parameterized convolution block of the present invention contains two branches. During inference, these parameters are merged into a single convolution operation, thereby reducing computational cost while enhancing the representation capability of the model. The merging process includes three transformation steps: BatchNormalization fusion, sequential convolution fusion, and multi-branch fusion. The fusion process of the present invention follows the detailed

[18] The method proposed in .

[0098] 2. Experimental verification of the present invention

[0099] 1. Experimental setup

[0100] Dataset and evaluation criteria: Consistent with other supervised SISR methods, this paper uses DF2K in the experimental setting [41,42]The dataset is used to train the model. The dataset generates low-resolution images by bicubic interpolation and contains 3,550 images in total, of which 3,450 are used for training and 100 are used for verification. The present invention uses peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM)

[43] To evaluate the model performance and compare it with other models on five standard benchmark datasets: Set5

[44] 、Set14

[45] 、BSD100

[46] 、Urban100

[47] and Manga109

[48] .

[0101] Implementation details: The proposed TranMamba model consists of 7 HTMB modules, and the feature dimensions of both input and output are set to 64. All convolution operations in the RepConv module are depthwise separable convolutions, and the dimension expansion factor of the linear mapping layer in VSSM and RepSGFN is 2. The image is randomly cropped to a size of 48×48 for input to the model, and the mini-batch size used is 8. The model is trained using the Adam optimizer, and the initial learning rate is set to 1×10 -3 , the beta parameter is set to (0.9, 0.99), and the total number of training iterations is 2×10 6 The learning rate is dynamically adjusted through the cosine annealing scheduler with a scheduling cycle of 2,000,000 times. The training process is implemented on the NVIDIA GeForce RTX 3060 GPU based on the PyTorch framework.

[0102] Table 1 Quantitative comparison results of the proposed method with the current state-of-the-art methods on public benchmark datasets. The best and second-best results are in bold and Underline - indicates that the item is not provided in the original paper.

[0103]

[0104] 2. Benchmark Results

[0105] As shown in Table 1, the present invention conducts a comparative analysis of several state-of-the-art lightweight super-resolution (SR) models including CAMixerSR, MambaIR, etc. The analysis covers the model parameters and performance indicators of ×2 and ×4 super-resolution tasks, and is tested based on five benchmark datasets. The results show that the TranMamba model has significant advantages over other models in terms of model efficiency and multi-scale SR capabilities. For example, TranMamba outperforms MambaIR in SR performance while maintaining a smaller model size. Although TranMamba is significantly better than the DAT model in performance, it does not show a significant advantage in model size, which indicates that there is still room for further optimization of the network structure. Figure 5 The visual SR results of TranMamba on the Urban100 dataset are presented, highlighting its advantages in restoring image details and textures. Overall, TranMamba's outstanding performance in terms of PSNR and parameter metrics proves its effectiveness in improving image quality and highlights its competitive advantage among lightweight SR solutions.

[0106] Model Complexity Comparison: We evaluate the complexity of different models by comparing their parameters, runtime, and multiply-accumulate operations (MACs). To ensure a fair evaluation, we only include models that provide pre-trained versions and can be reliably executed. For runtime comparison, we run each model 10 times on the Manga109 dataset and calculate the average. Figure 6 (a) shows the running time comparison of different models. The size of the circle indicates the number of parameters (for a detailed comparison of parameters, see Figure 1 ). The results show that the proposed TranMamba model has superior operational efficiency while maintaining high PSNR. Compared with other Transformer and Mamba-based models, TranMamba has fewer parameters, highlighting its advantages in lightweight SR tasks. In addition, Figure 6 As shown in (b), the computational complexity of TranMamba is comparable to that of the CNN-based SR model, but it surpasses the Transformer-based model in SR quality and achieves SOTA performance. This highlights the effectiveness of the present invention in balancing computational efficiency and excellent performance. Figure 6 As shown in (c), TranMamba outperforms in some aspects but has slightly higher memory consumption compared to other models. This suggests that further optimization may help reduce its memory requirements.

[0107] 3. Ablation experiment

[0108] Table 2. Effect of hybrid Transformer-Mamba architecture on model performance, GPU memory usage during training, and number of parameters. All models are trained from scratch with the same hyperparameter settings.

[0109]

[0110] Table 3. Comparative analysis of the computational complexity of the model by structural reparameterization. “Acts” represents the total number of elements in all “Conv2d” outputs, while “Mem” represents the memory consumption.

[0111]

[0112] Effectiveness of the Hybrid Transformer-Mamba Architecture: Table 2 shows the impact of different model architectures on performance, parameter size, and GPU memory usage on the ×4 super-resolution task. The pure Mamba architecture performs poorly in terms of PSNR and parameter size (mainly in the VSSM mapping layer), but its memory usage is relatively low, reflecting its advantage in execution efficiency. In contrast, the pure attention model improves performance and has fewer parameters, but memory usage increases significantly. This is due to the quadratic growth in computational complexity and memory requirements of self-attention. Finally, the hybrid Transformer-Mamba architecture maintains memory usage and parameter size at an intermediate level between the two while further improving PSNR. This shows that the hybrid architecture can effectively balance low memory usage and parameter size while enhancing super-resolution performance.

[0113] Effectiveness of Reparameterized Convolution Blocks: In TranMamba, the RepConv block is integrated into AggregationBlock and RepSGFN. This block adopts a multi-branch structure to enhance the model's ability to capture spatial features during training, and merges all parameters into a single convolutional layer during the inference phase to ensure high computational efficiency. Figure 3 The comparison of model complexity before and after reparameterization is shown, proving that the reparameterization technique significantly reduces the computational complexity while maintaining the image restoration quality.

[0114] Conclusion

[0115] This paper introduces TranMamba, a lightweight hybrid Transformer-Mamba architecture for single image super-resolution (SISR). By alternating the use of Transformer and Mamba modules, TranMamba effectively reduces the computational complexity of the self-attention mechanism. In addition, the Transformer Aggregation Block (TAB) and Mamba Aggregation Block (MAB) are designed to balance the extraction of global and local information, and the feature extraction ability of the model is further enhanced by the re-parameterized spatial gated feedforward network (RepSGFN). Extensive experiments show that TranMamba achieves state-of-the-art performance in SISR tasks while maintaining a lightweight structure. Although TranMamba alleviates some of the complexity of the Transformer module through its hybrid architecture, the design may still face complexity trade-offs in certain scenarios. Future research can explore the possibility of further simplifying the model structure.

[0116] References

[0117] [1]Dong C, Loy CC, He K, et al.Image super-resolution using deepconvolutional

[0118] networks[J].IEEE transactions on pattern analysis and machineintelligence,2015,

[0119] 38(2):295-307.

[0120] [2]Dong C, Loy CC, Tang X. Accelerating the super-resolutionconvolutional neural

[0121] network[C] / / Computer Vision–ECCV 2016:14th European Conference,Amsterdam,

[0122] The Netherlands,October 11-14,2016,Proceedings,Part II 14.SpringerInternational

[0123] Publishing,2016:391-407.

[0124] [3]Shi W,Caballero J,Huszár F,et al.Real-time single image and videosuper-resolution

[0125] using an efficient sub-pixel convolutional neural network[C] / / Proceedings ofthe IEEE

[0126] conference on computervision and pattern recognition.2016:1874-1883.

[0127] [4]Hui Z,Gao X,Yang Y,et al.Lightweight image super-resolution withinformation

[0128] multi-distillation network[C] / / Proceedings of the 27thacminternational conference

[0129] on multimedia.2019:2024-2032.

[0130] [5]Liu J,Tang J,Wu G.Residual feature distillation network forlightweight image

[0131] super-resolution[C] / / Computer vision–ECCV 2020workshops:Glasgow,UK,August

[0132] 23–28,2020,proceedings,part III 16.Springer International Publishing,2020:41-55.

[0133] [6]Li Z,Liu Y,Chen X,et al.Blueprint separable residual network forefficient image

[0134] super-resolution[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2022:833-843.

[0135] [7]Liang J,Cao J,Sun G,et al.Swinir:Image restoration using swintransformer[C] / / Proceedings of the IEEE / CVF international conference oncomputer vision.2021:1833-1844.

[0136] [8]Choi H,Lee J,Yang J.N-gram in swin transformers for efficientlightweight image super-resolution[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition.2023:2071-2081.

[0137] [9]Zhou Y,Li Z,Guo C L,et al.Srformer:Permuted self-attention forsingle image super-resolution[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:12780-12791.

[0138]

[10] Wang Y,Liu Y,Zhao S,et al.CAMixerSR:Only Details Need More"

[0139] Attention"[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2024:25837-25846.

[0140]

[11] Lu Z,Li J,Liu H,et al.Transformer for single image super-resolution[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:457-466.

[0142]

[12] Chen Z,Zhang Y,Gu J,et al.Dual aggregation transformer for imagesuper-resolution[C] / / Proceedings of the IEEE / CVF international conference oncomputer vision.2023:12312-12321.

[0143]

[13] Gu A,Dao T.Mamba:Linear-time sequence modeling with selectivestate spaces.

[0144] arXiv 2023[J].arXiv preprint arXiv:2312.00752.

[0145]

[14] Dao T,Gu A.Transformers are SSMs:Generalized models and efficientalgorithms through structured state space duality[J].arXiv preprint arXiv:2405.21060,2024.

[0146]

[15] Guo H,Li J,Dai T,et al.Mambair:A simple baseline for imagerestoration with state-space model[J].arXiv preprint arXiv:2402.15648,2024.

[0147]

[16] Lieber O,Lenz B,Bata H,et al.Jamba:A hybrid transformer-mambalanguage model[J].arXiv preprint arXiv:2403.19887,2024.

[0148]

[17] Hatamizadeh A,Kautz J.MambaVision:A Hybrid Mamba-TransformerVision Backbone[J].arXiv preprint arXiv:2407.08083,2024.

[0149]

[18] Ding X,Zhang X,Han J,et al.Diverse branch block:Building aconvolution as an inception-like unit[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition.2021:10886-10895.

[0150]

[19] Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.

[0151]

[20] Zhang A,Ren W,Liu Y,et al.Lightweight image super-resolution withsuperpixel token interaction[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:12728-12737.

[0152]

[21] Gu A,Johnson I,Goel K,et al.Combining recurrent,convolutional,andcontinuous-time models with linear state space layers[J].Advances in neuralinformation processing systems,2021,34:572-585.

[0153]

[22] Fu D Y,Dao T,Saab K K,et al.Hungry hungry hippos:Towards languagemodeling with state space models[J].arXiv preprint arXiv:2212.14052,2022.

[0154]

[23] Gu A,Goel K,RéC.Efficiently modeling long sequences withstructured state spaces[J].arXiv preprint arXiv:2111.00396,2021.

[0155]

[24] Smith J T H,Warrington A,Linderman S W.Simplified state spacelayers for sequence modeling[J].arXiv preprint arXiv:2208.04933,2022.

[0156]

[25] Ba J L,Kiros J R,Hinton G E.Layer Normalization[J].2016.DOI:10.48550 / arXiv.1607.06450.

[0157]

[26] Nair V,Hinton G E.Rectified linear units improve restrictedboltzmann machines[C] / / Proceedings of the 27th international conference onmachine learning(ICML-10).2010:807-814.

[0158]

[27] Zhao H,Gallo O,Frosio I,et al.Loss functions for imagerestoration with neural networks[J].IEEE Transactions on computationalimaging,2016,3(1):47-57.

[0159]

[28] Liu,Y.,Tian,Y.,Zhao,Y.,Yu,H.,Xie,L.,Wang,Y.,Ye,Q.,Liu,Y.:Vmamba:Visual state space model.arXiv preprint arXiv:2401.10166(2024).

[0160]

[29] Yu W,Luo M,Zhou P,et al.Metaformer is actually what you need forvision[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:10819-10829.

[0161]

[30] Huang Z,Zhang Z,Lan C,et al.Adaptive frequency filters asefficient global token mixers[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:6049-6059.

[0162]

[31] Ding X,Guo Y,Ding G,et al.Acnet:Strengthening the kernelskeletons for powerful cnn via asymmetric convolution blocks[C] / / Proceedingsof the IEEE / CVF international conference on computer vision.2019:1911-1920.

[0163]

[32] Ding X,Zhang X,Ma N,et al.Repvgg:Making vgg-style convnets greatagain[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2021:13733-13742.

[0164]

[33] Tolstikhin I O,Houlsby N,Kolesnikov A,et al.Mlp-mixer:An all-mlparchitecture for vision[J].Advances in neural information processing systems,2021,34:24261-24272.

[0165]

[34] Shi D.TransNeXt:Robust Foveal Visual Perception for VisionTransformers[C] / / Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2024:17773-17783.

[0166]

[35] Ramachandran P,Zoph B,Le Q V.Swish:a Self-Gated ActivationFunction[J].2017.DOI:10.48550 / arXiv.1710.05941.

[0167]

[36] Howard A G.Mobilenets:Efficient convolutional neural networks formobile vision applications[J].arXiv preprint arXiv:1704.04861,2017.

[0168]

[37] Hendrycks D,Gimpel K.Gaussian error linear units(gelus)[J].arXivpreprint arXiv:1606.08415,2016.

[0169]

[38] Vaswani A.Attention is all you need[J].Advances in NeuralInformation Processing Systems,2017.

[0170]

[39] Wei G,Zhang Z,Lan C,et al.Activemlp:An mlp-like architecture withactive token mixer[J].arXiv preprint arXiv:2203.06108,2022.

[0171]

[40] Zhang X,Zeng H,Guo S,et al.Efficient long-range attention networkfor image super-resolution[C] / / European conference on computer vision.Cham:Springer Nature Switzerland,2022:649-667.

[0172]

[41] Agustsson E,Timofte R.Ntire 2017 challenge on single image super-resolution:Dataset and study[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition workshops.2017:126-135.

[0173]

[42] Lim B,Son S,Kim H,et al.Enhanced deep residual networks forsingle image super-resolution[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition workshops.2017:136-144.

[0174]

[43] Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:fromerror visibility to structural similarity[J].IEEE transactions on imageprocessing,2004,13(4):600-612.

[0175]

[44] Bevilacqua M,Roumy A,Guillemot C,et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[J].2012.

[0176]

[45] Zeyde R,Elad M,Protter M.On single image scale-up using sparse-representations[C] / / Curves and Surfaces:7th International Conference,Avignon,France,June 24-30,2010,Revised Selected Papers 7.Springer Berlin Heidelberg,2012:711-730.

[0177]

[46] Martin D,Fowlkes C,Tal D,et al.A database of human segmentednatural images and its application to evaluating segmentation algorithms andmeasuring ecological statistics[C] / / Proceedings eighth IEEE internationalconference on computer vision.ICCV 2001.IEEE,2001,2:416-423.

[0179]

[47] Huang J B,Singh A,Ahuja N.Single image super-resolution fromtransformed self-exemplars[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2015:5197-5206.

[0180]

[48] Matsui Y,Ito K,Aramaki Y,et al.Sketch-based manga retrieval usingmanga109 dataset[J].Multimedia tools and applications,2017,76:21811-21838。

Claims

1. A lightweight hybrid Transformer-Mamba network for super-resolution - TranMamba, characterized in that The network aims to learn an end-to-end mapping function F(·) that transforms the low-resolution image I LR Zoom in to high resolution image Wherein, θ represents a learnable parameter, and the overall structure of the network is divided into three parts: feature expansion layer, feature extraction layer and image reconstruction layer; Feature extension layer: The extension layer is mainly used to enhance the representation ability of the model. By increasing the channel dimension of the input image, it can capture more feature information and provide richer input data for subsequent feature extraction. The feature extension layer is formulated as: F s =Conv3(I LR ) (2) Among them, Conv3(·) represents a 3×3 convolution kernel, F s Represents the shallow features extracted by the feature expansion layer; Feature extraction layer: In order to extract the shallow features F from the input s These features are then used to generate deep features F through the feature extraction layer. d ; The feature extraction layer is formulated as: As shown in the above equation, the feature extraction layer consists of the feature extraction function F ext (·) and residual connections; where represents the i-th hybrid Transformer-Mamba block, HTMB, and LN(·) represents LayerNorm [25] , σ r (·) represents the ReLU activation function, and ConvB(·) represents a convolutional block containing three convolutional layers and two activation functions; Reconstruction layer: In the image reconstruction layer, deep features F d Upsample to restore the high-resolution image: Among them, UP s (·) represents sub-pixel convolution [3] , s is the magnification factor; during the training process, the L1 loss function [27] is used as the cost function, and its optimization objective can be calculated as: Where N is the batch size, and These are low-resolution and high-resolution images respectively.

2. A lightweight hybrid Transformer-Mamba network for super-resolution - TranMamba according to claim 1, characterized in that: The HTMB module is formed by alternating the Transformer aggregation module TAB and the Mamba aggregation module MAB, and the MAB module is built based on the VSSM; The structural formula of HTMB is: F HTMB (·)=F MAB (F TAB (·)) (6) Among them, F TAB (·) indicates Transformer Aggregation Block, TAB, Transformer Aggregation Block, F MAB (·) indicates Mamba aggregation block MAB, Mamba Aggregation Block; both modules use MetaFormer architecture [29] ; Unify these two modules with F AB (·) indicates that it is expressed as: Among them, F in and F h Represent the input features and hidden layer features respectively; F TM represents a Token mixer, and RepSGFN represents a reparameterized spatially gated feedforward network; The Token mixer F TM A dual-branch architecture is adopted in the g (·) and a local branch F l (·), these two branches work together to extract comprehensive features; F TM The structural formula of is defined as follows: F TM (·)=F g (·)·CA(F l )+F l (·)·SA(F g ) (8) Where SA(·) and CA(·) represent the spatial attention module and channel attention module respectively; F l (·) is a re-parameterized convolution block; the global branch F g (·) has two different forms: Among them, F VSSM represents the visual state space module, F Swin Represents the self-attention module in Swin Transformer.

3. A lightweight hybrid Transformer-Mamba network for super-resolution - TranMamba according to claim 2, characterized in that: The visual state space module F VSSM It adopts the multi-layer perceptron MLP module structure and combines it with the gating mechanism. VSSM The architecture of consists of two branches, denoted as F1(·) and F2(·), as follows: Where DWConv(·) represents a 3×3 deep convolutional layer; SSM(·) represents a two-dimensional selective scanning module (SS2D); σ s (·) denotes SiLU (Swish) activation function.

4. A lightweight hybrid Transformer-Mamba network for super-resolution - TranMamba according to claim 2, characterized in that: The re-parameterized spatial gated feedforward network is to add a re-parameterized convolution block to the spatial gated feedforward network SGFN; the formula is as follows: Among them, σ g (·) represents the GeLU activation function; Split(·) represents the hidden feature F h Divide into two equal parts, namely F h1 and F h2 ; F RepConv (·) represents a reparameterized convolution operation.

Images