Lightweight image super-resolution method based on efficient attention module

Abstract

Single image super-resolution aims at recovering a high-resolution image from a low-resolution image, and is an important and challenging task in computer vision. With wide application of streaming media and high-definition equipment, development of an efficient and high-precision super-resolution method becomes an important subject. A traditional deep learning method generally faces relatively high calculation overhead although the image reconstruction performance is remarkably improved, and the application of the traditional deep learning method on resource-limited equipment is limited. In order to solve the problem, the invention provides a lightweight super-resolution model based on a Transform structure. An efficient multi-scale attention module (PEMA) and a simplified feed-forward network (SAB) are designed, so that the calculation complexity is effectively reduced, and the model performance is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and mainly relates to image super-resolution tasks, and specifically is a lightweight super-resolution image restoration method based on a Metaformer structure, which is used for efficient image super-resolution tasks on resource-constrained devices. Background Art

[0002] Single Image Super-Resolution (SISR) is an important technology to restore high-resolution (HR) images from low-resolution (LR) images. It is widely used in medical image processing, remote sensing image analysis, video enhancement, security monitoring and other fields. The core challenge of SISR is to reconstruct more high-frequency details from limited pixel information, which places extremely high demands on the feature modeling ability of the algorithm.

[0003] In recent years, deep learning technology has made breakthrough progress in the field of computer vision, promoting the performance of multiple tasks including image classification, object detection and image super-resolution. Among them, the convolutional neural network (CNN)-based model achieves excellent reconstruction effects through hierarchical feature extraction. As the network depth increases, the model can extract richer features and significantly improve image quality. However, deep CNN models are usually accompanied by exponential growth in computational complexity and memory requirements, which limits their application in resource-constrained devices (such as mobile devices or embedded systems). In addition, deep network stacking may cause problems such as gradient vanishing, increasing the difficulty of model optimization.

[0004] To overcome the limitations of CNN, the self-attention mechanism was introduced into image processing tasks. Visual Transformer (ViT) has demonstrated powerful capabilities in global feature modeling by modeling long-distance dependencies between features. However, since the computational complexity of the attention mechanism is proportional to the square of the input resolution, ViT places extremely high demands on hardware resources when processing high-resolution images, limiting its promotion in real-time applications.

[0005] To solve these problems, some studies have proposed lightweight model design and efficient attention mechanism. For example, the MetaFormer architecture proposes to abstract the Transformer into a general structure, and verifies the core contribution of the architecture itself to performance through simple pooling operations, significantly reducing computational overhead; the Efficient Multi-Scale Attention (EMA) module reduces the negative impact of channel dimensionality reduction on feature extraction through cross-space learning methods, and effectively captures multi-scale features. In image classification and object detection tasks, the EMA module performs well, especially in multi-scale feature processing; YOLOv10 further combines the partial self-attention (PSA) module and the consistent dual-label assignment strategy to achieve more efficient reasoning performance in object detection tasks, while achieving a good balance between global and local feature modeling. These methods maintain excellent performance while reducing computational complexity and parameter scale through innovative architecture design and module optimization. However, most of these studies focus on image classification or object detection, and there is still room for improvement in the dual needs of global modeling and local detail recovery in super-resolution tasks. Their emergence provides a new direction for deep learning tasks in resource-constrained scenarios, especially super-resolution tasks. Therefore, how to effectively combine global and local feature modeling in super-resolution tasks to improve model performance while reducing computational overhead has become the core goal of current research. This provides an important research direction for super-resolution applications in resource-constrained devices, and also poses new challenges for model design and optimization. Summary of the invention

[0006] In order to solve the problems of high model computational complexity and high resource consumption in existing single image super-resolution tasks, while taking into account both global feature modeling and local detail reconstruction, the present invention proposes a super-resolution model based on lightweight multi-scale attention, which improves image reconstruction effect while reducing computational overhead, and is particularly suitable for resource-constrained application scenarios, such as mobile devices or embedded systems.

[0007] The technical solution of the present invention includes:

[0008] 1. Lightweight design of Metaformer structure: The lightweight network architecture based on Metaformer, combined with an efficient multi-scale attention mechanism, can capture global and local features when processing low-resolution images and enhance the ability to restore image details.

[0009] 2. Parallel Efficient Multi-scale Attention Module (PEMA): This paper designs a new Parallel Efficient Multi-scale Attention Module (PEMA). The PEMA module uses parallel sub-networks to process information of different scales. By separating spatial features, it reduces the computational complexity and can efficiently capture the detailed features and global context in the image.

[0010] 3. Simplified Feedforward Network Module (SAB): In order to further reduce the computational overhead, the present invention proposes a simplified feedforward network module (Simple Attention Block, SAB). The SAB module significantly reduces the number of model parameters by replacing the traditional multi-layer perceptron (MLP) structure with a deep convolutional structure while maintaining a strong feature modeling capability.

[0011] 4. Residual connection and module stacking design: The present invention adopts appropriate module stacking and residual connection design to avoid the gradient vanishing problem caused by deep stacking while maintaining high efficiency. Experiments show that a moderate number of stacking can achieve a good balance between performance and computational efficiency.

[0012] 5. Training and optimization methods: During the training process, the present invention adopts L1 loss as the optimization target, and combines data enhancement and optimization algorithm (Adam optimizer) for model training to ensure that the model can effectively converge on multiple standard data sets and achieve a higher super-resolution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 : Schematic diagram of the architecture of the super-resolution model described in the present invention

[0014] Figure 2 :Schematic diagram of the structure of the efficient multi-scale attention module (PEMA)

[0015] Figure 3 :Schematic diagram of the simplified feedforward network module (SAB)

[0016] Figure 4 : Schematic diagram of the comparison results between the present invention and the existing super-resolution model DETAILED DESCRIPTION

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0018] Combined with Figure 1 The model architecture of the present invention is described in detail.

[0019] The super-resolution model of the present invention includes a feature extraction module of an input image, a stack of multiple PEMA modules, and a reconstruction module. After the input low-resolution image is subjected to preliminary convolution to extract features, the features will be sent to multiple PEMA modules for further cross-space feature extraction. In each PEMA module, the input features are first processed by channel segmentation and point-by-point convolution, and then divided into two parts, one of which enters an efficient multi-scale attention module for global feature modeling, and the other retains local features. Finally, the two parts are fused by element-level multiplication and sent to the next module after 1×1 convolution processing.

[0020] Combined with Figure 2 The implementation method of the efficient multi-scale attention module of the present invention is described in detail.

[0021] Figure 2 The structure of the PEMA module is shown. The success of the visual Transformer in various tasks stems from its implementation of a multi-head attention mechanism, which can effectively capture long-range dependencies in sequences. However, the existing self-attention mechanism is computationally expensive, which is not conducive to efficient super-resolution (SR) design. To address this problem, inspired by the design of efficient multi-scale attention modules EMA and PSA based on cross-space learning, and in view of the ubiquitous redundancy of attention heads, we design an efficient self-attention module PEMA for low-resolution images, using a parallel sub-network to establish short and long dependencies. This gating mechanism is used to process feature channels separately, allowing the attention module to focus on capturing the global features of the space, while the remaining channels can retain local features. Specifically, we evenly split the feature channel into two parts after the 3×3 convolution. One part is input into the self-attention module composed of the EMA module, and then the two parts are element-wise multiplied to preserve the dimension and fused through 1×1 convolution. Finally, layer normalization is used to preserve instance details and accelerate convergence.

[0022] The entire attention module process can be expressed by the following formula:

[0023] [X 0 ,X 1 ]=Split(X),

[0024]

[0025] Where Split corresponds to the channel splitting operation, PWConv represents point-wise convolution, EMA represents efficient multi-scale attention module [], λ 1 Represents a learnable scaling factor. Stands for element-wise multiplication.

[0026] Combined with Figure 3The simplified feedforward network module (SAB) of the present invention is described in detail.

[0027] Figure 3 The structure of the SAB module is shown. In the transformer, FFN is an important component, which mainly enhances the modeling ability of the model by reducing and increasing the dimensionality. However, the native MLP layer is too heavy as a lightweight super-resolution feedforward network, such as Figure 3 As shown in the figure, we use a 3×3 depth convolution as a simple feedforward layer. Also in view of the redundancy and lightweight requirements of the attention head, we design the module into a PSA structure as a feedforward network to reduce parameters and computation. Specifically, given the input feature X, X is evenly divided into two parts by channel. One part is input into the simple attention layer to obtain the attention map, and the other part retains the original feature. Then the two parts are element-wise multiplied and finally fused through a 1×1 convolution. The specific process is as follows:

[0028]

[0029] Z=λ 2 Y+X,

[0030] Where Split corresponds to the channel splitting operation, DWConv represents the depthwise convolution, and λ 2 Represents a learnable scaling factor. Stands for element-wise multiplication.

[0031] Combined with Figure 4 The final image result comparison of the present invention is described in detail.

[0032] like Figure 4 As shown in Figure 2, we visualize the visual comparison of current advanced lightweight SR methods. Experimental results show that on multiple standard datasets, our model significantly reduces the computational overhead while maintaining high reconstruction performance, outperforming the current mainstream lightweight super-resolution methods. The lightweight Transformer super-resolution model proposed in this paper not only achieves leading performance, but also has significant advantages in the consumption of computing resources, and has broad application prospects, especially in practical applications with limited resources.

[0033] Main references

[0034] MetaFormer《Metaformer is actually what you need for vision》

[0035] Yu W,Luo M,Zhou P,et al.Metaformer is actually what you need forvision[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition.2022:10819-10829.

[0036] YOLOv10《Yolov10:Real-time end-to-end object detection》

[0037] Wang A,Chen H,Liu L,et al.Yolov10:Real-time end-to-end objectdetection[J].arXiv preprint arXiv:2405.14458,2024.

[0038] EMA《Efficient multi-scale attention module with cross-spatiallearning》Ouyang D,He S,Zhang G,et al.Efficient multi-scale attention modulewith cross-spatial learning[C] / / ICASSP 2023-2023IEEE International Conferenceon Acoustics,Speech and Signal Processing(ICASSP).IEEE,2023:1-5.

Claims

1. A lightweight super-resolution image restoration method based on Transformer structure, characterized in that: The following steps are involved: a) A lightweight network architecture based on Transformer is adopted to restore high-resolution images from low-resolution images through an efficient multi-scale attention module (PEMA) and a simplified feedforward network module (SAB); b) The efficient multi-scale attention module (PEMA) processes information of different scales through parallel sub-networks, separates spatial features to reduce computational complexity, and can efficiently capture detailed features and global context in the image; c) The simplified feedforward network module (SAB) adopts a deep convolutional structure to replace the traditional multi-layer perceptron (MLP) structure, thereby reducing the number of model parameters while maintaining a strong feature modeling capability; d) During the training process, L1 loss is used as the optimization target, and data enhancement and optimization algorithms are combined for model training. The whole process can be expressed as: F d =MUST(F) Among them, F d It is the high-frequency feature used for final recovery. PEAB is the intermediate feature extraction module. At the same time, a long residual structure is added at the end to introduce the initial feature F. Finally, the final super-resolution image is obtained through the reconstruction module, which can be expressed as: I SR =f R (F d +F) Among them, f R It is a reconstruction module consisting of 3×3 convolution and shuffle layers.

2. The lightweight super-resolution image restoration method based on Transformer structure according to claim 1 is characterized in that: The efficient multi-scale attention module (PEMA) includes: a) Channel segmentation operation, which divides the input features into two parts; b) A portion of it is input into an efficient multi-scale attention module for global feature modeling; c) the other part retains local features; d) The two features are fused through element-wise multiplication, processed using 1×1 convolution, and then sent to the next module. We designed an efficient self-attention module for low-resolution images, using a parallel sub-network to establish short and long dependencies. This gating mechanism is used to process feature channels separately, allowing the attention module to focus on capturing global features of the space, while the remaining channels can retain local features. Specifically, we evenly divide the feature channel into two parts after the 3×3 convolution. One part is input into the self-attention module composed of EMA modules, and then the two parts are element-wise multiplied to maintain the dimension and fused through 1×1 convolution. Finally, layer normalization is used to preserve instance details and accelerate convergence. The process of the entire attention module can be expressed by the following formula: [X0, X1] = Split(X), Where Split corresponds to the channel splitting operation, PWConv represents point-wise convolution, EMA represents the efficient multi-scale attention module, and λ1 represents a learnable scaling factor. Stands for element-wise multiplication.

3. The lightweight super-resolution image restoration method based on Transformer structure according to claim 1 is characterized in that: The simplified feedforward network module (SAB) comprises: a) Process the input features through 3×3 depth convolution; b) splitting the processed features into two parts according to the channel; c) One part is input into the simple attention layer to obtain the attention map, and the other part retains the original features; d) The two parts are fused through element-wise multiplication and finally fused through 1×1 convolution. We use 3×3 deep convolution as a simple feedforward layer. Also in view of the redundancy and lightweight requirements of the attention head, we design the module into a PSA structure as a feedforward network to reduce parameters and computation. Specifically, given the input feature X, X is evenly divided into two parts by channel. One part is input into the simple attention layer to obtain the attention map, and the other part retains the original feature. Then the two parts are element-wise multiplied and finally fused through 1×1 convolution. The specific process is as follows: Z=λ2Y+X, Where Split corresponds to the channel splitting operation, DWConv represents the depthwise convolution, and λ2 represents a learnable scaling factor. Stands for element-wise multiplication.

Images