Lightweight single-image super-resolution reconstruction method based on local and global feature collaborative enhanced perception

Abstract

The invention discloses a lightweight single-image super-resolution reconstruction method based on local and global feature collaborative enhancement perception, which is used for solving the problems of insufficient utilization of high and low frequency clues in single-image super-resolution, low feature fusion efficiency and difficulty in consideration of structural consistency and visual fidelity of reconstructed images. The network adopts a double-flow heterogeneous architecture; a local branch uses multi-type differential convolution explicit coding image edge and texture prior to enhance detail characterization capability; the global branch effectively models long-range dependency and low-frequency semantic information by integrating local, cross-regional and global multi-level spatial self-attention mechanisms. The frequency sensing fusion module provided by the invention generates a channel specific space weight based on frequency characteristics, self-adaptively fuses double-branch characteristics, accurately balances structure maintenance and detail enhancement requirements in a characteristic fusion process, and effectively reduces characteristic redundancy. According to the super-resolution network, the model complexity is remarkably reduced, high-resolution images which are consistent in structure, vivid in vision and rich in details can be generated while lightweight design is kept, and an effective scheme is provided for efficient and high-performance lightweight single-image super-resolution reconstruction model design.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing and imaging technology, and specifically relates to a lightweight single image super-resolution reconstruction method based on local-global feature collaborative enhancement perception. BACKGROUND

[0002] In today's digital era, the demand for image data processing and optimization is growing. Image super-resolution reconstruction technology, as a key means to improve image quality, has received widespread attention. Traditional image super-resolution methods mainly rely on interpolation algorithms such as bilinear interpolation and bicubic interpolation. Although these methods are simple to calculate, they often result in image edge blurring, detail loss, and other problems, making it difficult to meet the demand for high-quality image reconstruction.

[0003] With the development of deep learning technology, convolutional neural networks (CNN) have gradually become the mainstream method in the field of image super-resolution reconstruction. CNN can effectively extract local features of images by learning a large number of low-resolution and high-resolution image pairs, thus achieving better super-resolution results. However, CNN has certain limitations in processing global information, and it is not sufficient to capture long-distance dependencies and global structural information in images, which to some extent limits the further improvement of its performance. In recent years, the Transformer architecture has achieved great success in natural language processing, and its powerful self-attention mechanism can effectively capture global information and long-distance dependencies. This advantage has prompted researchers to introduce Transformer into the field of image super-resolution reconstruction to make up for the shortcomings of CNN in global information processing. Image super-resolution reconstruction methods based on Transformer model the image through self-attention mechanism, which can restore better global structure and detail information of the image, thus significantly improving the super-resolution effect. However, with the improvement of model performance, the complexity and computational cost of the model also increase dramatically. In practical applications, especially for mobile devices and embedded systems in resource-constrained scenarios, these complex models face great challenges. Therefore, lightweight network design has become an important research direction in the field of image super-resolution reconstruction. Lightweight networks aim to optimize network structure, reduce parameter quantity and computational load, so that the model can run efficiently on resource-constrained devices without significantly reducing model performance. Current super-resolution reconstruction technology has made certain progress in the design of CNN and Transformer-based models, but there is still room for improvement in terms of balancing high performance and lightweight. SUMMARY

[0004] The main purpose of the present invention is to overcome the shortcomings of some of the existing technologies mentioned above, and to propose a lightweight single image super-resolution reconstruction method based on the CNN-Transformer collaborative mechanism. It aims to combine the advantages of CNN and Transformer, and at the same time achieve a balance between high performance and low computational cost through innovative network structure design and optimization strategies, so as to meet the image super-resolution reconstruction needs in different application scenarios. This method can effectively utilize the high and low frequency clues implicit in low-resolution images, achieve efficient feature fusion, and ensure the structural consistency and visual realism of the reconstruction results. This innovative method provides a new solution for the field of single image super-resolution reconstruction, effectively balancing detail recovery and global structural rationality, and has broad application prospects and important technical value.

[0005] To achieve the above objectives, the present invention adopts the following technical solutions:

[0006] A lightweight single image super-resolution reconstruction method based on local-global feature collaborative enhancement perception includes the following steps:

[0007] Step 1: Get a low-resolution image

[0008] Step 2: Shallow feature extraction: Use convolution operations to process the low-resolution image and extract shallow features. This process can capture the basic texture and structural information of the image, laying the foundation for subsequent deep feature extraction. The specific formula is:

[0009]

[0010] in, is the input low-resolution image, is a 3×3 convolution operation, It is a shallow feature map.

[0011] Step 3: Deep Feature Extraction: Use A cascade of residual local-global cross-integration modules and a convolution operation are used to extract deep features from shallow feature maps. This process aims to further explore image details, especially high-frequency details, to improve the quality of the reconstructed image. The specific extraction method includes the following sub-steps:

[0012] Step 3.1: First, split the input feature map into the first sub-feature map along the channel dimension and the second sub-feature map Then The 3x3 convolution and the multi-difference convolution cooperative calculation are simultaneously performed, and the results are added and output through an activation function. The multi-difference convolution cooperative calculation includes parallel operations of central difference convolution, axial difference convolution, vertical difference convolution and horizontal difference convolution. In the inference stage, the 3x3 convolution and the multi-difference convolution are combined into a single convolution calculation to reduce the calculation overhead. Then the output of the 3x3 convolution is added to the output of the multi-difference convolution to enhance the expression ability of the features. The maximum pooling, 1x1 convolution and activation function processing are sequentially performed. Finally, the outputs of steps (2) and (3) are spliced, and after 1x1 convolution, they are added to the original input feature map to enhance the expression ability of the features.

[0013] Step 3.2: In order to enhance the ability of the model to interact across windows, the present application draws on the idea of Swin Transformer and introduces a hybrid attention module (HAB) in the local-global cross integration module (LGCIM). HAB works cooperatively through parallel window multi-head self-attention (W-MSA) and shift window multi-head self-attention (SW-MSA) branches, explicitly capturing the structural similarity of local regions and extracting global information of feature maps. Given an input feature, HAB first divides it into two parts along the channel dimension, and inputs them into the W-MSA branch and the SW-MSA branch respectively for processing. The output feature maps of the two branches are then concatenated along the channel dimension, normalized by layer normalization (LN), and added to the original input (i.e., residual connection is added again) to obtain the intermediate feature . Next, it is fed into a multi-layer perceptron (MLP) layer. The output of the MLP is again normalized by layer normalization (LN) and added to (i.e., residual connection is added again), and finally the output feature map of the HAB is obtained . Among them, and represent the parts of the input feature map after division and input into the W-MSA and SW-MSA branches respectively. This design of the present application can significantly improve the model's perception of global information while maintaining computational efficiency, thereby achieving better performance in the image super-resolution reconstruction task.

[0014] Step 3.3: Global information guided grid attention module (GG-GAB) or structure guided global attention module (SG-GAB): These two modules aim to explicitly model the similarity across regions of an image with linear computational complexity, thereby improving the model's perception of global information while maintaining computational efficiency. The GG-GAB module focuses on global information through sparse sampling of grid points, while the SG-GAB module uses adaptive average pooling to map the structural information of an image block to a low-dimensional space, where self-attention is calculated. The following is a detailed description of these two modules:

[0015] The GG-GAB module focuses on global information through sparse sampling of grid points, and its core is to efficiently capture the similarity between different regions in the image. The specific steps are as follows: First, layer normalization is performed on the input: the input feature map is processed by layer normalization to stabilize the training process and improve the generalization ability of the model. Then, grid division is performed: the feature map is divided into multiple small blocks according to the preset grid interval, and each small block contains the information of a local region. Then, global interaction features are generated, that is, for each grid block, the maximum and average value features are extracted, and these features are spliced and passed through a convolution layer to generate global interaction features. This process can capture the global information between different grid blocks. Then, multi-head self-attention is calculated, based on the global interaction features, the attention weights of each grid block are calculated, and these weights are used to weight the value vector to obtain the attention features of each grid block. This process allows the model to pass information between different grid blocks and enhances the global perception ability. Finally, grid inverse operation and output are performed: the attention features of all grid blocks are restored to the original size and connected with the input feature map in residual, and finally processed by multi-layer perception (MLP) to obtain the output of the module. The GG-GAB module focuses on global information through sparse sampling, avoiding the problem of increasing computational complexity with the square of image size in traditional global self-attention mechanism, thus maintaining efficient computation while significantly improving the model's perception of the overall structure of the image.

[0016] The SG-GAB module maps the structural information of image blocks to a low-dimensional space through adaptive average pooling and calculates self-attention in this space, and its core is to efficiently capture global structural similarity. The specific steps are as follows: First, layer normalization is performed on the input feature map to stabilize the training process and improve the generalization ability of the model. Then, window division is performed, and the feature map is divided into multiple non-overlapping window blocks, each containing local region information. Then, multi-scale structural feature extraction is performed, and for each window block, multi-scale structural features are extracted through adaptive average pooling, which can reflect the structural information of the window block at different scales. Then, these multi-scale features are spliced and compressed to obtain the structural statistics of each window block. Global attention calculation is based on the compressed structural statistics to calculate global attention weights, and these weights are used to weight the value matrix to obtain global attention features. This process allows the model to efficiently capture global structural similarity in a low-dimensional space. Finally, the output is obtained: the calculated global attention features are connected with the input feature map in residual, and finally processed by multi-layer perception (MLP) to obtain the output of the module. The SG-GAB module efficiently calculates global attention in a low-dimensional space through adaptive average pooling and multi-scale feature extraction, thus maintaining computational efficiency while significantly improving the model's perception of the overall structure of the image.

[0017] Step 3.4: Based on the high-frequency feature map and the low-frequency feature map obtained in steps 3.1 and 3.2, first add the high-frequency feature map and the low-frequency feature map to generate an initial feature . Then split the initial feature along the channel into subgraphs, and perform a two-dimensional discrete cosine transform (2D-DCT) on each subgraph to extract frequency components. The frequency components are spliced, and a channel weight is generated through a fully connected layer and an activation function. At the same time, perform channel global average pooling and maximum pooling on , and generate a spatial weight after splicing the results through a convolution layer and an activation function. Multiply and channel by channel to generate an initial spatial importance map . Subsequently, perform channel shuffling and grouped convolution on and to generate the final spatial weight . Finally, modulate the feature according to the formula and perform weighted summation. After adding the modulated feature to ,

[0018] Step 4: Image reconstruction: fuse the shallow feature map and the deep feature map through convolution and sub-pixel convolution operations and perform upsampling to obtain the super-resolution reconstruction result. This process integrates the extracted features to generate the final high-resolution image. Advantages and positive effects of the present application

[0019] The present application realizes lightweight single-image super-resolution reconstruction by synergistically enhancing local and global features. The EDHFB module can effectively extract high-frequency detail features, the HAB module can capture local and global information, the GG-GAB and SG-GAB modules can explicitly model the similarity of image cross-regions with linear computational complexity, and the FAFM module can dynamically adjust the spatial weights of each channel, thereby strengthening high-frequency details while maintaining the consistency of low-frequency structures. The synergistic effect of these modules enables the present application to effectively improve the quality of image super-resolution reconstruction while maintaining a lightweight model, especially in the recovery of high-frequency details, and has high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The overall architecture diagram of the lightweight single-image super-resolution reconstruction method based on local-global feature synergistic enhancement perception provided by the present application is shown in the figure.

[0021] Figure 2A schematic diagram of the structure of the detailed high-frequency feature extraction module constructed by the present invention;

[0022] Figure 3 Schematic diagram of the hybrid attention module constructed for the present invention;

[0023] Figure 4 Schematic diagram of the global information guided grid attention structure constructed by the present invention;

[0024] Figure 5 Schematic diagram of the structure-guided global attention architecture constructed for the present invention;

[0025] Figure 6 Schematic diagram of the grid attention calculation guided by global information constructed by the present invention;

[0026] Figure 7 Schematic diagram of the structure-guided global attention calculation constructed for the present invention;

[0027] Figure 8 Schematic diagram of the frequency-driven channel-specific attention mechanism structure in the frequency-aware fusion module constructed for the present invention;

[0028] Figure 9 The high-resolution image after super-resolution reconstruction constructed by the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the present invention provides a single image super-resolution reconstruction method, comprising the following steps:

[0031] The main purpose of the present invention is to overcome the shortcomings of some of the existing technologies mentioned above, and to propose a lightweight single image super-resolution reconstruction method based on the CNN-Transformer collaborative mechanism. It aims to combine the advantages of CNN and Transformer, and at the same time achieve a balance between high performance and low computational cost through innovative network structure design and optimization strategies, so as to meet the image super-resolution reconstruction needs in different application scenarios. This method can effectively utilize the high and low frequency clues implicit in low-resolution images, achieve efficient feature fusion, and ensure the structural consistency and visual realism of the reconstruction results. This innovative method provides a new solution for the field of single image super-resolution reconstruction, effectively balancing detail recovery and global structural rationality, and has broad application prospects and important technical value.

[0032] To achieve the above objectives, the present invention adopts the following technical solutions:

[0033] A lightweight single-image super-resolution reconstruction method based on local-global feature collaborative enhancement perception, comprising the following steps:

[0034] Step 1: Obtain a low-resolution image

[0035] Step 2: Shallow feature extraction: use convolution operation to process the low-resolution image to extract the shallow features of the image. This process can capture the basic texture and structure information of the image, laying the foundation for subsequent deep feature extraction. The specific formula is:

[0036]

[0037] wherein, is the input low-resolution image, is a 3x3 convolution operation, is a shallow feature map.

[0038] Step 3: Deep feature extraction: use a cascaded residual local-global cross-integration module and a convolution operation to extract deep features from the shallow feature map. This process aims to further explore the detailed information of the image, especially the high-frequency details, to improve the quality of the reconstructed image. The specific extraction method includes the following sub-steps:

[0039] Step 3.1: The enhanced detail high-frequency feature extraction module of this embodiment is as shown in Figure 2 . First, the input feature map is split into a first sub-feature map and a second sub-feature map along the channel dimension. Then, 3x3 convolution and multi-difference convolution collaborative calculation are performed on at the same time, and the results are added and output after the activation function. Multi-difference convolution collaborative calculation includes parallel operations of center difference convolution, axial difference convolution, vertical difference convolution and horizontal difference convolution. In the inference stage, 3x3 convolution and multi-difference convolution are combined into a single convolution calculation to reduce computational overhead. Subsequently, maximum pooling, 1x1 convolution and activation function processing are performed on in turn. Finally, the outputs of steps (2) and (3) are spliced, and after 1x1 convolution, they are added to the original input feature map to enhance the expression ability of the features.

[0040] Step 3.2: The hybrid attention module in this embodiment is as shown in Figure 3As shown, to enhance the model's ability to interact across windows, the present application draws on the idea of Swin Transformer and introduces a hybrid attention module (HAB) in the local-global cross-integration module (LGCIM). HAB works in collaboration through parallel window multi-head self-attention (W-MSA) and shifted window multi-head self-attention (SW-MSA) branches, explicitly capturing the structural similarity of local regions and extracting global information of feature maps. Given the input feature, HAB first divides it into two parts along the channel dimension, and inputs them into the W-MSA branch and the SW-MSA branch, respectively, for processing. The output feature maps of the two branches are then concatenated along the channel dimension, normalized by layer normalization (LN), and added to the original input (i.e., adding a residual connection), to obtain the intermediate feature . Next, the output of the MLP is again normalized by layer normalization (LN) and added to the original input (i.e., adding a residual connection again), to finally obtain the output feature map of the HAB . Among them, and represent the parts of the input feature map after division and input into the W-MSA and SW-MSA branches, respectively. This design of the present application can significantly improve the model's perception of global information while maintaining computational efficiency, thereby achieving better performance in the image super-resolution reconstruction task.

[0041] Step 3.3: Global Information Guided Grid Attention Module (GG-GAB) or Structure Guided Global Attention Module (SG-GAB): These two modules aim to explicitly model the similarity across regions of an image with linear computational complexity, thereby improving the model's perception of global information while maintaining computational efficiency. The GG-GAB module focuses on global information through sparsely sampled grid points, while the SG-GAB module uses adaptive average pooling to map the structural information of an image block to a low-dimensional space, where self-attention is calculated. The following is a detailed description of these two modules:

[0042] The structure of the global information guided grid attention module of the present embodiment is as shown in Figure 4 , and the attention mechanism is as shown in Figure 6 . The GG-GAB module focuses on global information through sparsely sampled grid points, and its core lies in efficiently capturing the similarity between different regions in an image.

[0043] The specific steps are as follows: first, layer normalization is performed on the input: the input feature map is subjected to layer normalization processing to stabilize the training process and improve the generalization ability of the model. Then, grid division is performed: the feature map is divided into multiple small blocks according to the preset grid interval, and each small block contains information of a local region. Then, global interaction features are generated, that is, for each grid block, the maximum and average value features are extracted, and these features are spliced through a convolution layer to generate global interaction features. This process can capture global information between different grid blocks. Then, multi-head self-attention is calculated, based on the global interaction features, the attention weights of each grid block are calculated, and these weights are used to weight the value vectors to obtain the attention features of each grid block. This process allows the model to pass information between different grid blocks and enhances global perception ability. Finally, grid inverse operation and output are performed: the attention features of all grid blocks are restored to the original size and are connected with the input feature map in residual, and finally the output of the module is obtained through further processing by a multi-layer perceptron (MLP). The GG-GAB module focuses on global information through sparse sampling, avoiding the problem of increasing computational complexity with the square of the image size in the traditional global self-attention mechanism, thereby maintaining efficient computation while significantly improving the model's perception of the overall structure of the image.

[0044] The structure of the structure-guided global attention module of the embodiment is shown in Figure 5 The attention mechanism is shown in Figure 7 The SG-GAB module maps the structural information of the image block to a low-dimensional space through adaptive average pooling and calculates self-attention in the space, and the core is to efficiently capture global structural similarity.

[0045] The specific steps are as follows: First, layer normalization is performed on the input feature map to stabilize the training process and improve the model's generalization ability. Window partitioning is then performed to divide the feature map into multiple non-overlapping window blocks, each containing information about a local region. Multi-scale structural feature extraction is then performed. For each window block, adaptive average pooling is used to extract multi-scale structural features. These features reflect the structural information of the window block at different scales. These multi-scale features are then concatenated and compressed to obtain structural statistics for each window block. Global attention is then calculated based on the compressed structural statistics. Global attention weights are then weighted summed over the value matrix using these weights to obtain global attention features. This process allows the model to efficiently capture global structural similarity in a low-dimensional space. Finally, the output is obtained: the calculated global attention features are residually connected to the input feature map and further processed through a multi-layer perceptron (MLP) to obtain the module's output. The SG-GAB module efficiently calculates global attention in a low-dimensional space through adaptive average pooling and multi-scale feature extraction, thereby significantly improving the model's perception of the global structure of the image while maintaining computational efficiency.

[0046] Step 3.4: The frequency-driven channel-specific attention mechanism in the frequency-aware fusion module of this embodiment is as follows Figure 8 As shown, according to the high-frequency feature map and low-frequency feature map obtained in step 3.1 and step 3.2, first the high-frequency feature With low frequency characteristics Add to generate initial features Then Split along the channel into Sub-images are generated, and two-dimensional discrete cosine transform (2D-DCT) is performed on each sub-image to extract frequency components. The frequency components are concatenated and channel weights are generated through the fully connected layer and activation function. At the same time, Perform channel global average pooling and maximum pooling, and generate spatial weights after splicing the results through convolutional layers and activation functions .Will and Multiply channel by channel to generate the initial spatial importance map . Then and Perform channel shuffling and grouped convolution to generate the final spatial weights Finally, the modulation characteristic is calculated by the formula = Weighted summation. The modulation feature is combined with 、 After addition, the fused features are output through convolution.

[0047] Step 4: Image reconstruction: The shallow feature map is fused with the deep feature map through convolution and sub-pixel convolution operations and then upsampled to obtain a super-resolution reconstruction result. This process integrates the extracted features to produce the final high-resolution image.

[0048] This embodiment uses bicubic interpolation to obtain the corresponding low-resolution image from the original high-resolution image. The training set images are randomly rotated 、 、 Or horizontal flip, then randomly cropped into 64x64 pixel blocks. Each training batch contains 64 randomly selected image blocks, and the total number of training iterations is 500K. The optimizer uses Adam ( , ), the loss function uses loss. The initial learning rate is set to , and halved when the number of iterations reaches the milestone of [250K, 400K, 450K, 475K]. The number of RLGCIMs is set to 4, and each RLGCIM has two LGCIMs. The first LGCIM consists of an EDHFB and a module consisting of a HAB and GG-GAB cascade, while the second LGCIM consists of an EDHFB and a module consisting of a HAB and SG-GAB cascade. The number of filters is 48. The number of attention heads for HAB, GG-GAB, and SG-GAB are 6, 4, and 6 respectively, and the sampling interval in GG-GAB is set to , the image block size in SG-GAB is The method is implemented based on the PyTorch framework, and the model training is completed using the NVIDIA GeForce RTX 4090 GPU.

[0049] like Figure 9 As shown in FIG, the super-resolution reconstruction result of the present invention is given. The obtained super-resolution reconstruction result not only achieves performance improvement in indicators, but also has very good visual effects.

[0050] In the implementation of the present application, first, a low-resolution image is acquired as the input of super-resolution reconstruction; then, the shallow features of the image are extracted by convolution operation to capture basic texture and structure information. Subsequently, the deep features of the image are further mined through multiple cascaded residual local-global cross-integration modules, each module containing three major components of enhanced detail high-frequency feature extraction (EDHFB), global information guided grid attention (GG-GAB) or structure guided global attention (SG-GAB), and frequency-aware fusion (FAFM). EDHFB is used to capture detail information, GG-GAB and SG-GAB are used to explicitly model global information, and FAFM is used to fuse high-frequency and low-frequency features. Finally, the shallow and deep features are fused through convolution and sub-pixel convolution operations and up-sampling to obtain the super-resolution reconstruction result.

Claims

1. A lightweight single image super-resolution reconstruction method based on local-global feature collaborative enhancement perception, characterized by: include: Step 1: Get a low-resolution image; Step 2: Use convolution operation to extract shallow features of the low-resolution image to obtain a shallow feature map; Step 3: Use N cascaded residual local-global cross integration modules and a convolution operation to extract deep features from the shallow feature map to obtain a deep feature map; Step 4: The shallow feature map is fused with the deep feature map through convolution and sub-pixel convolution operations and upsampled to obtain the super-resolution reconstruction result.

2. The lightweight single image super-resolution reconstruction method based on local-global feature collaborative enhanced perception according to claim 1, characterized in that: The shallow feature extraction formula of S2 is: ,in, is the input low-resolution image, is a 3×3 convolution operation, It is a shallow feature map.

3. The lightweight single image super-resolution reconstruction method based on local-global feature collaborative enhanced perception according to claim 1, characterized in that: The residual local-global cross integration module in S3 includes an enhanced detail high-frequency feature extraction module (EDHFB) for extracting local high-frequency features of the input feature map; a hybrid attention module (HAB) cascaded with a global information guided grid attention module (GG-GAB) or a structure guided global attention module (SG-GAB) for extracting low-frequency global information of the input feature map; the results of the above two feature maps are fused in the frequency-aware fusion module (FAFM); The output features of the global cross-integration module Expressed as: ,in, Indicates the The frequency-aware fusion module in the global cross-integration module, Indicates the The enhanced detail high-frequency feature extraction module in the global cross-integration module, Indicates the A global information guided grid attention module or a structure guided global attention module in a global cross integration module, Indicates the A hybrid attention module in a global cross-integration module.

4. The detail-enhanced high-frequency feature extraction module according to claim 3, characterized in that: Execute the following process: Split the input feature map into the first sub-feature map along the channel dimension and the second sub-feature map ;right Execute 3×3 convolution and multi-difference convolution collaborative calculations at the same time, add the results and output them through the activation function; Perform maximum pooling, 1×1 convolution and activation function processing in sequence; concatenate the above outputs, add them to the original input feature map after 1×1 convolution; wherein, the multi-difference convolution collaborative calculation includes parallel operations of center difference convolution, axial difference convolution, vertical difference convolution and horizontal difference convolution; in the inference stage, the 3×3 convolution and multi-difference convolution are combined into a single convolution calculation.

5. The hybrid attention module according to claim 3, characterized in that Execute the following process: Split the input feature map into the first branch and the second branch ;right Perform windowed multi-head self-attention (W-MSA) on Perform shifted window multi-head self-attention (SW-MSA); concatenate the output features of step (2), normalize them layer by layer, and add them to the original input features to generate intermediate features; Multi-layer perceptron (MLP) calculation is performed on the intermediate features, and then the output is added to the intermediate features after layer normalization; among them, W-MSA uses fixed window division, and SW-MSA uses window cyclic shift to achieve cross-window interaction; the addition operations in steps 5.3 and 5.4 both constitute residual connections.

6. The global information guided grid attention module according to claim 3, characterized in that Execute the following process: Perform layer normalization on the input feature map; grid-spaced Divide the feature map into Grid blocks; perform channel maximum extraction and average extraction on each grid block, and then generate global interaction features through 1×1 convolution after splicing the results; For each grid block: Mapping generates a query vector , key vector , value vector ; Based on global interaction features and Calculate the first attention weight and combine it with Calculate the second attention weight; combine the second attention weight with Multiply to generate grid block attention results; restore all grid block results to their original size, normalize them, process them with MLP and connect them with the original input residual before output; the channel maximum value / average value extraction adopts global maximum pooling / average pooling respectively; the grid division adopts uniform sampling strategy, and the grid interval It is a configurable parameter.

7. The structure-guided global attention module according to claim 3, characterized in that Execute the following process: Perform layer normalization on the input feature tensor; divide the feature map into indivual Window blocks; generate a query matrix for each window block , key matrix , value matrix ;right and Perform multi-scale adaptive average pooling in parallel to extract structural features at 1×1, 2×2, and 3×3 resolution scales; Flatten and splice multi-scale structural features to generate compressed structural statistics; based on compressed structural statistics and Calculate the global attention weight, and Multiply to generate attention features; add the attention features to the original input, and then process the output through layer normalization and MLP processing; among them, the multi-scale pooling results are flattened and spliced ​​along the channel dimension; The fusion of attention features and original input adopts residual connection.

8. The frequency perception fusion module according to claim 3, characterized in that: Execute the following process: The high-frequency features With low frequency characteristics Add to generate initial features ;right Split along the channel into sub-images, and perform two-dimensional discrete cosine transform (2D-DCT) on each sub-image to extract the frequency component; Concatenate frequency components and generate channel weights through fully connected layers and activation functions ;right Perform channel global average pooling and maximum pooling, and generate spatial weights after splicing the results through convolutional layers and activation functions ;Will and Multiply channel by channel to generate the initial spatial importance map ;right and Perform channel shuffling and grouped convolution to generate the final spatial weights ; Modulation characteristics according to the formula = Weighted summation; the modulation characteristics and 、 After addition, the fused features are output through convolution.

Images