Lightweight multi-scale global attention enhancement network for image super-resolution

Abstract

The invention discloses a lightweight multi-scale global attention enhancement network for image super-resolution. The lightweight multi-scale global attention enhancement network comprises a shallow feature learning module, an LMGAE deep feature extraction module, a multi-layer feature fusion module and an image reconstruction module. Extracting local features through a given degraded LR graph by using a shallow feature extraction module formed by single 3 * 3 convolution; the local features are sent to N cascaded LMGAE deep feature extraction modules for feature extraction; the LMGAE deep feature extraction module is composed of a multi-scale global attention module and a local information fusion module, and the two modules both adopt a residual learning strategy; and then up-sampling is carried out through a 3 * 3 convolution layer and a pixel-shuffle layer, and an SR image is obtained through reconstruction. According to the image super-resolution network provided by the invention, the number of parameters of the network and the calculation cost are reduced, the problem of super-resolution of a single image is effectively solved, and better performance is achieved in the aspect of super-resolution performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and analysis, and specifically relates to a lightweight multi-scale global attention enhancement network for image super-resolution. Background Art

[0002] Single image super-resolution refers to the image reconstruction process of restoring a low-resolution image (LR) to a high-resolution image (HR). This image reconstruction technology has been widely used in medical imaging, remote sensing images, monitoring systems, etc. In the process of generating high-resolution images, traditional methods mainly rely on interpolation techniques, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, to restore pixel information around the image.

[0003] The subsequent rise of convolutional neural networks (CNNs) has surpassed traditional interpolation-based methods in terms of image quality. In order to further improve performance, efforts are being made to train models using larger and deeper networks to accurately restore more details. However, these large models often consume huge computing resources, rely on expensive hardware and require longer computing times, which may result in high costs or long waits for application scenarios that require high-resolution images. Therefore, it is particularly urgent and necessary to build a lightweight and high-performance model in order to effectively address such issues.

[0004] Although CNN-based methods have made significant progress, they can only process local information due to the size of the convolution kernel. The self-attention mechanism (SA) in Transformer is introduced to process the non-local area of ​​the feature. It uses a redundant attention mechanism, resulting in a quadratic relationship between the computational complexity and the image size, which requires higher computing resources. Nevertheless, this discontinuous window shifting method cannot integrate information outside the window, so it has certain limitations in capturing global information. Summary of the invention

[0005] In order to solve the above problems, this application proposes a lightweight multi-scale global attention enhancement network (LMGAE-Net) for image super-resolution, which alleviates the impact of window discontinuity through a multi-scale global attention module (MGAB). At the same time, in order to learn local fine features more deeply, a local information fusion module (LIFB) is designed, and multiple shift fusion modules (MSFB) are used to capture local information, which not only reduces the number of network parameters and computational cost, but also achieves excellent performance in super-resolution (SR) performance.

[0006] In order to solve the above technical problems, the present invention is implemented in the following ways:

[0007] A lightweight multi-scale global attention enhancement network for image super-resolution, including a shallow feature learning module, a LMGAE deep feature extraction module, a multi-layer feature fusion module (MFF) and an image reconstruction module;

[0008] Given a degraded LR graph I LR ∈R 3×H×W , where H and W represent the height and width of the LR image, respectively, and a shallow feature learning module consisting of a single 3×3 convolution is used. SF (·), extract local features F0∈R C×H×W , where C represents the number of channels of the intermediate features, and its specific expression is as follows:

[0009] F0=f SF (I LR )

[0010] F0 is sent to N cascaded LMGAE deep feature extraction modules for feature extraction. The output f of the nth LMGAE deep feature extraction module is n (1≤n≤N) The expression is as follows:

[0011]

[0012] in, Represents the nth LMGAE deep feature extraction module function;

[0013] The LMGAE deep feature extraction module is composed of a multi-scale global attention module (MGAB) and a local information fusion module (LIFB), and both modules adopt a residual learning strategy; the output feature map of the n-1th LMGAE module is directly input into the nth LMGAE module, and the output function expression of the nth LMGAE module is as follows:

[0014]

[0015] Where X = (f MGAB (F n-1 )+F n-1 );

[0016] All LMGAE deep feature extraction modules are input to the multi-layer feature fusion module (MFF module), which fuses the output features of all LMGAE deep feature extraction modules through a 1×1 convolution, and further extracts features with the help of the proposed MSFB module. In order to assist network learning, a global residual learning mechanism is also introduced; the output expression of the multi-layer feature fusion module is as follows:

[0017] F MFF =f MFF(Concat[F n , F n-1 , ..., F1])+F0

[0018] Among them, F n , F n-1 ,…,F1 represents the output of all previous LMGAE modules;

[0019] Then, upsampling is performed through a 3×3 convolutional layer and a pixel-shuffle layer in the image reconstruction module to reconstruct the SR image I SR , its specific expression is as follows:

[0020] I SR =f UP (W*F MFF )

[0021] Where W represents the weight of the 3×3 convolutional layer, f UP Indicates a pixel-shuffle operation.

[0022] Furthermore, the multi-scale global attention module (MGAB) takes a set of input features X∈R C×H×W Divided into G groups, expressed as Each set of features uses a different window size M g Calculate the self-attention of group G, and set the features of group G to be Channel separation, the computational complexity of the G group self-attention (SA) is expressed as

[0023] After completing self-attention, the scattered channels are re-merged, and a 1×1 convolutional layer is used to integrate feature information from different groups. The overall process of the multi-scale global attention module can be expressed as follows:

[0024] [X0, X1, ..., X g ]=Split(X)

[0025] X=Conv 1×1 (Concat[X0, X1, ..., X g ])

[0026] Among them, Split(·) represents the channel splitting operation, SA(·) represents the self-attention calculation process, Concat(·) represents the channel concatenation, Conv 1×1 (·) represents a 1×1 convolution operation, and X represents the input features.

[0027] Furthermore, the local information fusion module (LIFB) is composed of an enhanced spatial attention module (ESA) and a multi-group shift fusion module (MSFB) in sequence, and the expression of the local information fusion module processing flow is as follows:

[0028] F LIFB =F MSFB (F ESA (X)

[0029] The enhanced spatial attention module reduces the number of channels of the input features through a 1×1 convolution layer, reduces the spatial size of the features using strided convolution and maximum pooling layers, extracts features through a set of convolution layers, and restores the spatial size of the features through an interpolation-based upsampling operation, thereby enhancing the output F of the spatial attention module. ESA The expression is as follows:

[0030]

[0031] Among them, f sigmoid represents the sigmoid function, represents the weight of the 1×1 convolutional layer, represents the weight of a 3×3 convolution with a stride of 2, represents the weight of the 1×1 convolutional layer that restores the channel dimension, Represents the output of the first layer of ESA, Indicates the output of ESA layers 2 to 5, f pool represents the maximum pooling operation, f up represents the upsampling function implemented by bilinear interpolation, Conv g Represents the Group Conv layer;

[0032] The multiple groups of shift fusion modules will output the feature F ESA The features are evenly divided into nine groups and moved along nine different spatial directions (down, right, up, left, unchanged, down right, up right, down left, up left). The features are further extracted using 1×1 convolution, and the information of the surrounding nine pixels is integrated to effectively expand the receptive field without adding additional learnable parameters and large computational overhead, while maintaining the arithmetic complexity comparable to that of a single 1×1 convolution. Given the input feature Y, the process of the multi-group shift fusion module (MSFB) is expressed as follows:

[0033] F MSFB =Conv 1×1 (Concat(Y0, Y1, ..., Y9))

[0034] [Y0, Y1,...,Y9]=Split(Y)

[0035] Y i =fshift (Y i ), i∈{0, 1, ..., 9}

[0036] Among them, f shift (·) indicates a shift operation.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The proposed multi-scale global attention module (MGAB) divides features into different groups through a novel grouping strategy, and uses windows of different sizes to independently calculate self-attention, effectively expanding the receptive field and improving the ability to capture remote information; the proposed multi-group shift fusion module (MSFB) captures local information, and uses the computing resources of 1×1 convolution to achieve the receptive field range of 3×3 convolution, while maintaining a low computing cost, effectively learning and fusing local features, and improving the network's ability to restore image details. The proposed lightweight multi-scale global attention enhancement network (LMGAE-Net) for image super-resolution reduces the number of network parameters and computing costs, and effectively solves the single image super-resolution problem; rigorous qualitative and quantitative evaluations are carried out on commonly used benchmark datasets, and good performance is achieved in super-resolution performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the structure of a lightweight multi-scale global attention enhancement network (LMGAE-Net) for image super-resolution of the present invention;

[0040] Figure 2 Schematic diagram of the structure of the lightweight multi-scale global attention module (LMGAE) of the present invention;

[0041] Figure 3 Schematic diagram of the structure of the multi-scale global attention module (MGAB) of the present invention;

[0042] Figure 4 It is a structural schematic diagram of the local information fusion module (LIFB) of the present invention;

[0043] Figure 5 Schematic diagram of the structure of the enhanced spatial attention module (ESA) of the present invention;

[0044] Figure 6 It is a schematic diagram of the structure of a multi-group shift fusion module (MSFB) of the present invention;

[0045] Figure 7 A schematic diagram showing a comparison of the visual effects of ×4SR on the Urban100 and BSD100 datasets of the present invention;

[0046] Figure 8This is a schematic diagram of the effect of adjusting the number of groups on test performance according to the present invention. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0048] like Figure 1 As shown, a lightweight multi-scale global attention enhancement network for image super-resolution includes a shallow feature learning module, a LMGAE deep feature extraction module, a multi-layer feature fusion module (MFF) and an image reconstruction module;

[0049] Given a degenerate LR graph ILR∈R 3×H×W , where H and W represent the height and width of the LR image, respectively, and a shallow feature learning module consisting of a single 3×3 convolution is used. SF (·), extract local features F0∈R C×H×W , where C represents the number of channels of the intermediate features, and its specific expression is as follows:

[0050] F0=f SF (I LR )

[0051] F0 is sent to N cascaded LMGAE deep feature extraction modules for feature extraction. Assuming that the model has N LMGAE modules, the output f of the nth LMGAE deep feature extraction module n (1≤n≤N) The expression is as follows:

[0052]

[0053] in, represents the nth LMGAE deep feature extraction module function, F n Represents the output of this module.

[0054] like Figure 2 As shown in FIG. 1 , the LMGAE deep feature extraction module is composed of a multi-scale global attention module (MGAB) and a local information fusion module (LIFB), and both modules adopt a residual learning strategy; the output feature map F of the n-1th LMGAE module n-1 Directly input into the nth LMGAE module, the output function expression of the nth LMGAE module is as follows:

[0055]

[0056] Where X = (f MGAB (F n-1 )+F n-1 );

[0057] All LMGAE deep feature extraction modules are input to the multi-layer feature fusion module (MFF module), which fuses the output features of all LMGAE deep feature extraction modules through a 1×1 convolution, and further extracts features with the help of the proposed MSFB module. In order to assist network learning, a global residual learning mechanism is also introduced; the MFF module fuses the features of all previous LMGAE modules. The output expression of the multi-layer feature fusion module is as follows:

[0058] F MFF =f MFF (Concat[F n , F n-1 , ..., F1])+F0

[0059] Among them, F n , F n-1 ,…,F1 represents the output of all previous LMGAE modules;

[0060] Then, upsampling is performed through a 3×3 convolutional layer and a pixel-shuffle layer in the image reconstruction module to reconstruct the SR image I SR , its specific expression is as follows:

[0061] I SR =f UP (W*F MFF )

[0062] Where W represents the weight of the 3×3 convolutional layer, f UP Indicates a pixel-shuffle operation.

[0063] The window-based self-attention mechanism plays an important role in reducing the huge computational overhead of the Transformer model. Assuming that the size of the feature map is C×H×W, when a non-overlapping window of size M×M is used, its computational complexity is reduced to 2M. 2 HWC; the window size M plays a decisive role in the calculation range of self-attention. Choosing a larger M value helps the network capture global features more effectively and deeply mine self-similar information. Simply increasing M will lead to a quadratic increase in computational cost and resource requirements.

[0064] In order to learn long-range features more efficiently, a multi-scale global attention module (MGAB) is proposed, such as Figure 3 As shown, the multi-scale global attention module (MGAB) takes a set of input features X∈R C×H×W Divided into G groups, expressed as Each set of features uses a different window size M gCalculate the self-attention of the Gth group. This method not only balances the computational efficiency and feature representation ability, but also enables the network to capture rich contextual information at multiple scales, thereby improving the overall performance of the model. Set the Gth group of features to be Channel separation, the computational complexity of the G group self-attention (SA) is expressed as

[0065] After completing self-attention, the scattered channels are re-merged, and a 1×1 convolutional layer is used to integrate feature information from different groups. The overall process of the multi-scale global attention module can be expressed as follows:

[0066] [X0, X1, ..., X g ]=Split(X)

[0067] X i =SA(X i ), i∈{0, 1, ..., g}

[0068] X=Conv 1×1 (Concat[X o , X1, ..., X g ])

[0069] Among them, Split(·) represents the channel splitting operation, SA(·) represents the self-attention calculation process, Concat(·) represents the channel concatenation, Conv 1×1 (·) represents a 1×1 convolution operation, and X represents the input features.

[0070] like Figure 4 As shown, the local information fusion module (LIFB) is composed of a lightweight enhanced spatial attention module (ESA) and a multi-group shift fusion module (MSFB) in sequence, and the expression of the local information fusion module processing flow is as follows:

[0071] F LIFB =F MSFB (F ESA (X)

[0072] In order to ensure efficiency while improving the expressiveness of the model, we adopted a lightweight enhanced spatial attention module (ESA), which can effectively enhance the performance of the model in the spatial dimension. Figure 5 As shown in the figure, the enhanced spatial attention module reduces the number of channels of the input features through a 1×1 convolution layer, reduces the spatial size of the features by using strided convolution and maximum pooling layers, extracts features through a set of convolution layers, and restores the spatial size of the features through an interpolation-based upsampling operation. The output F of the lightweight enhanced spatial attention module ESA The expression is as follows:

[0073]

[0074] Among them, f sigmoid represents the sigmoid function, represents the weight of the 1×1 convolutional layer, represents the weight of a 3×3 convolution with a stride of 2, represents the weight of the 1×1 convolutional layer that restores the channel dimension, Represents the output of the first layer of ESA, Indicates the output of ESA layers 2 to 5, f pool represents the maximum pooling operation, f up represents the upsampling function implemented by bilinear interpolation, Conv g represents the Group Conv layer, and X represents the input features of LIFB;

[0075] In order to deeply capture local features, the past usually relies on 1×1 or 3×3 convolution layers. However, 1×1 convolution is not enough to capture enough contextual information due to its limited receptive field. Although 3×3 convolution can expand the receptive field, it consumes a lot of computing resources. Therefore, in order to expand the receptive field without sacrificing computing efficiency, this application proposes a multi-group shift fusion module (MSFB). Figure 6 As shown, the multiple groups of shift fusion modules will output feature F ESA The features are evenly divided into nine groups and moved along nine different spatial directions (down, right, up, left, unchanged, bottom right, top right, bottom left, top left). The features are further extracted using 1×1 convolution. This design enables the 1×1 convolution to integrate the information of the surrounding nine pixels, effectively expanding the receptive field without adding additional learnable parameters and large computational overhead, and maintaining the arithmetic complexity comparable to that of a single 1×1 convolution. Given the input feature Y, the process of the multi-group shift fusion module (MSFB) is expressed as follows:

[0076] F MSFB =Conv 1×1 (Concat(Y0, Y1, ..., Y9))

[0077] [Y0, Y1,...,Y9]=Split(Y)

[0078] Y i =f shift (Y i ), i∈{0, 1, ..., 9}

[0079] Among them, f shift (·) indicates a shift operation.

[0080] In the embodiments of this application, the effectiveness of the proposed LMGAE-Net on five commonly used SR benchmark datasets is verified through quantitative and qualitative evaluation methods.

[0081] Datasets and evaluation metrics: 800 images from the DIV2K dataset were selected as the training set, and the corresponding low-resolution (LR) images were created by applying the bicubic downsampling technique to the HR images. To verify the performance of the model, it was tested on five commonly used benchmark datasets, including Set5, Set14, BSD100, Urban100, and Manga109; to evaluate the quality of the reconstructed images, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) were used, where the PSNR and SSIM values ​​were calculated on the Y channel after converting the image from RGB format to YCbCr color space.

[0082] Experimental details: During the training phase, data augmentation methods such as random rotation of 90°, 180°, 270° and horizontal flipping were introduced. The LMGAE-Net architecture consists of 16 LMGAE modules with 60 channels. These modules use 4×4, 8×8, 16×16 and 32×32 window sizes to calculate MGAB on four equally divided channels. 16 64×64 image blocks were randomly selected from low-resolution (LR) images as the basic input units for training. The Adam optimizer was used for model training with parameters set to β1=0.9 and β1=0.99, and the initial learning rate was set to 1×10 -3 , the total number of iterations is set to 300,000. To guide model training, a composite indicator of mean absolute error (MAE) loss and frequency loss function based on fast Fourier transform (FFT) is used. The experimental operations are completed in an environment equipped with NVIDIA A100 Tensor Core GPU with the help of Pytorch framework.

[0083] To evaluate the performance of the LMGAE-Net model, it is compared with the state-of-the-art lightweight SR at different scaling factors and the quantitative results are analyzed.

[0084] Table 1 Performance comparison of different lightweight SR models on five benchmark models

[0085]

[0086]

[0087] Table 1 shows the quantitative comparison of various lightweight methods on five benchmark datasets. The results show that LMGAE-Net achieves the best performance on all datasets. Although LMGAE-Net has more parameters than RepRFN, its performance has been significantly improved. Taking the ×3 scale as an example, LMGAE-Net achieves a high PSNR score of 28.65 on the Urban100 dataset, which is better than RepRFN's 28.06.

[0088] Qualitative comparison: The visual quality of the SR results of LMGAE-Net and six representative models (including CARN, IMDN, RFDN, ShuffleMixer, SAFMN and RepRFN) on the ×4 magnification task is compared. Figure 7 As shown in the figure, it is observed that most of the comparison methods produce blurred or inaccurate edges and textures during image restoration, while the method of the present application can restore more accurate and clear edge details. This advantage of LMGAE-Net is particularly prominent in images containing repeated patterns and clear edges. In the reconstruction process, most methods will introduce artifacts and deformations, while in comparison, it can restore clearer patterns and edges.

[0089] In order to deeply understand the working principle of LMGAE-Net, a series of ablation experiments are conducted to analyze the contribution of each component in the method. In order to ensure fair comparison with the designed baseline model, all experiments are conducted at ×4 magnification scale and use consistent training configuration. The experiment starts with a streamlined baseline model, from which the MGAB module and LIFE module are removed (model ①). Then, the MGAB module (model ②) and MSFB module (model ③) are gradually introduced into the baseline model, and the effect after removing the ESA module is shown (model ④). Finally, all these modules together constitute the complete version of the method (model ⑤), and the specific results are shown in Table 2.

[0090] Table 2 PSNR / SSIM comparison of 4×4 models under different settings

[0091]

[0092] As the core component of the LMGAE-Net architecture, MGAB performs well in acquiring global information. By comparing model ① and model ②, we found that the introduction of MGAB has achieved stable improvements in PSNR and SSIM on all five data sets, with the maximum increase of PSNR being 0.51dB. In order to gain a deeper understanding of its mechanism, MGAB was further explored, and the effect of the grouping strategy was tested by adjusting the number of groups N. Figure 8As shown in the figure, the performance of the model has been steadily improved with the increase in the number of groups, which confirms the effectiveness of the grouping strategy.

[0093] Effectiveness of MSFB: The MSFB module can efficiently capture local features while reducing computational costs. Comparing model ① with model ③, it can be clearly observed that after the introduction of the MSFB module, the number of model parameters is reduced by about 38.8%, and the performance is also significantly improved. In order to further verify the effectiveness of the MSFB module, a series of ablation experiments are carried out. As shown in Table 3, under the premise of keeping the number of parameters constant, only learning the left, middle, right (①) or top, middle, and bottom (②) positions of pixels will cause the model performance to drop significantly. The module that only moves up, down, left, and right (③) can only capture the information of the surrounding 4 pixels and itself, and its performance is still not as good as the module that moves along nine different directions (④). In general, the MSFB module performs well in effectively encoding local information and reducing the number of parameters.

[0094] Table 3 The impact of different mobile groups on experimental results

[0095]

[0096]

[0097] In order to confirm the actual effect of this attention module, an ablation experiment was conducted. As shown in model ④ in Table 2, compared with the model without ESA, the complete LMGAE-Net achieved performance improvement on five commonly used datasets. This result shows that ESA can effectively enhance the overall expressiveness of the model.

[0098] The above description is only an implementation mode of the present invention. It is stated again that for ordinary technicians in this technical field, several improvements can be made to the present invention without departing from the principle of the present invention. These improvements are also included in the protection scope of the claims of the present invention.

Claims

1. A lightweight multi-scale global attention enhancement network for image super-resolution, characterized by: It includes shallow feature learning module, LMGAE deep feature extraction module, multi-layer feature fusion module and image reconstruction module; Given a degraded LR graph I LR ∈R 3×H×W , where H and W represent the height and width of the LR image, respectively, and a shallow feature learning module consisting of a single 3×3 convolution is used. SF (·), extract local features F0∈R C×H×W , where C represents the number of channels of the intermediate features, and its specific expression is as follows: F0=f SF (IN LR ) F0 is sent to N cascaded LMGAE deep feature extraction modules for feature extraction. The output f of the nth LMGAE deep feature extraction module is n (1≤n≤N) The expression is as follows: in, Represents the nth LMGAE deep feature extraction module function; The LMGAE deep feature extraction module is composed of a multi-scale global attention module and a local information fusion module, and both modules adopt a residual learning strategy; the output feature map of the n-1th LMGAE module is directly input into the nth LMGAE module, and the output function expression of the nth LMGAE module is as follows: Where X = (f MGAB (F n-1 )+F n-1 ); All LMGAE deep feature extraction modules are input to the multi-layer feature fusion module, which fuses the output features of all LMGAE deep feature extraction modules through a 1×1 convolution and further extracts features with the help of the proposed MSFB module. The output expression of the multi-layer feature fusion module is as follows: F MFF =f MFF (Concat[F n ,F n-1 ,…,F1])+F0 Among them, F n , F n-1 ,...,F1 represents the output of all previous LMGAE modules; Then, upsampling is performed through a 3×3 convolutional layer and a pixel-shuffle layer in the image reconstruction module to reconstruct the SR image I SR , its specific expression is as follows: I SR =f UP (W*F MFF ) Where W represents the weight of the 3×3 convolutional layer, f UP Indicates a pixel-shuffle operation.

2. A lightweight multi-scale global attention enhancement network for image super-resolution according to claim 1, characterized in that: The multi-scale global attention module takes a set of input features X∈R C×H×W Divided into G groups, expressed as Each group of features uses a different window size Mg to calculate the self-attention of the G group, setting the G group features to be Channel separation, the computational complexity of the G group self-attention is expressed as After completing self-attention, the scattered channels are re-merged, and a 1×1 convolutional layer is used to integrate feature information from different groups. The overall process of the multi-scale global attention module can be expressed as follows: [X0,X1,...,X g ]=Split(X) X i =SA(X i ),i∈{0,1,...,g} X=Conv 1×1 (Concat[X0,X1,…X g ]) Among them, Split(·) represents the channel splitting operation, SA(·) represents the self-attention calculation process, Concat(·) represents the channel concatenation, Conv 1×1 (·) represents a 1×1 convolution operation, and X represents the input features.

3. The lightweight multi-scale global attention enhancement network for image super-resolution according to claim 1, characterized in that: The local information fusion module is composed of an enhanced spatial attention module and a multi-group shift fusion module in sequence, and the expression of the local information fusion module processing flow is as follows: F LIFB =F MSFB (F ESA (X)) The enhanced spatial attention module reduces the number of channels of the input features through a 1×1 convolution layer, reduces the spatial size of the features using strided convolution and maximum pooling layers, extracts features through a set of convolution layers, and restores the spatial size of the features through an interpolation-based upsampling operation. The output F of the lightweight enhanced spatial attention module ESA The expression is as follows: Among them, f sigmoid represents the sigmoid function, represents the weight of the 1×1 convolutional layer, represents the weight of a 3×3 convolution with a stride of 2, represents the weight of the 1×1 convolutional layer that restores the channel dimension, Represents the output of the first layer of ESA, Indicates the output of ESA layers 2 to 5, f pool represents the maximum pooling operation, f up represents the upsampling function implemented by bilinear interpolation, Conv g Represents the Group Conv layer; The multiple groups of shift fusion modules will output the feature F ESA The features are evenly divided into nine groups and moved along nine different spatial directions. 1×1 convolution is used to further extract features while maintaining the arithmetic complexity equivalent to that of a single 1×1 convolution. Given the input feature Y, the process of multiple groups of shift fusion modules is expressed as follows: F MSFB =Conv 1×1 (Concat(Y0,Y1,...,Y9)) [Y0, Y1,...,Y9]=Split(Y) Y i =f shift (Y i ),i∈{0,1,...,9} Among them, f shift (·) indicates a shift operation.

Images