A lightweight method for small target detection in remote sensing images

Abstract

The present invention discloses a lightweight remote sensing image small target detection method; through a lightweight convolution module LFC, the amount of parameters and the amount of calculation required for the convolution operation are reduced to complete feature extraction and downsampling under the premise of limited computing resources and storage space; a receptive field expansion module RFEB is adopted to expand the receptive field of the backbone network through multi-branch dilated convolution, and make full use of local and global context information to solve the problem of insufficient feature representation of small targets in remote sensing images; a hierarchical fusion attention module HFAB is used to achieve feature fusion from coarse-grained to fine-grained by progressively stacking spatial attention and self-attention, so that the feature representation of small targets is richer; the method provided by the present invention reduces the amount of model parameters and calculation overhead while maintaining the target detection accuracy, effectively reduces the calculation complexity, and improves the feasibility of the target detection algorithm in edge computing and real-time monitoring scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a lightweight remote sensing image small target detection method. Background Art

[0002] Using images acquired by remote sensing satellites for target detection has become a key task in the industrial field. Remote sensing images can observe the earth over a large area and obtain valuable remote sensing data. They also obtain information quickly and in a short period of time. They are widely used in agriculture, forestry, climate management, environmental protection and other fields. Compared with traditional target detection tasks, remote sensing images have the characteristics of high resolution, random direction, multi-class variation, multi-scale scenes and dense small targets, which makes target detection in remote sensing images more complicated.

[0003] Despite the rapid development of optical remote sensing image technology, there are still many challenges in the detection of small targets in radar remote sensing images. Although radar images have high resolution, their overall image quality is poor. The features of the targets to be detected are usually subtle, with dim features, low contrast and incomplete information, which greatly increases the difficulty of detection. In addition, radar remote sensing images are inevitably subject to uncontrollable observation conditions and interference in the imaging chain during the acquisition process, such as atmospheric disturbances and platform motion. This leads to the aliasing of targets and backgrounds, further exacerbating the difficulty of small target detection. At the same time, large-scale remote sensing images usually contain rich contextual information. If this information is ignored during the actual detection process, it may lead to false detection. With the increase in the band of radar sensors and the improvement of image resolution, the amount of remote sensing image data generated is also growing sharply, bringing huge challenges to downstream task processing.

[0004] With the rapid growth of remote sensing image data, the mode of transmitting data to the ground for processing can hardly meet the performance requirements of time-sensitive applications (such as emergency rescue), because the huge data transmission process may cause serious time delays; therefore, deploying the detection algorithm directly on the edge terminal device with limited computing resources to achieve real-time processing can significantly reduce the pressure of data transmission and shorten the time delay from information acquisition to decision-making; however, the resource limitations of the terminal device (such as power consumption, weight and volume) put forward higher requirements on the reliability, speed and scale of the processing algorithm.

[0005] In recent studies, geometric and topological constraints are introduced in the convolution operation to flexibly cope with the shape changes of tubular structures, ensuring that the segmentation results can accurately reflect the geometric characteristics of tubular structures, thereby improving the accuracy and stability of segmentation. This method reduces the number of parameters and floating-point operations (FLOPs) by exploiting the redundancy in the filter; however, in the process of compensating for the loss of accuracy by increasing the network width, the memory access overhead of this method also increases;

[0006] The Involution operation breaks through the limitations of the fixed convolution kernel in the traditional convolution operation. Involution dynamically adjusts the shape of the convolution kernel at each position so that it can better adapt to local changes in the image, thereby significantly improving the feature expression ability of the convolution operation. However, the problem that comes with it is the increase in computational complexity and model complexity.

[0007] LSKNet has designed a special large convolution kernel decomposition mechanism, which reduces the number of parameters while improving performance and enhancing the receptive field in target detection. However, this method also brings a lot of edge noise while expanding the receptive field, which will affect the detection performance.

[0008] MAE (Masked Autoencoder) improves the training efficiency and feature representation ability of visual transformers through random masking strategies, lightweight decoder design, and encoder-decoder separation architecture. MAE has made great improvements in the number of parameters and computational complexity, but this method is still difficult to adapt to fields with limited computing resources. Summary of the invention

[0009] In view of this, the present invention provides a lightweight small target detection method for remote sensing images, which effectively improves the representation capability of target features, reduces noise interference, and ensures the efficiency and accuracy of the algorithm under limited computing resources and storage space, so as to efficiently cope with remote sensing image detection tasks.

[0010] To achieve the above object, the present invention provides a lightweight remote sensing image small target detection method, which adopts an improved CSPDarknet as a backbone network, including a CBS module, a CSP module, a lightweight feature convolution LFC module, an SPPF module and a receptive field expansion RFE module; adopts an improved PAN as a neck network, including a CBS module and a hierarchical fusion attention HFA module, and comprises the following steps:

[0011] S1. Preprocess the input image data and use the CBS module with a convolution kernel of 3 to expand the 3 channels of the original input image to 16 channels. The feature map size is reduced to 320×320 to obtain the feature map ;

[0012] S2, use the LFC module with a convolution kernel of 3 to transform the feature map The channel is expanded to 32, and the feature map size is reduced to 160×160, and the feature map is obtained. ;

[0013] S3, using CSP module to enhance feature map without changing the number of channels and feature map size Feature expression, get feature map ;

[0014] S4, repeat the operations of step S2 and step S3, expand the number of channels to 64, reduce the size of the feature map to 80×80, and obtain the feature map ;

[0015] S5. Repeat step S2 to expand the number of channels to 128 and reduce the feature map size to 40×40 to obtain the feature map. ;

[0016] S6. Use the RFE module to map the feature map Processing to obtain feature map ;

[0017] S7, repeat the operations of step S5 and step S6, expand the number of channels to 256, reduce the size of the feature map to 20×20, and obtain the feature map ;

[0018] S8. Use SPPF module to enhance feature map Feature expression, get feature map ;

[0019] S9. Feature map Perform upsampling to restore the feature map size to 40×40 and align it with the feature map along the channel direction. Splicing, feature map ;

[0020] S10, feature map Input to the HFA module to get the feature map ;

[0021] S11, repeat the operations of step S9 and step S10, expand the feature map size to 80×80, and Splice to get the feature map ;

[0022] S12, use the CBS module with a convolution kernel of 3 to transform the feature map The number of channels is changed to 64, and the feature map size is reduced to 40×40, which is the same as the feature map output in step S10. Splice and repeat step S10 to obtain the feature map ;

[0023] S13, use the CBS module with a convolution kernel of 3 to process the feature map output by step S12 , the number of channels is changed to 128, and the feature map size is reduced to 20×20, which is the same as the feature map output in step S8 Splice and repeat step S10 to obtain the feature map ;

[0024] S14, output the detection result through the YOLOv8 detection head;

[0025] The YOLOv8 detection head performs feature mapping on the , feature map and feature map The features are classified and located to obtain the feature map Detection results, feature maps Detection results and feature maps The detection results are collected and a detection result set is formed to obtain the target detection result.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The Lightweight Feature Convolution (LFC) proposed in this invention is an efficient lightweight convolution operation that significantly reduces the number of model parameters and computational complexity while maintaining high detection accuracy as much as possible. LFC is suitable for deployment in devices with limited computing resources and is used for real-time processing of remote sensing images, thereby reducing data transmission pressure and shortening the time delay from information acquisition to decision-making.

[0028] The present invention enhances the representation ability of the model by introducing a receptive field expansion block (RFEB); RFEB adopts multi-branch dilated convolution technology to expand the receptive field of the backbone network, so that the model can better capture small targets and their contextual information, thereby improving the detection performance;

[0029] The present invention proposes a hierarchical fusion attention block (HFAB), which realizes feature fusion from coarse-grained to fine-grained by progressively stacking spatial attention and self-attention mechanisms. This feature fusion strategy enables the model to more fully express the detailed information of small targets, effectively alleviating the shortcomings of traditional detection algorithms in background confusion and small target recognition.

[0030] Under the synergistic effect of LFC, RFEB and HFAB, the present invention significantly reduces the computational complexity while maintaining accuracy, and can run under limited hardware conditions, so it has great application potential in real-time processing tasks in the future; this lightweight design also enables the present invention to adapt to more application scenarios, especially in situations that require rapid response and high timeliness, such as emergency rescue and other scenarios, which has great potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a flow chart of a lightweight remote sensing image small target detection method of the present invention;

[0032] Figure 2 Schematic diagram of the lightweight feature convolution LFC module of the present invention;

[0033] Figure 3 Schematic diagram of the receptive field expansion REF module of the present invention;

[0034] Figure 4 Schematic diagram of the receptive field expansion REF module of the present invention;

[0035] Figure 5 This is a schematic diagram of the hierarchical fusion attention HFA module of the present invention;

[0036] Figure 6 Schematic diagram of the hierarchical fusion attention HFA module of the present invention. DETAILED DESCRIPTION

[0037] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0038] In order to effectively improve the representation ability of target features, reduce noise interference, and ensure the efficiency and accuracy of the algorithm under limited computing resources and storage space, and efficiently cope with remote sensing image detection tasks, the present invention provides a lightweight remote sensing image small target detection method, which adopts an improved CSPDarknet as a backbone network, including a CBS module, a CSP module, a lightweight feature convolution LFC module, an SPPF module and a receptive field expansion RFE module; an improved PAN is used as a neck network, including a CBS module and a hierarchical fusion attention HFA module, and includes the following steps:

[0039] S1. Re-segment the input JPG format image into a size of 640×640. Use the CBS module with a convolution kernel of 3 to expand the 3 channels of the original input image to 16 channels. The feature map size is reduced to 320×320 to obtain the feature map. ;

[0040] S2, use the LFC module with a convolution kernel of 3 to transform the feature map The channel is expanded to 32, and the feature map size is reduced to 160×160, and the feature map is obtained. , the LFC module operates on the feature map including the following steps:

[0041] S21, the feature map with the number of channels C Split along the channel direction to obtain and :

[0042] ;

[0043] in, and The number of channels is , Indicates segmentation along the channel dimension, the feature map after segmentation Use deep convolution to get feature maps :

[0044] ;

[0045] in, Denotes deep convolution. Deep convolution carefully extracts local features of each channel while maintaining low parameter and computational complexity, thereby improving the network's ability to capture local details and enhancing the accuracy of feature expression.

[0046] S22, feature map after segmentation Use deformable convolution to get feature map :

[0047] ;

[0048] in, Deformable convolution represents deformable convolution, which can adapt to more complex shapes and structures, while enhancing the ability to express the details of the target object in the feature map, and more effectively deal with the background occlusion problem of small targets in remote sensing images, thereby improving the accuracy of detection;

[0049] S23, the feature map and feature map Splice along the channel direction and use batch normalization function And the activation function Processing to obtain feature map :

[0050] ;

[0051] in, represents splicing along the channel dimension, represents the batch normalization operation, represents the SiLU activation function;

[0052] S3, using CSP module to enhance feature map without changing the number of channels and feature map size Feature expression, get feature map ;

[0053] S4, repeat the operations of step S2 and step S3, expand the number of channels to 64, reduce the size of the feature map to 80×80, and obtain the feature map ;

[0054] S5. Repeat step S2 to expand the number of channels to 128 and reduce the feature map size to 40×40 to obtain the feature map. ;

[0055] S6. Use the RFE module to map the feature map Processing to obtain feature map , the RFE module operates on the feature map including the following steps:

[0056] S61, use convolution with a convolution kernel of 1×1 Processing feature maps , and then use layer normalization to increase Get feature map , use the RFE module to split the features along the channel dimension to obtain , the formula is as follows:

[0057] ;

[0058] ;

[0059] ;

[0060] in, Representation layer normalization, Represents the convolution kernel size is the convolution operation, Indicates segmentation along the channel dimension;

[0061] S62, use 7-9(4)-17-9(4)-17-9(4)-1 (indicating depthwise convolution with kernel 7, dilated convolution with kernel 9 and dilation rate 4, and 1×1 point convolution), 5-7(3)-15-7(3)-15-7(3)-1 (indicating depthwise convolution with kernel 5, dilated convolution with kernel 7 and dilation rate 3, and 1×1 point convolution), 3-5(2)-13-5(2)-13-5(2)-1 (indicating depthwise convolution with kernel 3, dilated convolution with kernel 5 and dilation rate 2, and 1×1 point convolution) multi-scale dilated convolution Expanding the receptive field through deep convolution Extract the feature map and multiply it to get the feature map , the formula is as follows:

[0062] ;

[0063] Residual Block Chaining After deep convolution and Add to get the feature map :

[0064] ;

[0065] in, represents the feature map dot product, represents splicing along the channel dimension, represents multi-scale dilated convolution, represents depthwise convolution;

[0066] S63, use convolution with a convolution kernel of 1×1 The feature map The number of channels is doubled and a 7×7 depth convolution is used Processing the expanded feature map :

[0067] ;

[0068] After processing, the feature map Perform convolution and connect with feature maps through residual connection Add together to get the feature map , the formula is as follows:

[0069] ;

[0070] in, represents the multiplication of feature maps, Representation layer normalization, represents a convolution with a convolution kernel of 1, represents the depthwise convolution, represents convolution;

[0071] S7, repeat the operations of step S5 and step S6, expand the number of channels to 256, reduce the size of the feature map to 20×20, and obtain the feature map ;

[0072] S8. Use SPPF module to enhance feature map Feature expression, get feature map ;

[0073] S9. Feature map Perform upsampling to restore the feature map size to 40×40 and align it with the feature map along the channel direction. Splicing, feature map ;

[0074] S10, feature map Input to the HFA module to get the feature map , the HFA module operates on the feature graph including the following steps:

[0075] S101, feature map along the height and width directions Perform global pooling operations separately , , creating a one-dimensional sequence structure with two items , And decomposed into four different feature maps along the channel dimension , , , ,right , The two feature maps use multi-scale convolution operations with kernel sizes of 3, 5, 7, and 9 respectively. , the expression is as follows:

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] in, represents the average pooling along the height direction of the feature map, represents the average pooling along the width of the feature map, represents segmentation along the channel dimension, represents depthwise separable convolution, Represents multi-scale convolution;

[0085] S102, four feature maps , , , Splicing along the channel dimension, aggregating different semantic sub-features, and , Perform a deep convolution with a convolution kernel of 5 to extract the original features, and perform point-by-point multiplication with the concatenated feature map to eliminate redundant spatial information and avoid distraction, and obtain the feature map , , the formula is as follows:

[0086] ;

[0087] ;

[0088] in, represents splicing along the channel dimension, Represents feature map multiplication;

[0089] S103, feature map , Using group normalization ,Compared with traditional batch normalization, group normalization performs better in distinguishing the semantic differences between sub-features, and can effectively reduce the noise introduced by batch statistics and alleviate the semantic interference between sub-features;

[0090] Using Sigmoid activation function Get the spatial attention map , , and then with the output feature map Multiply to get the feature map , the expression is as follows:

[0091] ;

[0092] ;

[0093] ;

[0094] in, represents the group normalization operation, represents the Sigmoid activation function, Represents feature map multiplication;

[0095] S104, the feature map is processed along the height and width directions Perform global pooling operations at the same time Get feature map :

[0096] ;

[0097] in, Indicates global pooling in both height and width directions;

[0098] Using depthwise convolution The replacement fully connected layer generates learnable query tokens Q, key tokens K, and value tokens V:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] in, represents a depth-wise separable convolution with a kernel of 1 that does not change the channel dimension. represents the generation of query matrix, represents the generated key matrix, represents the generated value matrix;

[0104] Use a depthwise convolution with kernel 3 for the query token Q Generate an intermediate proxy token A:

[0105] ;

[0106] ;

[0107] in, represents a depth-wise separable convolution with a kernel of 3 that does not change the channel dimension. Indicates the generation of an intermediate matrix;

[0108] Use the proxy token A as the query to compute the first attention between the key token K and the value token V, and aggregate the proxy features from all values :

[0109] ;

[0110] in, represents the softmax activation function, represents the scaling factor;

[0111] A as a key, Used as a value in conjunction with the query matrix Q, the global information from the proxy features is broadcast to each query token, generating the output O:

[0112] ;

[0113] The feature map of output O is average pooled in height and width directions , get a feature map of size 1×1, and use the Sigmoid activation function Generate the final attention map, and multiply it element by element with the spatial attention result to get the feature map :

[0114] ;

[0115] in, Represents matrix multiplication;

[0116] S11, repeat the operations of step S9 and step S10, expand the feature map size to 80×80, and Splice to get the feature map ;

[0117] S12, use the CBS module with a convolution kernel of 3 to transform the feature map The number of channels is changed to 64, and the feature map size is reduced to 40×40, which is the same as the feature map output in step S10. Splice and repeat step S10 to obtain the feature map ;

[0118] S13, use the CBS module with a convolution kernel of 3 to process the feature map output by step S12 , the number of channels is changed to 128, and the feature map size is reduced to 20×20, which is the same as the feature map output in step S8 Splice and repeat step S10 to obtain the feature map ;

[0119] S14, output the detection result through the YOLOv8 detection head;

[0120] The YOLOv8 detection head performs feature mapping on the , feature map and feature map The features are classified and located to obtain the feature map Detection results, feature maps Detection results and feature maps The detection results are collected and a detection result set is formed to obtain the target detection result.

[0121] The performance of the experimental platform used in the embodiment is Intel(R) Xeon(R) Gold 6330 28-core 2.0GHz 205W CPU, NVIDIA GeForce RTX 4090 graphics card, programming language is Python 3.10, and PyTorch version is 2.2.2;

[0122] During the training process, the batch size was 4; in addition, the entire training process included 300 training cycles, using the machine gradient descent (SGD) optimizer, with an initial learning rate of 0.01, a momentum of 0.937, and a weight decay coefficient of 0.0005; in all data sets, the training set and the validation set were uniformly divided into 8:2.

[0123] Using the existing public dataset USOD, the server device is selected to input the images of the USOD remote sensing image target detection dataset into the trained network model designed by the present invention to obtain the final remote sensing image target detection results. The experimental results compared with the YOLO series are shown in Table 1.

[0124] ;

[0125] Among them, mAP@0.5 / % represents the average accuracy of target detection when the intersection-over-union ratio is 0.5, mAP@95 / % represents the average accuracy of target detection when the intersection-over-union ratio is 0.95, Recall represents the average recall rate, Parameters / M represents the number of parameters of the network model, and FLOPs / G represents the computational complexity of the network model.

[0126] The method provided by the present invention and the mainstream YOLO series algorithms were trained and verified under the same data set, and comparative experimental results of model detection accuracy and model scale were obtained; according to the experimental data in Table 1, it can be seen that the remote sensing image small target detection method proposed in the present invention not only realizes the lightweight network model, but also shows a significant accuracy advantage.

[0127] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A lightweight remote sensing image small target detection method, using the improved CSPDarknet as the backbone network, including CBS module, CSP module, lightweight feature convolution LFC module, SPPF module and receptive field expansion RFE module; using the improved PAN as the neck network, including CBS module, hierarchical fusion attention HFA module, characterized in that, The following steps are involved: S1. Preprocess the input image data and use the CBS module with a convolution kernel of 3 to expand the 3 channels of the original input image to 16 channels. The feature map size is reduced to 320×320 to obtain the feature map ; S2, use the LFC module with a convolution kernel of 3 to transform the feature map The channel is expanded to 32, and the feature map size is reduced to 160×160, and the feature map is obtained. ; S3, using CSP module to enhance feature map without changing the number of channels and feature map size Feature expression, get feature map ; S4, repeat the operations of step S2 and step S3, expand the number of channels to 64, reduce the size of the feature map to 80×80, and obtain the feature map ; S5. Repeat step S2 to expand the number of channels to 128 and reduce the feature map size to 40×40 to obtain the feature map. ; S6. Use the RFE module to map the feature map Processing to obtain feature map ; S7, repeat the operations of step S5 and step S6, expand the number of channels to 256, reduce the size of the feature map to 20×20, and obtain the feature map ; S8. Use SPPF module to enhance feature map Feature expression, get feature map ; S9. Feature map Perform upsampling to restore the feature map size to 40×40 and align it with the feature map along the channel direction. Splicing, feature map ; S10, feature map Input to the HFA module to get the feature map ; S11, repeat the operations of step S9 and step S10, expand the feature map size to 80×80, and Splice to get the feature map ; S12, use the CBS module with a convolution kernel of 3 to transform the feature map The number of channels is changed to 64, and the feature map size is reduced to 40×40, which is the same as the feature map output in step S10. Splice and repeat step S10 to obtain the feature map ; S13, use the CBS module with a convolution kernel of 3 to process the feature map output by step S12 , the number of channels is changed to 128, and the feature map size is reduced to 20×20, which is the same as the feature map output in step S8 Splice and repeat step S10 to obtain the feature map ; S14, output the detection result through the YOLOv8 detection head; The YOLOv8 detection head performs feature mapping on the , feature map and feature map The features are classified and located to obtain the feature map Detection results, feature maps Detection results and feature maps The detection results are collected and a detection result set is formed to obtain the target detection result.

2. A lightweight remote sensing image small target detection method according to claim 1, characterized in that: The input image is in JPG format, and the preprocessing includes re-segmenting the image into a size of 640×640.

3. The lightweight remote sensing image small target detection method according to claim 1, characterized in that: In step S2, using the LFC module to operate the feature map includes the following steps: S21, the feature map with the number of channels C Split along the channel direction to obtain and : in, and The number of channels is , Indicates segmentation along the channel dimension, the feature map after segmentation Use deep convolution to get feature maps : in, represents depthwise convolution; S22, feature map after segmentation Use deformable convolution to get feature map : in, represents deformable convolution; S23, the feature map and feature map Splice along the channel direction and use batch normalization function And the activation function Processing to obtain feature map : in, represents splicing along the channel dimension, represents the batch normalization operation, Represents the SiLU activation function.

4. The lightweight remote sensing image small target detection method according to claim 1, characterized in that: In step S6, using the RFE module to operate the feature map includes the following steps: S61, use convolution with a convolution kernel of 1×1 Processing feature maps , and then use layer normalization Get feature map , use the RFE module to split the features along the channel dimension to obtain , the formula is as follows: in, Representation layer normalization, Indicates that the convolution kernel size is The convolution operation, Indicates segmentation along the channel dimension; S62. Use multi-scale dilated convolution Expanding the receptive field through deep convolution Extract the feature map and multiply it to get the feature map , the formula is as follows: Feature Map After deep convolution and Add to get the feature map : in, represents the feature map dot product, represents splicing along the channel dimension, represents multi-scale dilated convolution, represents depthwise convolution; S63, use convolution with a convolution kernel of 1×1 The feature map The number of channels is doubled and a 7×7 depth convolution is used Processing the expanded feature map : After processing, the feature map Perform convolution and connect with feature maps through residual connection Add together to get the feature map , the formula is as follows: in, represents the multiplication of feature maps, Representation layer normalization, represents a convolution with a convolution kernel of 1, represents the depthwise convolution, Represents convolution.

5. The lightweight remote sensing image small target detection method according to claim 1, characterized in that: Using the HFA module to operate the feature graph in step S10 includes the following steps: S101, feature map along the height and width directions Perform global pooling operations separately , , creating a one-dimensional sequence structure with two items , And decomposed into four different feature maps along the channel dimension , , , ,right , The two feature maps use multi-scale convolution operations with kernel sizes of 3, 5, 7, and 9 respectively. , the expression is as follows: in, represents the average pooling along the height direction of the feature map, represents the average pooling along the width of the feature map, represents segmentation along the channel dimension, represents depthwise separable convolution, Represents multi-scale convolution; S102, four feature maps , , , Splice along the channel dimension and aggregate different semantic sub-features to obtain the feature map , , the expression is as follows: in, represents splicing along the channel dimension, Represents feature map multiplication; S103, feature map , Using group normalization , using the Sigmoid activation function Get the spatial attention map , , and then with the output feature map Multiply to get the feature map , the expression is as follows: in, represents the group normalization operation, represents the Sigmoid activation function, Represents feature map multiplication; S104, the feature map is processed along the height and width directions Perform global pooling operations at the same time Get feature map : in, Indicates global pooling in both height and width directions; Using depthwise convolution The replacement fully connected layer generates learnable query tokens Q, key tokens K, and value tokens V: in, represents a depth-wise separable convolution with a kernel of 1 that does not change the channel dimension. represents the generation of query matrix, represents the generated key matrix, represents the generated value matrix; Use a depthwise convolution with kernel 3 for the query token Q Generate an intermediate proxy token A: in, represents a depth-wise separable convolution with a kernel of 3 that does not change the channel dimension. Indicates the generation of an intermediate matrix; Using the proxy token A as the query, compute the first attention between the key token K and the value token V, aggregate the proxy features from all values : in, represents the softmax activation function, represents the scaling factor; A as a key, Used as a value in conjunction with the query matrix Q, the global information from the proxy features is broadcast to each query token, generating the output O: The feature map of output O is average pooled in height and width directions , get a feature map of size 1×1, using the Sigmoid activation function Generate the final attention map, and multiply it element by element with the spatial attention result to get the feature map : in, Represents matrix multiplication.

Images