A channel-wise spatial attention method

Abstract

The application discloses a channel-by-channel spatial attention method, which comprises the following four steps: S1, dividing the output feature map of an arbitrary layer of a convolutional neural network into a plurality of sub-tensors in the spatial dimension according to a step size; S2, calculating the spatial attention weight of each channel by average pooling, 1D convolution and a Sigmoid activation function; S3, multiplying the spatial attention weight obtained in step S2 with each sub-tensor in step S1 element by element to obtain an adjusted sub-tensor; and S4, performing an inverse operation on the segmentation process in step S1 to splice the adjusted sub-tensor into the same shape as the input to obtain a channel-by-channel spatial attention (CbCSA) output. The channel-by-channel spatial attention method provided by the application can accurately obtain the spatial attention weight of the feature map of any layer of the convolutional neural network in units of channels, and makes up for the defects of the existing spatial attention method in the global spatial attention modeling capability at the single channel level.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision technology, and specifically relates to a channel-by-channel spatial attention method. Background Technology

[0002] Attention mechanisms are a key method in computer vision for enhancing the feature representation capabilities of neural networks. They are widely used in various fields such as image classification, object detection, instance segmentation, face recognition, and image generation, demonstrating significant effectiveness. In recent years, researchers both domestically and internationally have continuously delved into this field, driving rapid technological advancements.

[0003] Attention mechanisms can be broadly categorized into three types: channel attention, spatial attention, and hybrid attention. Channel attention enhances the feature representation of a neural network by weighting different channels, while spatial attention focuses on adaptively highlighting specific regions in the spatial dimension of the feature map to strengthen the feature representation of key areas. Hybrid attention combines channel attention and spatial attention. To date, no patents specifically implementing spatial attention have been found.

[0004] To enhance the performance of convolutional neural networks, many scholars have conducted extensive research on spatial attention and published related papers. Sanghyun Woo et al. proposed CBAM by combining average pooling, max pooling, and 7×7 kernel size convolution. CBAM can replace conventional convolution to improve the performance of convolutional neural networks. Yuxuan Li et al. proposed LSKNet by combining depthwise separable convolution. LSKNet further enhances the receptive field through large selectable kernels, enabling the obtained spatial attention to better strengthen the feature representation of key regions.

[0005] Existing spatial attention methods still fall short in modeling global spatial attention for each channel, failing to finely adjust feature maps on a channel-by-channel basis, resulting in poor feature extraction capabilities and consequently affecting the performance of downstream tasks such as image classification, object detection, and semantic segmentation. Summary of the Invention

[0006] To address the limitation of existing spatial attention methods in finely adjusting feature maps on a channel-by-channel basis, this invention provides a channel-by-channel spatial attention method. This method employs a lightweight spatial attention approach comprising the following steps: For a convolutional neural network model, let the output feature map of each layer be X,

[0007] S1: Based on step size feature map It is divided into multiple subtensors in the spatial dimension. .

[0008] S2: Use average pooling, 1D convolution, and sigmoid activation function to obtain the spatial attention weights for each channel in the feature map individually.

[0009] S3: Spatial attention weights obtained from S2 and each subtensor in S1 Performing element-wise product operation yields the adjusted subtensor .

[0010] S4: Execute the reverse operation of S1. splicing into input The same shape yields the output of channel-by-channel spatial attention (CbCSA). .

[0011] S5: This method is a plug-and-play approach that can be placed at the end of each module of the backbone model; it can also be placed at the end of each module of the backbone model for pre-training to further improve the model's performance.

[0012] The specific steps of step S1 include:

[0013] Step S11: For any given feature map at height and in the width W direction with step size Segment it to obtain multiple local feature block tensors. ,set up and Let represent the segmentation indices in the height and width directions, respectively. Then the ... The formula for a tensor is:

[0014]

[0015] in , .for The shape of the window, if the window is completely within the feature map area, then If the window extends beyond the edge, the tensor size is... ,in , .

[0016] The specific steps of step S2 include:

[0017] Step S21: Simultaneously process the feature map while performing S1. Feature descriptors are obtained using average pooling with a window size of S and a stride of S. ,in

[0018] ,

[0019] Here and Used to compensate for height or width Cannot be Size deviation during division. Each pooling result element. It can be represented as:

[0020]

[0021] in , These represent the actual sizes of the current pooling window in the H and W directions, respectively. This indicates the total number of elements within the window. This operation extracts local feature information from each channel and adapts to possible size differences at the boundaries.

[0022] Step S22: In the channel direction Perform segmentation to generate feature tensors for each channel. This can be expressed as a formula:

[0023]

[0024] Step S23: Use the reshape function to... Remodeling :

[0025]

[0026] Step S24: Apply a 1D convolution with kernel size f and a sigmoid activation function to... Perform convolution and activation to obtain :

[0027]

[0028] 1D Convolution Enables Automatic Learning The sigmoid activation function maps the features learned by 1D convolution to a range between 0 and 1, which reflects the weight relationship between local feature blocks S×S in a single channel of the original feature map X.

[0029] Step S25: Use the reshape function to... Remodeling :

[0030]

[0031] Step S26: Along the channel dimension spliced ​​together :

[0032]

[0033] Step S27: Pair in the spatial dimension with a step size of 1 Segmentation to obtain each size is tensor :

[0034] ,

[0035] Obtained The reaction in step S11 The weighting relationships between the channels will be used in the following steps. right Adjustments will be made.

[0036] The specific steps of step S3 include:

[0037] Step S31: ... After broadcasting in the spatial dimension and Element-wise multiplication yields

[0038]

[0039] This indicates element-wise multiplication.

[0040] The specific steps of step S4 include:

[0041] Step S41: Combine all calibrated block features Concatenated with the input feature map The same dimensions yield the final output of the channel-by-channel spatial attention module. The formula is as follows:

[0042]

[0043] in Indicates pixel height The corresponding window index; Indicates pixel height The corresponding window index, simultaneously satisfying , , and The window size in the boundary region can be represented by the following formula. , Obtained through calculation.

[0044] The present invention has the following beneficial effects and advantages:

[0045] This invention addresses the shortcomings of existing spatial attention methods in modeling global spatial attention for each channel, which fails to finely adjust feature maps on a channel-by-channel basis. It proposes a channel-by-channel spatial attention method that separates channels and employs average pooling, according to... The pooling window size is used to obtain local feature representations of the feature map. Then, by using 1D convolution and the sigmoid function, the relationship between local feature representations is efficiently modeled with a small number of parameters, highlighting the key regions of the feature map and thus improving the performance of the convolutional neural network. Attached Figure Description

[0046] Figure 1 Flowchart of the channel-by-channel spatial attention method.

[0047] Figure 2 A schematic diagram of the Spatial Attention Extraction Module (SAEB) in the block diagram of the channel-by-channel spatial attention method.

[0048] Figure 3 A schematic diagram of the backbone network and the backbone network after embedding the channel-by-channel spatial attention method.

[0049] Figure 4 A schematic diagram of an image classification network (CbCSANet) based on the channel-wise spatial attention method.

[0050] Figure 5 Overall accuracy comparison chart.

[0051] Figure 6 Confusion matrix of ResNet-50 classification results on the NWPU-RESISC45 dataset.

[0052] Figure 7 Confusion matrix of classification results of CBAMNet-50 on the NWPU-RESISC45 dataset.

[0053] Figure 8 Confusion matrix of CbCSANet-50 classification results on the NWPU-RESISC45 dataset.

[0054] Figure 9 Visualization results based on GradCAM++. Detailed Implementation

[0055] To describe the present invention more specifically, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. (Appendix) Figure 4 The described embodiments are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0056] Example 1

[0057] This embodiment selects remote sensing image scene classification as the application scenario. Remote sensing image scene classification (RSSC) has become a research hotspot in the field of remote sensing in recent years. It mainly focuses on the high semantic information of scene images and classifies them into various categories such as forests, airports, and rivers based on their content. This technology is widely used in tasks such as land use and land cover, urban and rural planning, natural disaster detection, and geospatial object classification.

[0058] This embodiment uses the NWPU-RESISC45 remote sensing image dataset as the training dataset. NWPU-RESISC45 is a high-quality, publicly available dataset widely used for scene classification tasks in remote sensing images. Created by Northwestern Polytechnical University, this dataset contains 45 classes of remote sensing images with different scene characteristics, with 700 images per class, totaling 31,500 images. The spatial resolution of the images ranges from 0.2m to 30m, covering various typical scenes such as airports, bridges, forests, rivers, and cities, exhibiting rich category diversity and scene complexity. Each image in the dataset has a fixed size of 256×256 pixels and covers different shooting angles and lighting conditions, realistically reflecting the diversity and complexity of remote sensing scenes. Due to its abundant sample size, broad scene coverage, and high-quality annotations, NWPU-RESISC45 is widely used to evaluate and validate the performance of remote sensing image classification algorithms and is an important benchmark dataset in the field of remote sensing.

[0059] The neural network structure for classification constructed in this embodiment is shown in the attached figure. Figure 4 As shown, the classification network uses the ResNet-50 model as the backbone network and embeds the proposed channel-wise spatial attention method into the first, second, third and fourth stages of the ResNet-50 model to construct a ResNet-50 (CbCSANet-50) with embedded channel-wise spatial attention. The pooling window size of the channel-wise spatial attention method is S=4.

[0060] The data preprocessing strategy used in this embodiment is as follows: First, the image is randomly cropped to a size of 224×224. Then, the pixel values ​​are scaled from [0,225] to [0,1]. Afterward, the cropped image is normalized using the mean = [0.485, 0.456, 0.406] and the standard deviation = [0.229, 0.224, 0.225]. The formula is as follows:

[0061]

[0062]

[0063] in This represents the c-th channel. This represents the pixel value in the c-th channel. This represents the normalized pixel value on the c-th channel.

[0064] This embodiment uses the following model training strategy: The cross-entropy loss function is chosen as the loss function for model training. Cross-entropy loss is suitable for multi-class classification tasks. It effectively measures the difference between the model's output class probability distribution and the true class. The loss function formula is as follows:

[0065]

[0066] in Indicates the true label, This represents the predicted class probability. The optimizer chosen is the Adam optimizer, with an initial learning rate set to lr=0.001, and a momentum term. =0.9, =0.999. The training process uses a learning rate decay strategy; the learning rate is halved whenever the model's performance on the validation set no longer improves. Finally, the model is trained 250 times to obtain the model weights.

[0067] The training rate for the NWPU-RESISC45 remote sensing image dataset was 0.2, meaning 80% of the dataset was randomly divided into a training set and 20% into a validation set. The training set was used to train the CbCSANet-50 classification network, and the validation set was used to validate the performance of the trained classification network. The training input consisted of images and their corresponding class labels, with the cross-entropy between the model output value and the label used as the objective function. The model's weight parameters were iteratively calculated and updated using backpropagation and gradient descent until the loss value of the objective function was less than a preset threshold or a preset number of training epochs was reached, thus completing the training of the backbone model embedded with a channel-wise spatial attention mechanism.

[0068] To verify the effectiveness of the proposed channel-wise spatial attention method, the implementation example compared ResNet-50 and ResNet-50 embedded with CBAM (CBAMNet-50) as comparative models. In the comparative experiment, CbCSANet-50, CBAMNet-50 and ResNet-50 all adopted the same data preprocessing strategy and model training strategy.

[0069] Experimental results show that CbCSANet-50 has a significantly higher overall accuracy (OA) compared to ResNet-50 and CBAMNet-50. Figure 5 As shown, they increased by 1.88% and 1.21% respectively.

[0070] Appendix Figure 6 Appendix Figure 7 and attached Figure 8 The values ​​represent the confusion matrices of ResNet-50, CBAMNet-50, and CbCSANet-50 on the NWPU-RESISC45 dataset. Comparisons show that CbCSANet-50 improves overall classification accuracy by 2%, 12%, 0%, 9%, and 11% respectively for the five categories with the worst overall accuracy: church, industrial area, medium-sized residential palace, and train station. Compared to CBAMNet-50, CbCSANet-50 improves overall classification accuracy by 5%, 13%, -2%, 14%, and 2% for the same five categories. Experimental results indicate that CbCSANet-50 provides a significant overall improvement in classification performance on the NWPU-RESISC45 dataset compared to both CBAMNet-50 and ResNet-50.

[0071] Appendix Figure 9 This example demonstrates the visualization results of the final layer of the network using GradCAM++. Two categories with high inter-class similarity—church-palace and railway-train station—were selected for visualization. The visualization results show that for the church category, CbCSANet-50 more accurately covers the church's iconic area—the dome—compared to CBAMNet-50 and ResNet-50. Conversely, in the palace category, CbCSANet-50 more accurately covers the palace's iconic feature—the rectangular tiled roof. For the railway category, CbCSANet-50's area of ​​interest more perfectly covers the railway and trains while ignoring adjacent buildings. Finally, for the train station category, all three networks reasonably cover the train station's iconic buildings and the adjacent railway, but CbCSANet-50's area of ​​interest is more concentrated and reasonable.

Claims

1. A channel-wise spatial attention method for image classification, characterized in that, The steps include: using a remote sensing image dataset as the dataset, constructing a neural network for classification; for a convolutional neural network model, assuming the output feature map of each layer is... , S1: Based on step size feature map It is divided into multiple subtensors in the spatial dimension. ; Step S1 specifically includes: S11: For any given feature map At height and width In terms of direction by step size Segment it to obtain multiple local feature block tensors. ,set up and Let represent the segmentation indices in the height and width directions, respectively. Then, the... Tensor The formula is: ； in , , Representing feature maps respectively Element index in channel, height, and width; S2: Spatial attention weights for each channel in the feature map are obtained individually using average pooling, 1D convolution, and the sigmoid activation function; step S2 specifically includes: S21: When executing S1, the feature map is processed in parallel. Use window size Step size is Average pooling operation obtains feature descriptors ,in ； Here and Used to compensate for height or width Cannot be Dimensional deviations during divisibility This represents the modulo operation; each pooling result element It can be represented as: ； in , These represent the current pooling window in and Actual dimensions of the direction Indicates the total number of elements in the window. Representing feature maps respectively Element index in channel, height, and width; S22: In the direction of the channel Perform segmentation to generate feature tensors for each channel. : S23: Using the reshape function to... Remodeling : ； S24: Using a convolution kernel size of 1D convolution and sigmoid activation function for Perform convolution and activation to obtain : ； in Indicates the kernel size as 1D convolution, This represents the sigmoid activation function; S25: Using the reshape function to... Remodeling : ； S26: Along the channel dimension spliced ​​together : ； S27: In spatial dimensions, with a step size of 1 pair Segmentation to obtain each size is tensor : ； in Tensor Element index in the dimensions of channel, height, and width; S3: Spatial attention weights obtained from S2 and each subtensor in S1 Performing element-wise product operation yields the adjusted subtensor ; S4: Execute the reverse operation of S1. splicing into input The same shape yields the output of channel-by-channel spatial attention (CbCSA). ; S5: This method is a plug-and-play approach that is placed at the end of each module of the backbone model.

2. The channel-by-channel spatial attention method according to claim 1, characterized in that, Step S3 specifically includes: S31: Will After broadcasting in the spatial dimension and Element-wise multiplication yields ; ； This represents element-wise multiplication. and These represent the split indices in the height and width directions, respectively.

3. The channel-by-channel spatial attention method according to claim 1, characterized in that, Step S4 specifically includes: S41: All calibrated block features Concatenated with the input feature map The same dimensions yield the final output of the channel-by-channel spatial attention module. The formula is as follows: ； in Indicates pixel height The corresponding window index; Indicates pixel height The corresponding window index, simultaneously satisfying , , and The window size in the boundary region is represented by the following formula. , Obtained through calculation.

4. The channel-by-channel spatial attention method according to claim 1, characterized in that, Step S5 specifically includes: placing the channel-by-channel spatial attention method after each stage of the backbone network.

Images