A spatial attention calculation method based on channel attention
Generating spatial attention maps through channel separation technology and nonlinear operations, the problem of lack of combination of spatial attention and channel attention in the existing technology is solved, and the performance performance of deep convolutional neural networks is improved.
Patent Information
- Application Number
- CN202210087706.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-01-25
AI Technical Summary
Although the existing deep convolutional neural network has introduced channel attention and spatial attention mechanisms in visual tasks such as image classification, it has failed to effectively combine the two, resulting in a lack of close connection between spatial attention and channel attention, which affects the model performance. Performance.
A spatial attention calculation method based on channel attention is proposed. The channel refinement feature map is divided into important features and secondary important features through channel separation technology, and average pooling and maximum pooling operations are carried out to generate spatial feature descriptors. Combining nonlinear operations and convolutional layers, a two-dimensional spatial attention map is generated to realize the organic connection between spatial attention and channel attention.
While maintaining the original channel sequence, this method improves the combination of spatial attention and channel attention, enhances the characteristic representation of the network model in the spatial dimension, improves the model performance, and can seamlessly embed mainstream deep convolutional nerves In the network.
Smart Images

Figure CN114492631B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep convolutional neural networks, channel attention and spatial attention technology, and in particular to a spatial attention calculation method based on channel attention. Background Art
[0002] In the past few years, many scholars have been keen on exploring deeper neural networks to improve the performance of models in visual tasks. However, as the number of neural network layers increases, the accuracy of visual tasks such as image classification and object detection has not improved significantly, that is, the improvement in network performance is not proportional to the depth of the network layers.
[0003] Researchers in the biomedical field have found that in the human visual system, the attention mechanism plays an important role in obtaining important information. Inspired by this phenomenon, many scholars have tried to combine the attention mechanism in deep convolutional neural networks to improve model performance. Among the work of these scholars, the most representative one is Squeeze-and-Excitation (SENet) proposed by Hu et al. in 2017. This attention module strengthens the key features of the channel dimension of the feature map, thereby improving the performance of the model in image classification tasks. SENet is a relatively lightweight embedded attention module that can be embedded in any current mainstream deep convolutional network architecture. Based on the SENet module, Efficient Channel Attention (ECA) was proposed by Wang et al. in 2020. This module uses one-dimensional convolution to replace the fully connected layer in the original SENet, which greatly improves the derivation speed of channel attention and overcomes the accuracy loss caused by channel dimensionality reduction.
[0004] Both SENet and ECA modules reveal the relationship between feature map channels, but neither of them considers the importance of spatial attention, that is, they do not consider the key feature enhancement of feature maps in the spatial dimension. Spatial attention is the enhancement of key pixels of feature maps. In 2018, Woo et al. proposed the Convolutional Block Attention Module (CBAM). This attention module is the first module to consider the simultaneous use of channel attention and spatial attention. It connects the channel attention submodule and the spatial attention submodule in a sequential manner, that is, the key features in the channel dimension are first enhanced through the channel attention module, and then the key features in the spatial dimension are enhanced through the spatial attention module. However, the spatial attention submodule in the CBAM module is not organically combined with the channel attention submodule, that is, the spatial attention still puts important channels and less important channels together for pooling operations. This will cause the original key features in the feature map to weaken with the average pooling operation, and thus the performance of the network model is not fully utilized. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a spatial attention calculation method based on channel attention. The proposed channel separation technology makes the spatial attention and channel attention closely and organically linked, which is convenient for embedding into any mainstream deep convolutional neural network at present, thereby improving the performance of the network model.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] A spatial attention calculation method based on channel attention includes the following steps:
[0008] Step 1: Obtain channel attention map and channel refinement feature map;
[0009] Step 2: Group the channels of the channel refinement feature map obtained in step 1, and divide the channels of the channel refinement feature map into important features and less important features;
[0010] Step 3: Perform average pooling and maximum pooling operations in the channel axis direction on the important features and the less important features obtained in step 2; perform feature aggregation on the two pooling results;
[0011] Step 4: Pass the aggregated result obtained in step 3 through a shared convolutional layer to generate a pair of spatial feature descriptors;
[0012] Step 5: Perform nonlinear operation on the pair of spatial feature descriptors obtained in step 4, i.e., pass through a BN layer and a ReLU layer;
[0013] Step 6: Aggregate the two spatial feature descriptors obtained in step 5 in the channel dimension, and then pass through a convolution layer to obtain the spatial attention descriptor;
[0014] Step 7: Pass the spatial attention descriptor obtained in step 6 through the BN layer, ReLU layer and Sigmoid layer to obtain a 2D spatial attention map.
[0015] A further improvement of the technical solution of the present invention is that the channel attention map and channel refinement features obtained in step 1 are both generated by forward deduction of the SENet or ECA module; wherein the channel attention map is a 1D vector, and each value in the vector corresponds to the importance of the corresponding channel in the channel refinement feature.
[0016] A further improvement of the technical solution of the present invention is that: in step 2, the channel separation technology is used to group the channels of the channel refinement features into important features and less important features. The specific operation steps include:
[0017] (1) Obtaining the channel dimension of the channel refinement feature map, that is, the number of channels C;
[0018] (2) Introduce a hyperparameter: separation rate λ, which is the dividing line between important channels and less important channels; multiply the number of channels by the separation rate and take the nearest even number of the product to obtain the number of important channels C in ; Use the number of channels to subtract the number of important channels to obtain the number of less important channels C subin ; The above calculation process is shown in formulas (1) and (2):
[0019] C im =|C·λ| even (1)
[0020] C subim =CC im (2)
[0021] where |t| even It represents the even number closest to the t value;
[0022] (3) Find the top C in the channel attention map in The maximum value, at the same time in this C in Determine the smallest value m among the large values;
[0023] (4) Compare the value in the channel attention map with m, reassign the value greater than or equal to m to 1, and reassign the value less than m to 0, thereby obtaining the important channel mask important_mask;
[0024] (5) Compare the value in the channel attention map with m, reassign values greater than or equal to m to 0, and reassign values less than m to 1, thereby obtaining the sub-important channel mask sub-important_mask;
[0025] (6) The important channel mask and the less important channel mask are element-wise multiplied with the channel refinement feature map to obtain the important features and the less important features.
[0026] A further improvement of the technical solution of the present invention is that: in step 3, average pooling and maximum pooling operations are performed on both important features and less important features along the channel axis direction; after the pooling operation is completed, the two pooling results are aggregated in the channel dimension.
[0027] A further improvement of the technical solution of the present invention is that: in step 4, a common convolution layer is used for the aggregation result to obtain a pair of spatial feature descriptors, and the convolution kernel size of the convolution layer is 7×7.
[0028] A further improvement of the technical solution of the present invention is that in step 5, the two spatial feature descriptors obtained in step 4 are batch normalized and activated using a ReLU function.
[0029] A further improvement of the technical solution of the present invention is that: in step 6, the spatial feature descriptor after nonlinear operation is feature aggregated in the channel dimension, and then passed through a convolution layer to obtain a spatial attention descriptor; this convolution layer is different from the previous shared convolution layer, and its convolution kernel size is still 7×7, and the selection of the convolution kernel size is consistent with the idea of selecting the convolution kernel size of the shared convolution layer.
[0030] A further improvement of the technical solution of the present invention is that: in the step 7, a series of nonlinear operations are performed on the spatial attention descriptor, and the order of the nonlinear operations is BN layer, ReLU layer and Sigmoid layer, and finally a 2D spatial attention map is obtained; the attention map is a two-dimensional matrix, and the size is consistent with the original channel refinement feature map;
[0031] The above steps can be expressed as mathematical formulas, as shown in formulas (3), (4), (5), and (6):
[0032] F'=F'1+F'2 (3)
[0033] S1=Φ1(C2D 1,7×7 [AvgPool(F'1);MaxPool(F'1)]) (4)
[0034] S2=Φ1(C2D 1,7×7 [AvgPool(F'2);MaxPool(F'2)]) (5)
[0035] A S =Φ2(C2D 2,7×7 [S1; S2]) (6)
[0036] Wherein, (3) represents the process of channel separation, F' represents the channel refinement feature, F'1 represents the important feature, and F'2 represents the less important feature; (4) where Φ1 represents the set of BN layer and ReLU layer, i.e., nonlinear operation; C2D 1,7×7 represents a shared convolutional layer with a convolution kernel size of 7×7; [a; b] represents the aggregation of features a and b; AvgPool represents the average pooling operation; MaxPool represents the maximum pooling operation; (4) generates the spatial feature descriptor S1 corresponding to the important feature, and (5) generates the spatial feature descriptor S2 corresponding to the less important feature; (6) where Φ2 represents the set of BN layer, ReLU layer and Sigmoid layer; C2D 2,7×7 Represents the second convolutional layer, whose convolution kernel size is 7×7.
[0037] The further improvement of the technical solution of the present invention is:
[0038] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:
[0039] The channel separation technology proposed in the present invention divides the important channels and the less important channels into two parts while maintaining the original channel order, which makes the spatial attention and channel attention closely and organically linked.
[0040] The dual-branch spatial attention module proposed in the present invention can be seamlessly connected to the channel attention module, that is, it can be embedded in any current mainstream deep convolutional neural network.
[0041] The dual-branch spatial attention module proposed in the present invention can further enhance key features in the spatial dimension, thereby improving the performance of the network model.
[0042] In step 5, the two spatial feature descriptors are batch normalized and activated using the ReLU function to accelerate network training. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 Schematic diagram of the structure of the dual-branch spatial attention module of the present invention;
[0044] Figure 2 Flowchart for the implementation of the invented dual-branch spatial attention module;
[0045] Figure 3Schematic diagram of embedding a dual-branch spatial attention module when the residual unit already has a channel attention module. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below in conjunction with embodiments:
[0047] like Figures 1 to 3 As shown, a spatial attention calculation method based on channel attention is a new spatial attention module, a dual-branch spatial attention module. The channel attention module forward derives a channel attention map (Channel attention map) and a channel refined feature map (Channel refined feature map). According to the channel attention map, the dual-branch spatial attention module groups the channels of the channel refined feature map into important channels and less important channels. This organically links spatial attention and channel attention. Then, a series of nonlinear operations are performed on the divided channel groups to obtain a two-dimensional spatial attention map. The present invention has important guiding significance for calculating spatial attention based on channel attention.
[0048] Its structure diagram is as follows Figure 1 As shown, the following steps are included:
[0049] Step 1: Get the results of the forward derivation of the channel attention module: channel attention map and channel refined feature map.
[0050] The channel attention map is a one-dimensional vector with each value between 0 and 1. Here, we regard each value as the importance of the corresponding channel in the channel refinement feature map. This is the key to the close organic connection between spatial attention and channel attention.
[0051] Step 2: According to the physical meaning of each value in the channel attention map, the channel separation technology proposed in this invention is used to group the channels of the channel refinement feature map into two groups: important channels and sub-important channels, that is, important features and sub-important features. We name this channel grouping method as channel separation technology, and the specific operation is as follows:
[0052] (1) First, obtain the channel dimension of the channel refinement feature map, that is, the number of channels C;
[0053] (2) Introduce a hyperparameter: separation rate λ, which is the dividing line between important channels and less important channels. Multiply the number of channels by the separation rate and take the nearest even number of the product to get the number of important channels C.im Subtract the number of important channels from the number of channels to get the number of less important channels C. subim The above calculation process is shown in formulas (1) and (2):
[0054] C im =|C·λ| even (1)
[0055] C subim =CC im (2)
[0056] where |t| even Represents the even number closest to the value of t.
[0057] (3) Find the top C in the channel attention map im The maximum value, at the same time in this C im Determine the smallest value m among the large values.
[0058] (4) Compare the value in the channel attention map with m, reassign the value greater than or equal to m to 1, and reassign the value less than m to 0, thereby obtaining the important channel mask important_mask.
[0059] (5) Compare the value in the channel attention map with m, reassign the value greater than or equal to m to 0, and reassign the value less than m to 1, thereby obtaining the sub-important channel mask sub-important_mask.
[0060] (6) The important channel mask and the less important channel mask are element-wise multiplied with the channel refinement feature map to obtain the important features and the less important features.
[0061] It is worth noting that after channel separation, the channel order in the original channel refinement feature map remains unchanged, the important features retain the important channel information, and only the values in the less important channels are changed to 0. Conversely, the less important features retain the less important channel information, and only the values in the important channels are changed to 0.
[0062] Step 3: Perform average pooling and maximum pooling operations on both important features and less important features along the channel axis. This can obtain the average pooling and maximum pooling feature descriptors of important features and the average pooling and maximum pooling feature descriptors of less important features. Aggregate the average pooling and maximum pooling feature descriptors of important features in the channel dimension. Similarly, aggregate the average pooling and maximum pooling feature descriptors of less important features in the channel dimension.
[0063] Step 4: Use a shared convolution layer on the result of feature aggregation to obtain a pair of spatial feature descriptors. The size of the shared convolution layer is 7×7. It is worth noting that a larger convolution kernel means a larger receptive field, but the performance of the module does not improve with the increase of the size of the shared convolution kernel. The size of the final feature map needs to be considered at the same time.
[0064] Step 5: Batch normalize the two spatial feature descriptors and perform nonlinear operations on the pair of spatial feature descriptors, that is, pass them through the BN layer and use the ReLU function to activate them.
[0065] Step 6: After the nonlinear operation, the spatial feature descriptor is aggregated in the channel dimension and passed through a convolution layer to obtain the spatial attention descriptor. The size of this convolution layer is still 7×7, and the selection of the convolution kernel size is consistent with the idea of selecting the convolution kernel size of the shared convolution layer, which will not be repeated here.
[0066] Step 7: Perform a series of nonlinear operations on the spatial attention descriptor. The order of nonlinear operations is BN layer, ReLU layer, and Sigmoid layer. So far, the spatial attention map A is obtained. S The attention map is a two-dimensional matrix whose size is consistent with the channel refinement feature map.
[0067] The above steps can be expressed as mathematical formulas, as shown in formulas (3), (4), (5), and (6):
[0068] F'=F'1+F'2 (3)
[0069] S1=Φ1(C2D 1,7×7 [AvgPool(F'1);MaxPool(F'1)]) (4)
[0070] S2=Φ1(C2D 1,7×7 [AvgPool(F'2);MaxPool(F'2)]) (5)
[0071] A S =Φ2(C2D 2,7×7 [S1; S2]) (6)
[0072] Where (3) represents the process of channel separation, F' represents the channel refinement feature, F'1 represents the important feature, and F'2 represents the less important feature. (4) Where Φ1 represents the set of BN layers and ReLU layers, i.e., nonlinear operation; C2D 1,7×7represents a shared convolutional layer with a convolution kernel size of 7×7; [a; b] represents the aggregation of features a and b; AvgPool represents the average pooling operation; MaxPool represents the maximum pooling operation. The symbols in (5) and (4) are the same and will not be repeated here. (4) generates the spatial feature descriptor S1 corresponding to the important feature, and (5) generates the spatial feature descriptor S2 corresponding to the less important feature. (6) In the formula, Φ2 represents the set of BN layer, ReLU layer and Sigmoid layer; C2D 2,7×7 Represents the second convolutional layer, whose convolution kernel size is 7×7.
[0073] Step 8: Multiply the channel refined feature map and the spatial attention map element-wise to obtain the final refined feature map.
[0074] The final refined feature map can be expressed by the following formula, as shown in formula (7):
[0075]
[0076] Where F” represents the final refined feature; Represents matrix element-wise multiplication.
[0077] Here, taking the current mainstream residual convolutional neural network as an example, the dual-branch spatial attention module is embedded in the residual unit of the existing channel attention module. Its structure is shown in the figure Figure 3 As shown in the figure, the dual-branch spatial attention module will further strengthen the key features of the feature map in the spatial dimension after the channel attention module, thereby further improving the performance of the network model.
Claims
1. A spatial attention calculation method based on channel attention, characterized in that: The following steps are involved: Step 1: Obtain channel attention map and channel refinement feature map; Step 2: Group the channels of the channel refinement feature map obtained in step 1, and divide the channels of the channel refinement feature map into important features and less important features; The channel separation technology is used to group the channels of the channel refinement features into important features and less important features. The specific operation steps include: (1) Obtaining the channel dimension of the channel refinement feature map, that is, the number of channels C; (2) Introduce a hyperparameter: separation rate λ, which is the dividing line between important channels and less important channels; multiply the number of channels by the separation rate and take the nearest even number of the product to obtain the number of important channels C im ; Use the number of channels to subtract the number of important channels to obtain the number of less important channels C subim ; The above calculation process is shown in formulas (1) and (2): C im =|C · λ| even (1) C subim =C - C im (2) where |t| even It represents the even number closest to the t value; (3) Find the top C in the channel attention map im The maximum value, at the same time in this C im Determine the smallest value m among the large values; (4) Compare the value in the channel attention map with m, reassign the value greater than or equal to m to 1, and reassign the value less than m to 0, thereby obtaining the important channel mask important_mask; (5) Compare the value in the channel attention map with m, reassign values greater than or equal to m to 0, and reassign values less than m to 1, thereby obtaining the sub-important channel mask sub-important_mask; (6) element-wise multiplying the important channel mask and the less important channel mask with the channel refinement feature map, respectively, to obtain the important features and the less important features; Step 3: Perform average pooling and maximum pooling operations in the channel axis direction on the important features and the less important features obtained in step 2; perform feature aggregation on the two pooling results; Step 4: Pass the aggregated result obtained in step 3 through a shared convolutional layer to generate a pair of spatial feature descriptors; Step 5: Perform nonlinear operation on the pair of spatial feature descriptors obtained in step 4, i.e., pass through a BN layer and a ReLU layer; Step 6: Aggregate the two spatial feature descriptors obtained in step 5 in the channel dimension, and then pass through a convolution layer to obtain the spatial attention descriptor; Step 7: Pass the spatial attention descriptor obtained in step 6 through the BN layer, ReLU layer and Sigmoid layer to obtain a 2D spatial attention map.
2. According to claim 1, a spatial attention calculation method based on channel attention is characterized in that: The channel attention map and channel refinement features obtained in step 1 are both generated by forward deduction of the SENet or ECA module; wherein the channel attention map is a 1D vector, and each value in the vector corresponds to the importance of the corresponding channel in the channel refinement feature.
3. The spatial attention calculation method based on channel attention according to claim 1, characterized in that: In step 3, average pooling and maximum pooling operations are performed on both important features and less important features along the channel axis. After the pooling operation is completed, the two pooling results are aggregated in the channel dimension.
4. The spatial attention calculation method based on channel attention according to claim 1, characterized in that: In step 4, a common convolutional layer is applied to the aggregation result to obtain a pair of spatial feature descriptors. The convolution kernel size of the convolutional layer is 7×7.
5. The spatial attention calculation method based on channel attention according to claim 1, characterized in that: In step 5, the two spatial feature descriptors obtained in step 4 are batch normalized and activated using the ReLU function.
6. The spatial attention calculation method based on channel attention according to claim 1, characterized in that: In step 6, the spatial feature descriptors after nonlinear operations are aggregated in the channel dimension, and then passed through a convolution layer to obtain the spatial attention descriptor; this convolution layer is different from the previous shared convolution layer, and its convolution kernel size is still 7×7, and the selection of the convolution kernel size is consistent with the idea of selecting the convolution kernel size of the shared convolution layer.
7. According to claim 1, a spatial attention calculation method based on channel attention is characterized in that: In step 7, a series of nonlinear operations are performed on the spatial attention descriptor, and the order of the nonlinear operations is BN layer, ReLU layer and Sigmoid layer, and finally a 2D spatial attention map is obtained; the attention map is a two-dimensional matrix, and the size is consistent with the original channel refinement feature map; The above steps can be expressed as mathematical formulas, as shown in formulas (3), (4), (5), and (6): F'= F'1 + F'2 (3) S1= Φ1(C2D 1,7×7 [AvgPool(F'1); MaxPool(F'1)]) (4) S2= Φ1(C2D 1,7×7 [AvgPool(F'2); MaxPool(F'2)]) (5) A S = Φ2(C2D 2,7×7 [S1; S2]) (6) Wherein, (3) represents the process of channel separation, F' represents the channel refinement feature, F'1 represents the important feature, and F'2 represents the less important feature; (4) where Φ1 represents the set of BN layers and ReLU layers, i.e., nonlinear operation; C2D 1,7×7 represents a shared convolutional layer with a convolution kernel size of 7×7; [a; b] represents the aggregation of features a and b; AvgPool represents the average pooling operation; MaxPool represents the maximum pooling operation; (4) generates the spatial feature descriptor S1 corresponding to the important feature, and (5) generates the spatial feature descriptor S2 corresponding to the less important feature; (6) where Φ2 represents the set of BN layer, ReLU layer and Sigmoid layer; C2D 2,7×7 Represents the second convolutional layer, whose convolution kernel size is 7×7.
Citation Information
Patent Citations
Image semantic segmentation model based on context awareness and modeling method
CN111626300A
Lightweight neural network model based on channel attention module
CN113344188A
Shadow detection method based on mixed attention
CN113642634A
KR20190113119A
