A feature map weighting method and device based on three-band attention mechanism

Abstract

The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a feature map weighting method and device based on a three-band attention mechanism, which aims to improve the effectiveness of feature extraction in deep learning networks. The present invention proposes a new attention weighting mechanism, which increases the expressiveness of feature maps through weighted operations, focuses on important features and suppresses unimportant features. First, a global pooling operation is adopted to separate the original feature map into column strips, row strips and channel strips; secondly, a series of weighted operations of compression and expansion are applied to each strip; then, all three weighted strips are expanded to the original feature map size, and the three weighted feature maps are added, followed by nonlinear processing. Finally, the obtained weighted feature map is multiplied by the original feature map, and the result is used as the subsequent input. The feature map weighting method based on the three-band attention mechanism proposed in the present invention improves the effectiveness of the feature map.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a feature map weighting method and device based on a three-band attention mechanism. Background Art

[0002] The attention mechanism is a brain signal processing mechanism unique to human vision. Human vision rapidly scans the entire image to identify the target area of interest, then devotes more attention to this area to obtain more detailed information about the target of interest while suppressing other unnecessary information. With the development and widespread adoption of deep learning, the application of the attention mechanism in deep learning has also received increasing attention. The weighted operation of the attention mechanism can effectively optimize the feature map extraction operation of deep learning networks.

[0003] However, the attention mechanisms proposed in recent years have significant drawbacks. Some attention mechanisms focus only on the spatial aspect of features, ignoring the channels; others focus only on the channels and ignore the spatial aspect, resulting in incomplete weighting. Furthermore, most attention mechanisms employ convolutional operations for weighting, which largely ignore global information, making the weighting effect less pronounced. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the existing technology and provide a feature map weighting method and device based on a three-band attention mechanism.

[0005] The object of the present invention is achieved through the following technical solution: a feature map weighting method based on a three-band attention mechanism, comprising the following steps:

[0006] (1) Separate the original feature map into column strips, row strips, and channel strips;

[0007] (2) Compression, normalization, nonlinear activation, expansion, and normalization are performed on the column strips, row strips, and channel strips to obtain corresponding weighted column strips, weighted row strips, and weighted channel strips;

[0008] (3) Expanding the dimensions of the weighted column strips, the weighted row strips, and the weighted channel strips to the size of the original feature map to obtain an expanded column feature map, an expanded row feature map, and an expanded channel feature map; then adding the three expanded feature maps, and performing nonlinear processing on the addition result to obtain a weighted feature map;

[0009] (4) Multiply the weighted feature map with the original feature map, and add the multiplication result to the original feature map to obtain the input feature map, and use the input feature map as the subsequent input.

[0010] Furthermore, the step (1) includes the following sub-steps:

[0011] (1.1) Perform average pooling on the row and channel dimensions of the original feature map F to obtain column strips T HF ; The dimension of the original feature map F is R C×H×W , where C is the channel dimension, H is the column dimension, and W is the row dimension; the column strip T HF The dimension is R 1×H×1 ;

[0012] (1.2) Perform average pooling on the column and channel dimensions of the original feature map F to obtain the row strip T WF ; The row strip T WF The dimension is R 1×1×W ;

[0013] (1.3) Perform average pooling on the row and column dimensions of the original feature map F to obtain the channel strip T CF ; The channel strip T CF The dimension is R C×1×1 .

[0014] Furthermore, the step (2) includes the following sub-steps:

[0015] (2.1) Using a multi-layer perceptron to compress the elements of the column strip, row strip, and channel strip, reducing the size of the hidden layer of the column strip, row strip, or channel strip by a factor of r;

[0016] (2.2) Use batch normalization to normalize the compressed elements, and use the ReLU activation function to perform nonlinear activation on the normalized elements;

[0017] (2.3) Using a multilayer perceptron to expand the nonlinearly activated elements, the size of the hidden layer of the column strip, row strip, or channel strip is expanded by a factor of r;

[0018] (2.4) Use batch normalization to normalize the expanded elements to obtain the weighted column strip T′ HF , weighted row strip T′ WF , weighted channel strip T′ CF .

[0019] Furthermore, the step (3) includes the following sub-steps:

[0020] (3.1) The weighted column strip T′ HF Multiply the row dimension by W and T′ HFMultiply the channel dimension by C to get the expanded column feature map T″ HF ;

[0021] The weighted row strip T′ WF Multiply the column dimension of H by T′ WF Multiply the channel dimension by C to get the expanded row feature map T″ WF ;

[0022] The weighted channel strip T′ CF Multiply the row dimension by W and T′ CF Multiply the column dimension by H to get the expanded channel feature map T″ CF ;

[0023] The expanded column feature map T″ HF The dimension is R C×H×W , the expanded row feature map T″ WF The dimension is R C×H×W , the expanded channel feature map T″ CF The dimension is R C×H×W ;

[0024] (3.2) The expanded column feature map T″ HF , expanded row feature map T″ WF And the expanded channel feature map T″ CF Add them and perform sigmoid nonlinear processing on the addition result to obtain the TSAM feature map T F , that is, T F =σ(T″ HF +T″ WF +T″ CF ), where σ represents the sigmoid nonlinear processing.

[0025] Furthermore, the step (4) is specifically as follows: the weighted feature map T F Multiply the original feature map F and add the multiplication result to the original feature map F to obtain the input feature map F′, that is, F′=F+F×T F , where × represents element-wise multiplication; the input feature map F′ is used as the subsequent input.

[0026] The present invention also provides a feature map weighting device based on a three-band attention mechanism, comprising one or more processors for implementing the above-mentioned feature map weighting method based on the three-band attention mechanism.

[0027] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned feature map weighting method based on the three-band attention mechanism.

[0028] The present invention has the following beneficial effects: The proposed method is applicable to most deep convolutional neural networks based on image processing. Compared with other mainstream deep convolutional neural network attention weighting mechanisms, this invention adopts a compression and expansion mechanism for attention weighting, which allows the weighting of the original feature map to be processed with only a small number of parameters and floating-point calculations. In addition, the invention has a simple structure and is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart of a feature map weighting method based on a three-band attention mechanism;

[0030] Figure 2 The overall network structure diagram of a feature map weighting method based on a three-band attention mechanism;

[0031] Figure 3 A gradient change heatmap of a feature map weighting method based on a three-band attention mechanism applied to the Xception network;

[0032] Figure 4 A feature map weighting method based on a three-band attention mechanism is applied to the Resnet50 structure to show the insertion position.

[0033] Figure 5 This is a segmentation effect diagram of a feature map weighting method based on a three-band attention mechanism applied to a landslide dataset;

[0034] Figure 6 This is a segmentation effect diagram of a feature map weighting method based on a three-band attention mechanism applied to a flood dataset;

[0035] Figure 7 The structural diagram of a feature map weighting device based on a three-band attention mechanism. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, rather than to represent all embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0037] Example 1

[0038] like Figure 1 and Figure 2 As shown, the present invention provides a feature map weighting method based on a three-band attention mechanism, comprising the following steps:

[0039] Step 1: Separate the original feature map into column strips, row strips, and channel strips, corresponding to the rows, columns, and channels of the original feature map respectively;

[0040] Step 1 is implemented through the following sub-steps:

[0041] (1.1) Perform average pooling on the row and channel dimensions of the original feature map F to obtain column strips T HF ; The dimension of the original feature map F is R C×H×W , where C is the channel dimension, H is the column dimension, and W is the row dimension; the column strip T HF The dimension is R 1×H×1 ;

[0042] (1.2) Perform average pooling on the column and channel dimensions of the original feature map F to obtain the row strip T WF ; The row strip T WF The dimension is R 1×1×W ;

[0043] (1.3) Perform average pooling on the row and column dimensions of the original feature map F to obtain the channel strip T CF ; The channel strip T CF The dimension is R C×1×1 .

[0044] The above steps are different from the previous attention mechanism that directly performs convolution on the feature map.

[0045] Step 2: Compression, normalization, nonlinear activation, expansion, and normalization are performed on the column strips, row strips, and channel strips to obtain corresponding weighted column strips, weighted row strips, and weighted channel strips; this allows for a significant reduction in the number of parameters while performing the weighting.

[0046] Step 2 is implemented through the following sub-steps:

[0047] (2.1) Use a multi-layer perceptron (MLP) to compress the elements of the column strips, row strips, and channel strips, and reduce the size of the hidden layer of the column strips, row strips, or channel strips by a factor of r; that is, reduce the dimension of the column strips to Reduce the dimension of the row strips to Reduce the channel strip dimensions to The purpose of using multi-layer perceptron (MLP) for compression processing is to reduce the number of parameters.

[0048] (2.2) Use Batch Normalization (BN) to normalize the compressed elements, and use the ReLU activation function to perform nonlinear activation on the normalized elements;

[0049] (2.3) Use a multi-layer perceptron (MLP) to expand the elements after nonlinear activation, and expand the size of the hidden layer of the column strip, row strip, or channel strip by r times; that is, the dimension of the column strip is increased from Expand to R 1×H×1 , change the row strip dimensions from Expand to R 1×1×W , change the channel strip dimensions from Expand to R C×1×1 The purpose of using a multi-layer perceptron for expansion processing is to expand the strip dimension to its original size for easy subsequent processing.

[0050] (2.4) Use batch normalization to normalize the expanded elements to obtain the weighted column strip T′ HF , weighted row strip T′ WF , weighted channel strip T′ CF The weighted column strip T' HF The dimension is R 1×H×1 , the weighted row strip T′ WF The dimension is R 1×1×W , the weighted channel strip T′ CF The dimension is R C ×1×1 ;.

[0051] Step 3: Expand the dimensions of the weighted column strips, weighted row strips, and weighted channel strips to the size of the original feature map to obtain the expanded column feature map, the expanded row feature map, and the expanded channel feature map; then add the three expanded feature maps, and perform nonlinear processing on the addition result to obtain the weighted feature map;

[0052] Step 3 is implemented through the following sub-steps:

[0053] (3.1) The weighted column strip T′ HF Multiply the row dimension by W and T′ HF Multiply the channel dimension by C to get the expanded column feature map T″ HF ;

[0054] The weighted row strip T′ WF Multiply the column dimension of H by T′ WF Multiply the channel dimension by C to get the expanded row feature map T″ WF ;

[0055] The weighted channel strip T′ CF Multiply the row dimension by W and T′ CFMultiply the column dimension by H to get the expanded channel feature map T″ CF ;

[0056] The expanded column feature map T″ HF The dimension is R C×H×W , the expanded row feature map T″ WF The dimension is R C×H×W , the expanded channel feature map T″ CF The dimension is R C×H×W ;

[0057] (3.2) The expanded column feature map T″ HF , expanded row feature map T″ WF And the expanded channel feature map T″ CF Add them and perform sigmoid nonlinear processing on the addition result to obtain the weighted feature map T F , that is, T F =σ(T″ HF +T″ WF +T″ CF ), where σ represents the sigmoid nonlinear processing.

[0058] Step 4: Multiply the weighted feature map with the original feature map, and add the multiplication result to the original feature map to obtain the input feature map, and use the input feature map as the subsequent input;

[0059] Step 4 is as follows:

[0060] Since we are conducting embedding experiments in the ResNet residual network in the experiment, we need to consider the mechanism of the residual mechanism;

[0061] The weighted feature map T F Multiply the original feature map F and add the multiplication result to the original feature map F to obtain the input feature map F′, that is, F′=F+F×T F , where × represents element-wise multiplication;

[0062] The input feature map F′ is used as the subsequent input.

[0063] Figure 3This paper presents a gradient change heatmap of a feature map weighting method based on a three-band attention mechanism applied to an Xception network. The three-band attention mechanism is embedded in the Xception network and compared with the original Xception network without any embedding. By comparing the heatmap states of different training rounds, it can be seen that the Xception network embedded with the three-band attention mechanism converges faster and more accurately, further verifying the role of the feature map weighting method based on the three-band attention mechanism provided by the present invention in promoting feature extraction.

[0064] Figure 4 A feature map weighting method based on a three-band attention mechanism is applied to the Resnet50 structure to show the insertion position. Figure 4 The figure shows the location where the three-band attention mechanism is embedded in the ResNet network, which is also the specific embedding location for our experiments. The experiments show that embedding the three-band attention mechanism in the position shown in the figure can achieve better results.

[0065] Figure 5 This figure shows the segmentation results of a feature map weighting method based on a three-band attention mechanism applied to a landslide dataset. The baseline network used is the Unet network. Mainstream attention mechanisms such as SE, CBAM, and BAM are embedded in the Unet network and compared with the three-band attention mechanism. It can be seen that the three-band attention mechanism is more effective in improving the performance of the Unet network than other attention mechanisms.

[0066] Figure 6 This figure shows the segmentation results of a feature map weighting method based on a three-band attention mechanism applied to the flood dataset. The baseline network used is the Unet network. Mainstream attention mechanisms such as SE, CBAM, and BAM are embedded in the Unet network and compared with the three-band attention mechanism. It can be seen that the three-band attention mechanism is more effective in improving the performance of the Unet network than other attention mechanisms.

[0067] See also Figure 7 , an embodiment of the present invention provides a feature map weighting device based on a three-band attention mechanism, including one or more processors for implementing the feature map weighting method based on the three-band attention mechanism in the above embodiment.

[0068] The embodiment of the feature map weighting device based on the three-band attention mechanism of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 7 As shown, this is a hardware structure diagram of any device with data processing capability where the feature map weighting device based on the three-band attention mechanism of the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0069] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0070] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0071] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the feature map weighting method based on the three-band attention mechanism in the above embodiment is implemented.

[0072] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A feature map weighting method based on a three-band attention mechanism, suitable for deep convolutional neural networks based on image processing, characterized in that: The following steps are involved: (1) Separate the original feature map into column strips, row strips, and channel strips; The step (1) includes the following sub-steps: (1.1) Perform average pooling on the row and channel dimensions of the original feature map F to obtain column strips T HF ; The dimension of the original feature map F is , where C is the channel dimension, H is the column dimension, and W is the row dimension; the column strip T HF The dimension is ; (1.2) Perform average pooling on the column and channel dimensions of the original feature map F to obtain row strips T WF ; The row strip T WF The dimension is ; (1.3) Perform average pooling on the row and column dimensions of the original feature map F to obtain the channel strip T CF ; The channel strip T CF The dimension is ; (2) Compression, normalization, nonlinear activation, expansion, and normalization are performed on the column strips, row strips, and channel strips to obtain the corresponding weighted column strips, weighted row strips, and weighted channel strips; The step (2) includes the following sub-steps: (2.1) Using a multilayer perceptron to compress the elements of the column strip, row strip, and channel strip, reducing the size of the hidden layer of the column strip, row strip, or channel strip by a factor of r; (2.2) Use batch normalization to normalize the compressed elements and use the ReLU activation function to perform nonlinear activation on the normalized elements; (2.3) Using a multilayer perceptron to expand the nonlinearly activated elements, the size of the hidden layer of the column strip, row strip, or channel strip is expanded by a factor of r; (2.4) Use batch normalization to normalize the expanded elements to obtain weighted column stripes , weighted row strips , weighted channel strips ; (3) The dimensions of the weighted column strips, the weighted row strips, and the weighted channel strips are expanded to the size of the original feature map to obtain the expanded column feature map, the expanded row feature map, and the expanded channel feature map; then the three expanded feature maps are added together, and the addition result is nonlinearly processed to obtain the weighted feature map; The step (3) includes the following sub-steps: (3.1) The weighted column strips Multiply the row dimension by W and Multiply the channel dimension by C to get the expanded column feature map ; The weighted row strips Multiply the column dimension of by H, and Multiply the channel dimension by C to get the expanded row feature map ; The weighted channel strips Multiply the row dimension by W and Multiply the column dimension by H to get the expanded channel feature map ; The expanded column feature map The dimension is , the expanded row feature map The dimension is , the expanded channel feature map The dimension is ; (3.2) The expanded column feature map , expanded row feature map And the expanded channel feature map Add them and perform sigmoid nonlinear processing on the addition result to obtain the TSAM feature map T F ,Right now , in Indicates sigmoid nonlinear processing; (4) Multiply the weighted feature map with the original feature map, and add the multiplication result to the original feature map to obtain the input feature map, and use the input feature map as the subsequent input; The step (4) is specifically as follows: weighted feature map T F Multiply the original feature map F and add the multiplication result to the original feature map F to obtain the input feature map ,Right now ,in Represents the multiplication of elements; the input feature map as subsequent input.

2. A feature map weighting device based on a three-band attention mechanism, characterized in that: Includes one or more processors for implementing the feature map weighting method based on the three-band attention mechanism according to claim 1.

3. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the feature map weighting method based on the three-band attention mechanism described in claim 1.

Images