A target detection method based on spatial attention and channel attention

Abstract

The present invention discloses a target detection method based on spatial attention and channel attention, which collects and obtains a detection image, which is an infrared image; generates a convolutional neural network CNN according to a target feature map training, and performs target detection on the infrared image; in the convolutional neural network CNN, the feature map generated in a relatively shallow layer is spliced ​​with the feature map generated in a deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed; multiple pooling layers in the convolutional neural network CNN generate their corresponding weights; the attention of different channels where different pooling layers are located is enhanced or suppressed to generate a target feature map. By highlighting multiple targets of interest under different high-order semantics and suppressing background noise, multiple different pooling layers are used to selectively enhance or suppress specific channels, so that the allocation of attention resources is more reasonable.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a target detection method based on spatial attention and channel attention. Background Art

[0002] Infrared imaging technology can convert the invisible surface temperature of an object into a visible thermal image that characterizes the temperature distribution on the surface of the object. Infrared imaging technology uses the infrared spectrum of the target to perform imaging, which makes it have better anti-interference performance and a larger working range in bad weather compared to the visible spectrum. These characteristics make it highly valuable in security systems, fire alarms, autonomous driving and other fields.

[0003] For many years, object detection in infrared images has been a challenging task. Compared with optical images, infrared images are obtained by "measuring" the heat radiated by objects, which makes infrared images have problems such as low resolution, low contrast, and low signal-to-noise ratio (SNR). Therefore, the object detection task of infrared images is more difficult than that of visible light images. In daily life, the infrared characteristics of most objects are not obvious, and the boundaries between them are also fuzzy. When the background of the target area is complex, it will bring a lot of trouble to the task. In recent years, thanks to the rapid development of Convolutional Neural Network (CNN) and its powerful performance, infrared object detection has ushered in new development opportunities. Through a large amount of data training, CNN can be well adapted to the model of specific tasks. Influential methods such as SSD, YOLO, Retina-Net, FasterR-CNN, ResNet, etc. have demonstrated their excellent performance. However, these methods are designed for traditional object detection tasks, and less consideration is given to infrared object detection tasks. Optical images, thanks to their relatively high resolution, contain rich fine-grained details, which are extremely important for target detection tasks. However, due to the lack of information in infrared images, it is difficult to extract the inherent features of objects in the image for detection tasks.

[0004] In the prior art, some methods have been proposed to improve the infrared target feature extraction capability, such as deriving the optimal feature vector through the target to clutter ratio (TCR) metric, which can both represent the target and distinguish the target from the clutter; and combining the two-dimensional wavelet transform (decomposing the image into uniform subbands) with an efficient algorithm called Reed Xiaoli (RX) to detect multivariate anomalies; and using the target mapping function to calculate the feature vector obtained at each position in the image, with the larger value indicating the target position. These methods improve the detection performance of infrared targets. However, they involve a large number of mathematical operations and have high complexity, which means that they are prone to encounter operations that are not integrated into the deep learning framework. This makes it difficult for them to fully utilize the acceleration function of the GPU, ultimately resulting in reduced training and detection speeds, and poor versatility. On the other hand, inspired by the human visual attention mechanism, researchers introduced the attention mechanism into CNN to improve the performance of the model at a low cost. By roughly scanning the global area of ​​the image, humans can easily determine the target area that needs attention. Similarly, the attention mechanism will also give weights to the information processed by the method, and invest more attention resources in areas with high weights, obtain detailed information from areas with high weights, and suppress other irrelevant areas. However, most existing studies directly embed the attention mechanism into all CNN trunk residual structures. In fact, how to embed the attention mechanism into CNN and which layer of CNN to embed into also have a certain impact on the performance of the method. To this end, the present invention proposes a target detection method based on spatial attention and channel attention, aiming to solve the above technical defects. Summary of the invention

[0005] In order to achieve the above objectives, a target detection method based on spatial attention and channel attention is proposed, including:

[0006] Collecting and acquiring a detection image, wherein the detection image is an infrared image;

[0007] Generating a convolutional neural network (CNN) according to the target feature map training, and performing target detection on the infrared image according to the convolutional neural network (CNN);

[0008] In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed;

[0009] The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map.

[0010] Preferably, the spatial enhancement operation includes: obtaining global and local statistical features through a global average function and a local average function, and approximating the semantic vector represented by the group learning:

[0011]

[0012] Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the feature image after grouping;

[0013] Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i :

[0014] c ij =g ij ·x ij

[0015] Perform spatial normalization on c:

[0016]

[0017]

[0018]

[0019] Among them, m = H × W, is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a pair of parameters γ and β are introduced for each coefficient to scale and shift the normalized result:

[0020]

[0021] Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features.

[0022]

[0023] Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

[0024] Preferably, there are three pooling layers; enhancing or suppressing the attention of different channels in different pooling layers includes:

[0025] A(F)=ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))}

[0026] Where A(F) is the attention of the channel, the MLP weights of all inputs are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function;

[0027] Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

[0028]

[0029] Preferably, the ES{} operation includes: if any parameter value in the parameters is greater than 0.5, taking the maximum value as the feature map weight; if any value of the three vectors at the same position is less than 0.5, taking the minimum value as the feature map weight.

[0030] Preferably, in the convolutional neural network CNN, the feature map generated in a relatively shallow layer is spliced ​​with the feature map generated in a deep layer through a slice splicing operation, including:

[0031] Generate an all-zero tensor with the same shape as the tensor to be sliced ​​and concatenated, with the number of channels set to C;

[0032] Concatenate the shallow tensor with the all-zero tensor, and then concatenate the all-zero tensor with the deep tensor to form two new tensors with 2C channels to be operated.

[0033] Use the reshape operation to adjust the channel dimensions of these two tensors to (C, 2) and transpose them to (2, C);

[0034] Transform it to 2C channels according to the reshape function and add the two tensors together.

[0035] In addition, a target detection device based on spatial attention and channel attention is proposed, including:

[0036] An acquisition module is used to acquire and obtain a detection image, wherein the detection image is an infrared image;

[0037] A detection module, which generates a convolutional neural network (CNN) according to the target feature map training, and performs target detection on the infrared image according to the convolutional neural network (CNN);

[0038] In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed;

[0039] The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map.

[0040] Preferably, the spatial enhancement operation includes: obtaining global and local statistical features through a global average function and a local average function, and approximating the semantic vector represented by the group learning:

[0041]

[0042] Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the feature image after grouping;

[0043] Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i :

[0044] c ij =g ij ·x ij

[0045] Perform spatial normalization on c:

[0046]

[0047]

[0048]

[0049] Among them, m = H × W, is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a pair of parameters γ and β are introduced for each coefficient to scale and shift the normalized result:

[0050]

[0051] Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features.

[0052]

[0053] Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

[0054] Preferably, there are three pooling layers; enhancing or suppressing the attention of different channels in different pooling layers includes:

[0055] A(F)=ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))}

[0056] Where A(F) is the attention of the channel, the MLP weights of all inputs are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function;

[0057] Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

[0058]

[0059] In addition, an electronic device is proposed, including a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the target detection method based on spatial attention and channel attention as described above.

[0060] In addition, a computer storage medium is also proposed, which stores a program; the program is loaded and executed by a processor to implement the target detection method based on spatial attention and channel attention as described above.

[0061] The target detection method based on spatial attention and channel attention provided by the present invention collects and obtains a detection image, which is an infrared image; generates a convolutional neural network CNN according to the target feature map training, and performs target detection on the infrared image according to the convolutional neural network CNN; wherein, in the convolutional neural network CNN, the feature map generated in a relatively shallow layer is spliced ​​with the feature map generated in a deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed; the multiple pooling layers in the convolutional neural network CNN generate their corresponding weights; the attention of different channels where different pooling layers are located is enhanced or suppressed to generate the target feature map. The present invention effectively highlights multiple targets of interest under different high-order semantics and suppresses background noise through the spatial attention mechanism; in terms of the channel attention mechanism, multiple different pooling layers are used to selectively enhance or suppress specific channels, so that the allocation of attention resources is more reasonable; and it is embedded in the FPN structure instead of the backbone network to achieve better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a target detection method based on spatial attention and channel attention according to an embodiment of the present invention;

[0063] Figure 2 This is a general structural diagram of the SCMR-SGE module implemented in Example 1 of the present invention;

[0064] Figure 3 This is a comparison diagram of the group characteristics of the standard convolution in Example 1 of the present invention and this embodiment;

[0065] Figure 4 FIG. 1 is an overall structural diagram of the implementation of the joint attention (JA) module according to the first embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the structure of YOLOv5s according to the first embodiment of the present invention;

[0067] Figure 6 It is a part of the images in the FLIR data set of the first embodiment of the present invention;

[0068] Figure 7 This is an output comparison diagram of the FLIR data set detected by YOLOv5s and this method in Example 1 of the present invention;

[0069] Figure 8 This is a schematic diagram of the structure of a target detection device based on spatial attention and channel attention according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention are described below in conjunction with the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other if there is no conflict.

[0071] Embodiment 1

[0072] Reference Figure 1 , Figure 1 is a flow chart of a target detection method based on spatial attention and channel attention of the present invention. In this embodiment, a target detection method based on spatial attention and channel attention is proposed, including:

[0073] S1, collecting and acquiring a detection image, wherein the detection image is an infrared image;

[0074] S2, generating a convolutional neural network CNN according to the target feature map training, and performing target detection on the infrared image according to the convolutional neural network CNN;

[0075] In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed;

[0076] The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map.

[0077] Preferably, in the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slicing and splicing operation, including: generating an all-zero tensor with the same shape as the tensor to be sliced ​​and spliced, with the number of channels set to C; splicing the shallow tensor with the all-zero tensor, and then splicing the all-zero tensor with the deep tensor to form two new tensors to be operated with a channel number of 2C; adjusting the channel dimensions of the two tensors to (C, 2) using a reshape operation, and transposing them to (2, C); transforming them into 2C channels according to the reshape function, and adding the two tensors.

[0078] Specifically, in this embodiment, the features generated by the traditional CNN are composed of many sub-features, which are usually distributed in the features of each layer in the form of groups. By scaling the feature vectors of all positions in each group, the attention mechanism SGE module can suppress possible noise in space and highlight the correct semantic feature area with minimal additional computational burden. These feature vectors are determined only by the similarity between global features and local features within each group. However, each group in the attention mechanism SGE module consists of only a few adjacent feature maps, which reduces the information flow and information representation ability between groups. To solve this problem, the SSE module introduces the idea of ​​channel shuffling in Shufflenet, shuffling feature maps originally from different groups into new groups to strengthen the information flow between groups. However, this information exchange method can only make information flow between adjacent groups, and the input and output channels from distant groups are still unrelated. In addition, the SGE module uses global average pooling to approximate a set of semantic vectors for learning representation. Obviously, this operation can quickly collect global information in the feature map and is efficient in image classification tasks because they only need to focus on the most interesting areas in the feature map. However, this may cause problems in the field of object detection tasks, because they may have multiple regions of interest in one image. In this case, the global average pooling operation cannot separate all regions of interest well, resulting in the loss of information in some regions and causing missed detections.

[0079] To this end, in this example, a SCMR-SGE module is proposed, such as Figure 2The schematic diagram of the SCMR-SGE module implementation of this embodiment is shown. The FPN structure in YOLOv5 combines deep feature maps with shallow feature maps for feature fusion. The feature maps generated by the deep layer in CNN usually contain rich semantic information, but their resolution is low. The high-resolution feature maps generated by the shallow layer have less semantic information but rich spatial information. In order to better utilize the feature fusion function of FPN, the idea of ​​slicing splicing is introduced in this module. First, the feature maps generated by the relatively shallow layer in the convolutional neural network CNN (i.e., Shallow Feature Maps) and the feature maps generated by the deep layer (i.e., Deep Feature Maps) are spliced ​​(i.e., Sliced ​​Concatenate) through a slicing splicing operation. Then, SCMR-SGE divides these feature maps into G groups along the channel dimension. Afterwards, as described above, in order to approximate a more suitable semantic vector for the group of learning representations, SCMR-SGE adopts a multi-scale average pooling method to obtain global and local statistical features respectively. Finally, through spatial enhancement operations, more inter-channel related features are gathered, an attention factor is generated at each spatial position in the group, and more advanced semantic information is learned. Among them, the slice cascade function is implemented by a loop structure, which does not increase floating point operations (FLOPs). However, it is inaccurate to use FLOPs as the only measure of computational complexity. The original design of CASEnet led to a significant increase in network memory access costs and a decrease in computational parallelism. In the experiment of this embodiment, after adding the original slice operation, the time cost of the training phase and the delay of the detection phase were almost doubled compared with the original YOLOV5s. Therefore, this embodiment modifies the original operation to improve the operation efficiency. The modified slice splicing operation can be completed by the following steps: First, this embodiment generates an all-zero tensor with the same shape as the tensor to be sliced ​​and spliced, and the number of channels is set to C. After that, the shallow tensor is spliced ​​with the all-zero tensor, and then the all-zero tensor is spliced ​​with the deep tensor to form two new tensors to be operated with a channel number of 2C. Then the channel dimensions of the two tensors are adjusted to (C, 2) using the reshape operation, and transposed to (2, C), and finally reshaped to 2C channels, and the two tensors are added. The slicing and splicing operation ensures that each group contains feature maps generated by shallow and deep layers, which carry strong location information and semantic information respectively. Grouping features can significantly reduce the amount of computation. Figure 3 The following is a comparison diagram of the group characteristics of the present embodiment, the standard convolution and the present embodiment; Figure 3It can be seen that compared with standard convolution, each group learns different features with fewer parameters. Assuming that the size of the input feature map is C×H×W, the number of convolution kernels is N, and the channels are divided into G groups, the total number of parameters for the grouped features is N×(C / G)×H×W. The total number of parameters for standard convolution is N×C×H×W. That is, the total number of parameters is reduced to the original (1 / G). In particular, the remaining N×(C-(C / G))×H×W parameters can be regarded as 0, but the results are sometimes better.

[0080] In deep networks, it is difficult for CNN to obtain a good feature distribution. To solve this problem, the semantic features of key areas are enhanced by learning the overall information of each group. Among them, local pooling uses two pooling areas of different sizes to refine the receptive field. The process of spatial enhancement is as follows: global and local statistical features are obtained through global average function and local average function, and the semantic vector represented by the group learning is approximated.

[0081] Preferably, the spatial enhancement operation includes: obtaining global and local statistical features through a global average function and a local average function, and approximating the semantic vector represented by the group learning:

[0082]

[0083] Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the grouped image. In this embodiment, 3, 7 and global can be selected as the pool size respectively. In particular, when the pool size is set to global, the operation will be simplified to global average pooling. k is a rectangular area, which can be defined as 3, 7 and global. Taking area 3 as an example, this formula means to replace a pixel by taking the average of 9 pixels around a pixel point by 3 times 3. R C / G It is a common term in the field of image processing. R means picture, and C / G means that the channels are divided into G groups.

[0084] Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i :

[0085] c ij =g ij ·x ij

[0086] Perform spatial normalization on c:

[0087]

[0088]

[0089]

[0090] By performing a normalization operation on c in space, a large coefficient deviation between different samples can be prevented. Among them, m = H × W is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a pair of parameters γ and β are introduced for each coefficient to scale and shift the normalized result:

[0091]

[0092] Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features.

[0093]

[0094] Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

[0095] In this embodiment, all enhanced feature maps constitute the final output, which significantly eliminates the interference of background noise in the infrared image and highlights the area of ​​interest. This operation can greatly improve the performance of the initial YOLOv5s, but like the SE or CBAM module, the SCMR-SGE module can be embedded in the existing mainstream network structure, and the additional computational burden generated is almost negligible.

[0096] In order to improve the quality of representation, the existing technology proposes a squeeze-and-inspire (SE) module. By learning the interdependence between channels, this module is the first attention mechanism that considers the relationship between channels. The SE module adopts a "feature recalibration" strategy. The squeeze operation embeds global information, and the inspire operation implements adaptive recalibration. Through this process, the importance of each feature channel is automatically obtained, and then useful features are promoted according to the importance factor, and features that are useless for the current task are suppressed.

[0097] As mentioned above, pooling layers play an important role in deciding what information the network should pass. Figure 3is an example of different pooling layers. Suppose two photos are taken of the same scene, but there is a bonfire in one of the pictures. The maximum pooling will tend to express the characteristics of the bonfire, and the result of the average pooling will also be offset. If the target of this embodiment is not the bonfire, this will obviously interfere with the task of this embodiment. Characterizing the temperature distribution of an object is an important feature of infrared images and is also the difference between infrared images and visible light images, which means that similar objects have similar activation values ​​in infrared images. Therefore, this embodiment requires a smooth pooling method that allows similar pixels to share similar weights. Therefore, this embodiment utilizes the Log-Sum-Exp (LSE) pooling proposed in the paper "Pinheiro, P; Collobert, R. From image-level to pixel-level labeling with Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, JUN 7-12, 2015. [10.1109 / CVPR.2015.7298780]". The LSE pooling function is defined as:

[0098]

[0099] Where is the activation value of each pixel in the pooled region S, and S = s × s is the total number of pixels in S. r is a hyperparameter that controls the smoothness of the approximation. An infinite value means the effect is similar to max pooling, and a value of 0 means the effect is similar to average pooling. The advantage of this approach is that by controlling the value of r, pixels with similar activation values ​​will share similar weights during training. Due to overflow and underflow problems, the LSE function is further optimized as:

[0100]

[0101] x max {|x ij |,(i,j)∈S}

[0102] Among them, different pooling layers can collect other important clues about the characteristics of different objects to refine the attention at the channel level. Therefore, the representation ability of the network can be greatly improved by using different pooling layers at the same time, rather than using each pooling layer separately. Based on this conclusion, this embodiment further proposes a new joint attention (JA) module, such as Figure 4As shown in Figure 2, in this embodiment, three different pooling layers are used in the joint attention (JA) module. At the same time, the proposed method is mainly inspired by the above research. The overall structure of JA is as follows Figure 4 As shown. The idea of ​​this embodiment is simple: since different pooling methods extract different feature factors, it is best to directly compare the extracted feature factors. If any of the three methods gives a factor of high importance, then this channel will be enhanced, otherwise, this channel will be suppressed. In particular, this embodiment doubles the operation with the help of the upsampling function in the FPN network to make the result more reliable.

[0103] Preferably, there are three pooling layers; enhancing or suppressing the attention of different channels in different pooling layers includes:

[0104] A(F)=ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))}

[0105] In the formula, A(F) is the attention of the channel, the MLP weights of all inputs are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function. Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

[0106]

[0107] Preferably, the ES{} operation includes: if any parameter value in the parameters is greater than 0.5, taking the maximum value as the feature map weight; if any value of the three vectors at the same position is less than 0.5, taking the minimum value as the feature map weight.

[0108] Specifically, in this embodiment, the operation of the proposed JA module can be described as follows: First, three different pooling layers generate three different weights respectively. These three weights are then compared to selectively enhance or suppress them. Specifically, if any value of the three vectors at the same position is greater than 0.5, take the maximum value; conversely, if any value of the three vectors at the same position is less than 0.5, take the minimum value. The proposed channel attention calculation is A(F)=ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))}; where σ is the sigmoid function, all input MLP weights are shared, ES is the enhancement and suppression operation, σ is the sigmoid function, and then, this embodiment uses an upsampling function and repeats the ES operation to eliminate the offset caused by the linear interpolation calculation in the upsampling function as much as possible: The weights can be further calibrated after the upsampling stage, which makes the JA module more robust.

[0109] Further, regarding the arrangement of the modules of spatial attention and channel attention, in this example, since the YOLO method is very mature and uses the FPN network, this embodiment selects the latest work of the series, YOLOv5, as the basic method. Figure 5 As shown, it is a schematic diagram of the structure of YOLOv5s. YOLOv5 adopts a modular design concept and has four versions: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The only difference between these four versions is the number of stacked CSP modules. Stacking more CSP modules can obtain better feature extraction capabilities, which is conducive to the comparative experiment of this embodiment. By embedding the modules of this embodiment in the same position of the FPN network in different versions of YOLOv5, this embodiment can judge whether the method of this embodiment is universal based on the experimental results. Among them, YOLOv5m, YOLOv5l, and YOLOv5x stack 2, 3, and 4 groups of CSP modules respectively at the CSP module position of YOLOv5s. Because this embodiment wants to make this method as efficient as possible, although the modules proposed above are lightweight enough, this embodiment still hopes to use these models as little as possible. Therefore, this embodiment decides to focus on the neck network for feature fusion and refine the features extracted by the backbone network. As mentioned above, the two modules SCMR-SGE and JA focus on "where" and "what" need attention, respectively, and should be embedded in different layers of the neck network. In particular, the SCMR-SGE module requires a feature map containing relatively more semantic information. The Concatenate module marked with a blue rectangle (i.e., Position 1) receives the feature map generated by the last CSP module in the backbone network and the final output of the backbone network, and the SCMR-SGE module is suitable here. On the other hand, replacing the upsampling function with JA at the red rectangle mark (i.e., Position 2) can re-screen the feature map processed by the SCMR-SGE module without interfering with the coarse features extracted in the shallow layer. Thanks to the fact that YOLOv5s uses FPN and PANet for the neck network at the same time, the feature map processed in this way can carry stronger semantic features and flow to each detection head. The arrangement of this embodiment focuses on the feature fusion stage rather than the feature extraction stage, which is different from the usual practice of embedding the attention mechanism into the residual structure, but still benefits from the improvement of feature extraction capabilities.

[0110] In addition, in this embodiment, a series of comparative experiments were conducted to verify the advancedness and robustness of the proposed method. This section will also introduce the data set, experimental settings and evaluation indicators of this embodiment. In this example, the data set used is the FLIR-ADAS data set, which is collected from a car camera on the streets and roads in Santa Barbara, California. The data set was taken during the day (60%) and night (40%), with weather ranging from sunny to cloudy, and contains 8862 images for training and 1366 images for verification. The objects marked in the data set include cars, people, bicycles, and dogs. It should be pointed out that the data set lacks dog labels, which may result in poor generalization ability of the trained model and overfitting problems. Therefore, the experiments in this embodiment ignore the dog labels. In particular, even if the dog labels are ignored, the number of people, cars, and bicycles in the sample is still unbalanced. The bicycle data enhancement strategy used in some studies is not used in the method of this embodiment. Some images in the FLIR data set are as follows: Figure 6 shown.

[0111] All experiments were conducted on a PC with Intel(R) Core(R) i7-8700K CPU, NVIDIA RTX 3060 (12GB LHR) GPU, CUDA 11.1, CUDNN 9.1, and Win10 operating system. The deep learning framework is Pytorch 1.9.0. The optimizer uses stochastic gradient descent (SGD), and the learning rate decay strategy is cosine. The initial learning rate is 0.01, the final decay rate is 0.001, the batch size is 48, the preheating is 3 epochs, and the training epoch is optimized to 100. The input image resolution is 640×512.

[0112] In order to evaluate the performance of the proposed method, mAP50, mAP0.5:0.95 and detection delay are used as evaluation indicators. Average precision (AP) is the area enclosed by the precision-recall curve and the coordinate axis. Average precision (mAP) is a key indicator in the target detection algorithm. It is the average value of AP of multiple categories in target detection. The experiment in this embodiment has three categories, and mAP50 can be considered as the average value of AP values ​​of these categories. Generally speaking, the stronger the detection capability, the higher the mAP value. The precision and recall are as follows: Among them, TP (True Positive) is a positive sample that is detected correctly, and FP (False Positive) is a positive sample that is detected incorrectly. On the contrary, FP and FN are negative samples that are detected correctly and incorrectly, respectively. Whether a label is a positive sample or a negative sample is determined by the intersection over union (IoU). The definition of the intersection over union is as follows: Where P and T are the predicted bounding box and ground truth. In most cases, the IoU threshold is set to 0.5, that is, samples with IoU greater than 0.5 are considered positive samples. Then, AP is defined as: Where r is the recall rate, p is the precision rate, and p(r) is a function with r as a parameter. In most cases, mAP50 is the most important metric.

[0113] Furthermore, in this embodiment, an ablation experiment is performed to verify the proposed algorithm. In this example, the effectiveness of the proposed spatial enhancement module is verified. When comparing these methods, all parameters are set to the same. After a series of experiments, in order to obtain the best performance, this embodiment chooses to divide the feature map into 64 groups in the SGE module. The delay time of this method is calculated by appending all the video frames in the data set. This embodiment selects SE and CBAM modules to compare with the modules of this embodiment because these two modules are widely used in recent target detection methods. The original SGE module is used for control research. All modules here are respectively embedded in the following Figure 5 The corresponding connection structure of YOLOv5s shown in the Concatenate module (i.e., Position 1). The performance of each module is shown in the following table.

[0114]

[0115] As can be seen from the table, compared with SE and CBAM Module, the basic SGE module has similar performance but lower latency. Thanks to the feature grouping idea of ​​the SGE module, the improved module can still maintain a relatively low amount of computation. Compared with the original SGE module, the multi-receptive field SGE module increases the mAP50 by 0.5% at the cost of an additional 0.7ms latency. More importantly, when the SCMR-SGE module is adopted, the redesigned slice connection operation only increases the latency by 0.1ms while increasing the mAP50 by 0.2%. The above results show that the SCMR-SGE module can improve the detection performance at a small computational cost, which is crucial in real-time object detection tasks.

[0116] This example further verifies the effectiveness of the proposed JA module. The YOLOv5s method embedded with SE module and CBAM module is used as a comparative experiment. All modules here are added to Figure 5 In the upsampling structure marked by the red rectangle (i.e., Position 2), the performance of each module is shown in the following table.

[0117]

[0118] SE modules with three different pooling layers are embedded into the initial method. These modules improve the performance of the method. In particular, the SE module using a combination of average pooling and max pooling shows better performance. It is worth noting that the performance of the CBAM module is quite poor compared to the SE module. As the neural network goes deeper, channels may become more and more important, which makes it inappropriate to use spatial attention modules to filter deep feature maps. More importantly, when the SE module adopts more than two different pooling layers, the method of directly adding pooling results is no longer effective. Compared with the above modules, the proposed JA module increases mAP50 to 81.4%, which is higher than any other method. The results further confirm that different pooling layers can collect different salient features to refine the attention allocation on channels. At the same time, the detection speed loss of this method is acceptable.

[0119] This embodiment also uses different methods to conduct comparative experiments to verify the performance of the method of this embodiment. The performance of each method is shown in the following table.

[0120]

[0121] Compared with the original YOLOv5s, the method proposed in this embodiment has comprehensively improved the detection performance with little performance loss. The AP of these three targets has increased significantly. After adding two modules, the mAP50 reached 81.5%, an increase of 2.1% over before. It is worth mentioning that for the bicycle class, which is sample-scarce and whose features are not prominent in clutter, the AP increased by 3% to 68.3%. On the other hand, the parameters of the improved method only increased by 0.01M, resulting in a performance loss of 15 frames per second. This is acceptable, especially compared with YOLOv5m, the widened and deepened version of yolov5. The method of this embodiment is not only more accurate than the original YOLOv5m method, but also has great advantages in detection speed. This embodiment selects several representative detection results, such as Figure 7 As shown, the output results of the method of this embodiment are intuitively compared with the original YOLOv5s. The two methods set the same confidence thresholds for bounding boxes and IoU. Figure 7 In the figure, the FLIR dataset is detected by YOLOv5s and this method respectively; the first column is the ground truth, the second column is the result of YOLOv5s, and the third column is the result of the method of this embodiment; the area marked with blue circles indicates that YOLOv5s has missed the detection of objects here. The targets missed by the original YOLOv5s method are usually in clutter, such as bicycles and pedestrians with interfering objects such as trees in the background. These objects are partially obscured or their own thermal characteristics are not obvious, which puts higher requirements on the detection ability of this method. Figure 7 It can be seen that the method of this embodiment can effectively distinguish the target from the complex background and reduce missed detection. The method of this embodiment successfully detects the target that is more obscured, which intuitively demonstrates the superior performance of the method of this embodiment.

[0122] Therefore, in view of the low contrast, lack of fine-grained and texture information of infrared images, this embodiment proposes two new attention mechanisms to improve the feature extraction ability of the infrared target detection method. The SCMR-SGE module can simultaneously enhance high-order semantic information and suppress possible background noise. At the same time, the JA module can selectively enhance or suppress channel information to improve detection performance. In addition, this method is different from the previous method of embedding attention mechanisms in each module of the backbone network. It only uses SCMR-SGE and JA modules at two key positions of the FPN network to filter spatial information and channel information respectively. The experimental results on the FLIR-ADAS dataset show that this method can effectively improve the detection performance of infrared targets at a low cost. Under the condition of only based on YOLOv5m, the mAP value of the FLIR dataset reaches 82.7%, and the method proposed by us can maintain a detection speed of about 60fps, which is higher than SSD, Fast-RCNN, Retina-net and its improved algorithm.

[0123] Embodiment 2

[0124] In addition, if Figure 8 As shown, in the second embodiment, a target detection device based on space attention and channel attention is also proposed, including:

[0125] The acquisition module 10 is used to acquire and obtain a detection image, wherein the detection image is an infrared image;

[0126] A detection module 20 is configured to generate a convolutional neural network CNN according to the target feature map training, and to perform target detection on the infrared image according to the convolutional neural network CNN;

[0127] In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed;

[0128] The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map.

[0129] Preferably, the spatial enhancement operation includes: obtaining global and local statistical features through a global average function and a local average function, and approximating the semantic vector represented by the group learning:

[0130]

[0131] Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the feature image after grouping;

[0132] Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i :

[0133] c ij =g ij ·x ij

[0134] Perform spatial normalization on c:

[0135]

[0136]

[0137]

[0138] Among them, m = H × W, is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a pair of parameters γ and β are introduced for each coefficient to scale and shift the normalized result:

[0139]

[0140] Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features.

[0141]

[0142] Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

[0143] Preferably, there are three pooling layers; enhancing or suppressing the attention of different channels in different pooling layers includes:

[0144] A(F)=ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))}

[0145] Where A(F) is the attention of the channel, the MLP weights of all inputs are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function;

[0146] Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

[0147]

[0148] In addition, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the target detection method based on spatial attention and channel attention as described above.

[0149] In addition, an embodiment of the present invention provides a computer storage medium, which stores a program; the program is loaded and executed by a processor to implement the target detection method based on spatial attention and channel attention as described above.

[0150] In the scheme of the above embodiment of the present invention, the target detection method based on spatial attention and channel attention is provided, by collecting and acquiring the detection image, the detection image is an infrared image; generating a convolutional neural network CNN according to the target feature map training, and performing target detection on the infrared image according to the convolutional neural network CNN; wherein, in the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed; the multiple pooling layers in the convolutional neural network CNN generate their corresponding weights; the attention of different channels where different pooling layers are located is enhanced or suppressed to generate the target feature map. The present invention effectively highlights multiple targets of interest under different high-order semantics and suppresses background noise through the spatial attention mechanism; in terms of the channel attention mechanism, three different pooling layers are used to selectively enhance or suppress specific channels, so that the allocation of attention resources is more reasonable; and it is embedded in the FPN structure instead of the backbone network to achieve better performance.

[0151] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. Moreover, these modules can all be implemented in the form of software calling through processing elements. They can also all be implemented in the form of hardware. Some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware.

[0152] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and offsets to this specification. Such modifications, improvements and offsets are suggested in this specification, so such modifications, improvements and offsets still belong to the spirit and scope of the exemplary embodiments of this specification.

[0153] In addition, it will be understood by those skilled in the art that various aspects of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of this specification may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0154] It should be noted that if there is any inconsistency or conflict between the descriptions, definitions, and / or usage of terms in the supplementary materials of this specification and the contents described in this specification, the descriptions, definitions, and / or usage of terms in this specification shall prevail.

[0155] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A target detection method based on spatial attention and channel attention, characterized in that: include: Collecting and acquiring a detection image, wherein the detection image is an infrared image; Generating a convolutional neural network (CNN) according to the target feature map training, and performing target detection on the infrared image according to the convolutional neural network (CNN); In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed; The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map; The spatial enhancement operation includes: obtaining global and local statistical features through global average function and local average function, and the semantic vector represented by feature learning is approximated as: Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the feature image after grouping; Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i : c i j=g ij ·x i j Perform spatial normalization on c: Among them, m = H × W, is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a scaling parameter γ and a shift parameter β are introduced for each coefficient to scale and shift the normalized result: Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features. Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

2. The object detection method based on spatial attention and channel attention according to claim 1 is characterized in that: There are three pooling layers; the attention of different channels in different pooling layers is enhanced or suppressed, including: A(F) = ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))} where A(F) is the attention of the channel, all input MLP weights are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function; Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

3. The object detection method based on spatial attention and channel attention according to claim 2 is characterized in that: The ES{} operation includes: if any parameter value in the parameters is greater than 0.5, the maximum value is taken as the feature map weight; if any value of the three vectors at the same position is less than 0.5, the minimum value is taken as the feature map weight.

4. The object detection method based on spatial attention and channel attention according to claim 1 is characterized in that: In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slicing splicing operation, including: Generate an all-zero tensor with the same shape as the tensor to be sliced ​​and concatenated, with the number of channels set to C; Concatenate the shallow tensor with the all-zero tensor, and then concatenate the all-zero tensor with the deep tensor to form two new tensors with 2C channels to be operated. Use the reshape operation to adjust the channel dimensions of these two tensors to (C, 2) and transpose them to (2, C); Transform it to 2C channels according to the reshape function and add the two tensors together.

5. A target detection device based on spatial attention and channel attention, characterized in that: include: An acquisition module is used to acquire and obtain a detection image, wherein the detection image is an infrared image; A detection module, which generates a convolutional neural network (CNN) according to the target feature map training, and performs target detection on the infrared image according to the convolutional neural network (CNN); In the convolutional neural network CNN, the feature map generated in the relatively shallow layer is spliced ​​with the feature map generated in the deep layer through a slice splicing operation; the feature map is divided into G groups along the channel dimension; a multi-scale average pooling method is used to obtain global and local statistical features respectively, and a spatial enhancement operation is performed; The multiple pooling layers in the convolutional neural network CNN generate corresponding weights; the attention of different channels in different pooling layers is enhanced or suppressed to generate the target feature map; The spatial enhancement operation includes: obtaining global and local statistical features through global average function and local average function, and the semantic vector represented by feature learning is approximated as: Where k is the pooling area size, s = (k-1) / 2, x ij ∈R C / G ; x is the value of the pixel, ij is the pixel coordinate; C represents the number of channels, G is the number of groups into which the feature map is divided along the channel dimension; R C / G is the feature image after grouping; Then, a dot product operation is performed on each feature to measure the similarity c between the semantic feature g and the local feature x i : c i j=g ij ·x i j Perform spatial normalization on c: Among them, m = H × W, is a constant added for numerical stability, H is the feature image height, and W is the feature image width; ∈ In order to ensure that normalization is inserted in the network to represent the identity transformation, a pair of parameters γ and β are introduced for each coefficient to scale and shift the normalized result: Finally, the important coefficients generated by the sigmoid function are used to spatially scale the original features to obtain the enhanced features. Wherein, σ is the sigmoid function; all enhanced feature maps constitute the output of the spatial enhancement operation.

6. The target detection device based on spatial attention and channel attention according to claim 5 is characterized in that: There are three pooling layers; the attention of different channels in different pooling layers is enhanced or suppressed, including: A(F) = ES{σ(MLP(avgpool(F)),σ(MLP(maxpool(F)),σ(MLP(LSEpool))} where A(F) is the attention of the channel, all input MLP weights are shared, ES is the enhancement and suppression operation, and σ is the sigmoid function; Then, the upsampling function upsample() is used to repeat the ES operation to obtain the corrected channel attention:

7. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the target detection method based on spatial attention and channel attention as described in any one of claims 1-4.

8. A computer storage medium storing a program; the program is loaded and executed by a processor to implement the target detection method based on spatial attention and channel attention as described in any one of claims 1-4 above.

Images