A small target detection method based on secondary feature fusion mechanism of attention

Abstract

The present invention discloses a small target detection method based on a secondary feature fusion mechanism of attention, which utilizes a secondary feature fusion module based on a double-layer pyramid to fuse the feature information extracted by a backbone network from a to-be-processed image; wherein the backbone network of the secondary feature fusion module adopts a residual network, the first feature fusion from top to bottom is to continuously convolve and downsample the feature map extracted by the backbone network, the second feature fusion from bottom to top is to continuously convolve and upsample the feature map obtained for the first time, and then introduce an attention mechanism to add and fuse the feature maps of the same size respectively, retaining the feature information of small targets in the shallow feature map; then, prediction is performed based on the feature maps of different sizes of receptive fields formed by the addition fusion, and the prediction results are summarized. The present invention effectively solves the effective detection of vehicle targets in a haze environment.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small target detection method based on an attention-based secondary feature fusion mechanism. Background Art

[0002] With the improvement of embedded device performance and the development of edge computing technology, it is becoming increasingly popular to configure deep models on edge computing devices, which makes it possible for drones to carry embedded devices to perform high-altitude target detection tasks. As a new thing in recent years, drones have been introduced by more and more industries to perform related tasks because of their own advantages such as flexibility, lightness, no terrain control, and convenience in obtaining near-ground remote sensing data.

[0003] Among them, the effect of deep intelligent models is the key. With the development of convolutional neural networks (CNN), the performance of target detection algorithms has been greatly improved. Researchers have applied these convolutional neural networks to target detection, which has greatly improved the speed and accuracy compared with traditional target detection algorithms. The target detection algorithms based on deep learning are mainly divided into two categories: (1) Two-Stage target detection framework based on classification, represented by the R-CNN series. The classification-based Two-Stage target detection algorithm first uses algorithms such as Selective Search and Edge Boxes to generate candidate regions (Region proposals) that may contain the target to be detected, and then further classifies these candidate regions and calibrates the position to obtain the final detection results; (2) One-Stage target detection framework based on regression, represented by YOLO and SSD algorithms. The target detection method based on regression does not need to generate candidate regions. It obtains the target detection result by directly anchoring the feature map and finally maps it back to the original image. Compared with the Two-Stage algorithm, the speed is greatly improved. Although the two-stage target detection algorithm has high accuracy, it is very slow; although the one-stage target detection algorithm is fast, it has low accuracy. For the problem of object detection in complex backgrounds and ultimately deploying it on resource-constrained hardware platforms, the accuracy and speed of the model are great challenges.

[0004] Although the effect of deep models is getting better and better, the model effect depends on the data set. Most of the existing models are trained on public data sets, and these public data sets are basically shot in ideal environments, which makes most of the existing deep models unable to be applied to real complex environments. At the same time, the images taken by drones are generally in high-altitude scenes, which makes the pixels of the captured targets too small, and the existing mainstream detection models have poor detection effects on small targets.

[0005] It is becoming more and more popular to use drones equipped with intelligent algorithms to realize unmanned intelligent systems. For example, in the military field, military drones are equipped with intelligent detection algorithms to realize military target detection, etc. In the industrial field, industrial drones are equipped with intelligent detection algorithms to realize forest fire prevention and wildlife protection, etc. At present, target detection based on deep learning models is mostly conducted in ideal environments without considering actual complex environments, such as haze environments. At the same time, the target pixels photographed by drones at high altitudes are often too small, and the current mainstream detection models have poor detection effects on small targets. Therefore, how to effectively detect vehicle targets in haze environments; how to effectively detect small targets photographed by drones are technical problems that must be solved. Summary of the invention

[0006] The purpose of the present invention is to address the technical defects existing in the prior art and to provide a small target detection method based on a secondary feature fusion mechanism of attention. By designing a secondary feature fusion network based on the attention mechanism, real-time detection of embedded devices carried by drones in haze environments can be achieved, effectively improving the detection effect.

[0007] The technical solution adopted to achieve the purpose of the present invention is:

[0008] A small target detection method based on a secondary feature fusion mechanism of attention utilizes a secondary feature fusion module based on a double-layer pyramid to fuse feature information extracted by a backbone network from a to-be-processed image; wherein the backbone network of the secondary feature fusion module adopts a residual network, the first feature fusion from top to bottom is to continuously convolve and downsample the feature map extracted by the backbone network, the second feature fusion from bottom to top is to continuously convolve and upsample the feature map obtained for the first time, and then introduce an attention mechanism to add and fuse feature maps of the same size respectively, retaining the feature information of small targets in the shallow feature map; then prediction is performed based on the feature maps of receptive fields of different sizes formed by the addition and fusion, and the prediction results are summarized.

[0009] Preferably, the residual network obtains feature maps with downsampling multiples of 2 times, 4 times, 8 times, 16 times, and 32 times during the process of continuous convolution and downsampling.

[0010] Preferably, during the feature map fusion process, the residual network adds an attention mechanism between feature maps of the same size to focus on target features, so that the model increases the feature weight of the target area affected by haze, and focuses the model's attention on the area obscured by haze, so as to improve the feature extraction effect of the target.

[0011] Preferably, the attention mechanism adopts a hybrid attention mechanism that combines spatial attention and channel attention.

[0012] Preferably, after the input multi-channel feature map is processed by convolution, a channel attention mechanism is first introduced to assign different weights to each channel for processing, and the processing result is output. Then, based on the processing result, spatial attention is introduced to perform mean operation on multiple channels, learn the overall distribution of all channels, discard singular channels, process the image features in each channel equally, and finally output the result map.

[0013] Preferably, the pictures to be processed include pictures collected by an aircraft in a haze environment and / or a high-altitude environment, in which the target pixels are lower than a predetermined threshold.

[0014] This paper proposes an attention mechanism based on secondary feature fusion, which effectively solves the problem of effective detection of vehicle targets in haze environments. By embedding the attention mechanism into the second bottom-up feature fusion process, the feature information extracted from vehicle targets in different receptive fields is fully utilized, and the model attention is focused on effective target features, thus improving the target detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram showing the heat map of targets of different sizes on feature maps of different receptive fields.

[0016] Figure 2 It is a schematic diagram of the double-layer pyramid feature fusion method of the present invention.

[0017] Figure 3 This is the structural diagram of the secondary feature fusion module based on the attention mechanism.

[0018] Figure 4 Schematic diagram of the channel feature map of 64 channels.

[0019] Figure 5 Schematic diagram of the hybrid attention mechanism of the present invention.

[0020] Figure 6 Flowchart of the small target detection method based on the attention-based secondary feature fusion mechanism. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] Object detection algorithms based on deep models generally include a backbone network (backbone), a feature fusion network (neck), and a prediction network (head). The backbone network is used to extract feature information from the image for use by the subsequent network; the feature fusion network is placed between the backbone network and the prediction network to better utilize the features extracted by the backbone network; the prediction network is a network that obtains the network output content and uses the previously extracted features to predict the results.

[0023] In order to solve the problems in the background technology, the embodiment of the present invention designs bottom-up and top-down secondary feature pyramid fusion modules to solve the problem of detecting small targets and adds an attention mechanism in the secondary fusion process to solve the problem of detecting targets in haze environments.

[0024] In deep models based on convolutional neural networks, target features are closely related to the receptive field of feature maps. Feature maps at different stages correspond to receptive fields of different sizes, and accordingly, the abstract degree of the information they express is also different. Specifically, shallow feature maps have smaller receptive fields and are therefore more suitable for detecting small objects, while deep feature maps have larger receptive fields and are more suitable for detecting large objects.

[0025] As the convolution process continues to deepen, the receptive field of the feature map becomes larger and larger. As a result, the features of small targets gradually disappear as the convolution deepens. This can be seen by printing the heat maps at different stages of the convolution process, such as Figure 1 As shown in the figure, large targets such as zebras are more easily captured by high-level features, because large objects require larger receptive fields and high-level semantic features. For the detection of the flock of sheep below, it can be seen that fine-grained features in the underlying features are more needed for identification. Therefore, in order to improve the detection effect of small targets, it is necessary to fuse shallow features with deep features.

[0026] In order to make full use of the features of the targets on the feature maps with different receptive fields, the present invention designs a double-layer pyramid for secondary feature fusion module, such as Figure 2 As shown in the figure, the backbone network uses a residual network, and in the process of continuous convolution and downsampling, feature maps with downsampling multiples of 2, 4, 8, 16, and 32 are obtained. The first feature fusion from top to bottom is to continuously convolve and downsample the feature map extracted by the backbone network, and the second feature fusion from bottom to top is to continuously convolve and upsample the feature map obtained in the first time, and then add and fuse the feature maps of the same size respectively, and finally make predictions on the feature maps of different sizes of receptive fields, and finally summarize the prediction results. Through the above multiple fusions, the feature information of small targets in the shallow feature map is retained, and the detection effect of small targets is improved.

[0027] In order to achieve effective extraction of target features in a haze environment, the embodiment of the present invention designs an attention-based secondary fusion module, whose structure is as follows: Figure 3 As shown in the figure, an attention mechanism is added between feature maps of the same size to focus on target features to improve the feature extraction effect of the target.

[0028] The attention mechanism is very similar to human visual attention. Human attention is a brain signal processing mechanism unique to human vision. Humans quickly scan the global image to obtain the target area that needs to be focused on, get the focus of attention, and then pay more attention to this area to obtain more detailed information about the target that needs to be paid attention to, thereby suppressing other useless information. This is a means for humans to use limited attention resources to quickly filter out high-value information from a large amount of information. It is a survival mechanism formed by humans in long-term evolution, which greatly improves the efficiency and accuracy of visual information processing.

[0029] For example, if a person is given a newspaper with pictures printed on it, he or she will first look at the headline of the newspaper and then look at the prominent pictures. The attention mechanism in deep learning is essentially similar to the selective visual attention mechanism of humans, and its purpose is also to select the more critical information for the current task goal from a large amount of information.

[0030] The attention mechanism is often considered a resource allocation mechanism. The original network model evenly distributes all resources to the input data, but research has confirmed that there are correlations between data. For example, the more important parts of the data can be found, and resources can be reallocated according to the importance to highlight certain important features. Therefore, in a haze environment, the present invention needs to use the attention mechanism to increase the feature weight of the target area affected by haze. In layman's terms, it is to focus the model's attention on the area blocked by haze.

[0031] The attention module is divided into channel attention and spatial attention. The role of the channel attention mechanism is to assign different weights to each channel, so that the network can focus on important features and suppress unimportant features. The spatial attention mechanism is to weight the information of more important channels to less important channels, thereby improving the importance of each channel of the feature map as a whole.

[0032] Next, through a 64-channel feature Figure 4 To explain the difference between the two.

[0033] Channel attention: It is to assign different weights to each channel. For example, the shape of the horse is more obvious at 1 and 2, so naturally, the weight of channels 1 and 2 is larger, and the weight of channels 3 and 4 is smaller.

[0034] Spatial attention: Spatial attention is a mean operation on 64 channels to obtain a weight (w, x, h). The mean operation learns the overall distribution of all channels and discards the singular channels. For example, images 1 and 2 can well depict the shape of a horse, while images 3 and 4 cannot (but in essence, they also want to show the shape of a horse). However, after the mean operation, the weights of w, x, and h are shared, and a certain weight description is given to images 3 and 4, which is equivalent to giving a certain amount of attention to images 3 and 4, so that they can also depict the shape of a horse.

[0035] After understanding the design ideas of these two types of attention, a simple comparison shows that the two attention mechanisms have their own advantages and disadvantages. First, spatial attention ignores the information in the channel domain and treats the image features in each channel equally. This approach limits the spatial domain transformation method to the original image feature extraction stage, and its interpretability when applied to other layers of the neural network layer is not strong. Channel attention directly globally averages the information in a channel and ignores the local information in each channel. This approach is actually a relatively violent behavior. Therefore, the single attention mechanisms are integrated to design a hybrid attention mechanism model that takes advantage of each other's strengths and weaknesses. The attention mechanism used in the model of the present invention is a hybrid attention mechanism that combines spatial attention and channel attention. The specific method is as follows: Figure 5 As shown:

[0036] The above hybrid attention mechanism is finally embedded in the secondary feature fusion process, and the result prediction is performed on feature maps of different sizes, and finally the prediction results are summarized.

[0037] The network model of the present invention was experimentally compared with an open source detection model on the COCO dataset, and the detection effect on small objects was improved by 4.9% and the overall detection effect was improved by 2.1%.

[0038] The experimental comparison results are as follows:

[0039] AP AP50 AP75 APs APm APl ARs AR AAR DeNet 33.8 53.4 36.1 12.3 36.1 50.8 19.2 46.9 64.3 CoupleNet 34.4 54.8 37.2 13.4 38.1 50.8 20.7 53.1 68.5 Cascade-RCNN 42.8 62.1 46.3 23.7 45.5 55.2 - - - Mask-RCNN 39.8 62.3 43.3 22.1 43.2. 51.2 - - - DSSD 33.2 53.3 35.2 13 35.4 51.1 21.8 49.1 66.4 CornerNet 42.1 57.8 45.3 20.8 44.8 56.7 38.5 62.7 77.4 SSD 31.2 50.4 33.3 10.2 34.5 49.8 21.8 49.1 66.4 yolov4 43.8 60.7 46.9 25.3 48.6 56.7 45.8 69.1 79 yolov5 42.5 61.2 45.2 20.6 47.4 62.6 32.8 64.4 78.5 Ours 44.6 63.7 47.5 25.5 49.7 61.7 42.2 65.8 77

[0040] Table 1

[0041] At the same time, the present invention also conducted an ablation experiment on the network model designed by itself, and tested the effect on the vehicle dataset. A comparative experiment was conducted by adjusting the detection branch and the attention module, and the results are shown in Table 2:

[0042] P R AP0.5 AP Normal 71.7 75.6 70.8 48.1 Add-Detection 68.3 80.5 74.9 51.8 Add-Attention 72.3 76.4 72 48.8 Add-Detection-Attention 73.8 81.2 75.9 50.8

[0043] Table 2

[0044] The lightweight network model of the present invention makes it possible to deploy deep models on resource-constrained hardware devices and perform real-time detection. At the same time, a secondary feature fusion module based on the attention mechanism is designed to effectively extract weak target feature information in a haze environment, making it possible for drones to effectively detect target vehicles in a haze environment.

[0045] The experimental results show that the present invention can achieve real-time detection (30FPS) on the NVIDIA AGX XAVIER hardware device, improving the detection effect in haze environments.

[0046] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A small target detection method based on the secondary feature fusion mechanism of attention, characterized in that: The method uses a secondary feature fusion module based on a double-layer pyramid to fuse the feature information extracted by the backbone network from the image to be processed; wherein the backbone network of the secondary feature fusion module adopts a residual network, the first feature fusion from top to bottom is to continuously convolve and downsample the feature map extracted by the backbone network, the second feature fusion from bottom to top is to continuously convolve and upsample the feature map obtained for the first time, and then introduce an attention mechanism to add and fuse the feature maps of the same size respectively, retaining the feature information of small targets in the shallow feature map; then, prediction is performed based on the feature maps of receptive fields of different sizes formed by the addition and fusion, and the prediction results are summarized; The images to be processed include images collected by an aircraft in a haze environment and / or a high-altitude environment, where the target pixels are lower than a predetermined threshold; In the process of feature map fusion processing, the residual network adds an attention mechanism between feature maps of the same size to focus on target features, so that the model increases the feature weight of the target area affected by haze, and focuses the model's attention on the area blocked by haze, so as to improve the feature extraction effect of the target; The attention mechanism adopts a hybrid attention mechanism that combines spatial attention and channel attention; After the input multi-channel feature map is processed by convolution, the channel attention mechanism is first introduced to assign different weights to each channel for processing and output the processing results. Then, based on the processing results, spatial attention is introduced to perform mean operations on multiple channels to learn the overall distribution of all channels, discard singular channels, process the image features in each channel equally, and finally output the result map.

2. The small target detection method based on the attention-based secondary feature fusion mechanism according to claim 1 is characterized in that: The residual network obtains feature maps with downsampling multiples of 2 times, 4 times, 8 times, 16 times, and 32 times during the process of continuous convolution and downsampling.

Images