Attention mechanism-based aerial image target detection method and system and related device

Abstract

The invention provides an aerial image target detection method and system based on an attention mechanism and a related device, and the method comprises the steps: inputting an aerial image obtained by an unmanned plane into an aerial image target detection network model, and obtaining a target detection result of the aerial image; wherein a backbone network in the aerial image target detection network model is a YOLOv7 backbone network containing a background suppression module BSAM containing an attention mechanism, a neck structure is a bidirectional weighted fusion pyramid module TWFP, and a detection head structure is a combination of a generalization efficient layer aggregation network GELAN and a visual Swin Transformer model.According to the detection method, the detection process is simplified, the detection speed is increased, and the detection efficiency is improved. And meanwhile, relatively high detection precision is kept.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an aerial image target detection method and system based on an attention mechanism and related devices. Background Art

[0002] With the development of computer vision technology, target detection has become an important research direction in the field of computer vision. The main task of the target detection algorithm is to automatically identify the targets in an image and mark their locations and categories. In practical applications, target detection algorithms have been widely used in smart agriculture, traffic detection, disaster response and other fields. At the same time, applying some popular target detection algorithms to drones has gradually become a hot research field.

[0003] In the current field of general target detection research, natural image target detection technology cannot be directly applied to images captured by drones because aerial images generally have different challenges. In images captured by drones, due to the unique characteristics of small targets, target detection is still difficult. The difficulties are (1) small target size: small targets are usually small, which makes it difficult to accurately detect and describe the features of the target; (2) background interference: small targets usually appear in complex backgrounds, and background noise and interference will have a negative impact on target detection; (3) large target scale variation: small targets usually appear at different perspectives, and changes in perspective will cause changes in the appearance of the target, thereby increasing the difficulty of target detection; (4) target occlusion: small targets are small in size and have relatively few pixels, which may cause the target boundary to be blurred or not prominent enough in the image, making small targets susceptible to occlusion. These factors are the reason why the detection method has low detection accuracy for aerial images.

[0004] In the past few decades, the development of target detection algorithms can be roughly divided into the traditional target detection algorithm stage and the deep learning-based target detection algorithm stage.

[0005] The traditional target detection framework mainly includes three steps: target positioning, feature extraction and feature classification. The first step is to use sliding windows of different sizes to frame a part of the image as a candidate area; the second step is to extract visual features related to the candidate area. For example, the HOG features commonly used in pedestrian detection and general target detection; the third step is to use a classifier for recognition, such as the commonly used SVM model. Its essence is a sliding window + traditional machine learning classifier, using windows of different sizes and proportions to slide on the entire image at a certain step length, and then classify the areas corresponding to these windows, thereby realizing the detection of the entire image.

[0006] However, there are many problems with traditional target detection algorithms. The first is the slow detection speed. The sliding window area selection strategy of the traditional target detection algorithm is not targeted. It is necessary to perform sliding window detection on each position of the image. There are also problems of high time complexity and window redundancy. This method has a large amount of calculation, resulting in slow detection speed. Secondly, due to factors such as the diversity of target morphology, diversity of lighting changes, and diversity of backgrounds, it is not so easy to extract the features of the target. However, the quality of the extracted features directly affects the accuracy of classification. In addition, the traditional target detection algorithm has weak processing capabilities for occlusion and complex backgrounds. When the target is obscured by other objects or backgrounds, traditional algorithms are prone to missed detection.

[0007] Compared with traditional target detection methods, target detection methods based on deep learning have become the dominant method in this field, and target detection algorithms based on convolutional neural network (CNN) architecture have become mainstream.

[0008] In CNN-based algorithms, there are two types of network architectures: one-stage and two-stage. The first type is a two-stage detection method based on region proposals. An early example is R-CNN, which extracts region proposals and uses convolutional neural networks (such as AlexNet and VGG) for object classification and bounding box regression. Fast R-CNN has significantly improved processing speed and performance over traditional R-CNN through feature sharing and the introduction of RoIPooling. Faster R-CNN introduced the RegionProposal Network to generate candidate regions, and then used these candidate regions for target classification and localization through RoI pooling. It strikes a good balance between accuracy and speed.

[0009] However, the two-stage object detection network architecture also has many problems. The first point is that the detection speed is slow. The two-stage method requires two independent steps, so it is usually slower than the one-stage method. The second point is that the model is relatively complex, and it is difficult to train and deploy. The third point is that it is not suitable for real-time applications: due to the complexity of the model, the amount of calculation required is large, and it is not suitable for applications that require real-time performance.

[0010] The single-stage method is another type of method that uses the network to directly regress the size, position, and class of the object from the input image. The most widely recognized single-stage detection method is the YOLO series framework. Its core idea is to regard the target detection task as a regression problem, and to achieve efficient target detection by simultaneously predicting the bounding box and category of the target in a single neural network. Compared with traditional target detection methods, the YOLO series framework has faster detection speed and higher real-time performance, and is suitable for scenarios that require rapid target detection, such as autonomous driving, video surveillance and other fields.

[0011] The single-stage target detection algorithm has many advantages over the two-stage target detection algorithm. First of all, the calculation speed has been greatly improved, and the calculation cost has been reduced. It is suitable for the positioning and detection of small targets. But the accuracy may be slightly inferior to the two-stage detection algorithm. In the YOLO series of algorithms, it is found that the background interference in the image will affect the feature extraction of the detected target when the network backbone structure extracts features at the end. In addition, the PAN used in the neck network fusion needs to pay attention to the receptive field and feature fusion, which may not be sensitive enough for small targets. Most aerial images are small targets, so it is difficult to get a good fusion. The detection head of the YOLO series is composed of a detection head containing only a convolutional layer, which can only obtain local features. It is difficult to obtain information about the global context, resulting in poor global perception of the model. These are some of the current problems in the YOLO series of algorithms. Summary of the invention

[0012] In order to solve the problems existing in the prior art, the present invention provides an aerial image target detection method and system and related devices based on an attention mechanism, which fully utilizes the advantages of the attention mechanism and the visual Swin Transformer model, and the two complement each other. First, a background suppression module BSAM containing an attention mechanism is used to perform network feature extraction and background noise suppression at the end of the backbone network to reduce the problem of poor feature extraction ability caused by background suppression. The improved bidirectional weighted fusion pyramid module TWFP neck structure is used to replace the original PAN structure, and multi-scale fusion and improved perception of small targets are performed through weighted feature fusion and jump connection. The combination of a generalized efficient layer aggregation network GELAN and a visual Swin Transformer model is used to replace the original detection head containing only convolutional layers, so that the generalized efficient layer aggregation network GELAN and the visual Swin Transformer model can obtain local features and global context information and improve global perception capabilities. These improvements have better improved the detection performance of this method.

[0013] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an aerial image target detection method based on an attention mechanism, wherein an aerial image acquired by a drone is input into an aerial image target detection network model to obtain a target detection result of the aerial image;

[0014] Among them, the backbone network in the aerial image target detection network model is the YOLOv7 backbone network with the background suppression module BSAM including the attention mechanism, the neck structure is the bidirectional weighted fusion pyramid module TWFP, and the detection head structure is the combination of the generalized efficient layer aggregation network GELAN and the visual Swin Transformer model.

[0015] Furthermore, the YOLOv7 network backbone network includes 12 layers, the background suppression module BSAM with attention mechanism is located in the eleventh layer of the YOLOv7 network backbone network, and the fast spatial pyramid pooling module SPPF is located in the twelfth layer of the YOLOv7 network backbone network.

[0016] Furthermore, the background suppression module BSAM with attention mechanism is obtained by fusing the C3 module and the CBAMbottleneck with CBAM convolutional attention mechanism. The CBAMbottleneck with CBAM convolutional attention mechanism is connected to one of the first two convolutions of the C3 module. The CBAMbottleneck includes a 1x1 convolution layer, a 3x3 convolution layer, and a CBAM convolutional block attention mechanism. The features after the 1x1 convolution layer, the 3x3 convolution layer, and the CBAM convolutional block attention mechanism are residually connected to the input feature map, and the obtained feature map is convolved with the other branch of the C3 module to obtain the output feature map.

[0017] Furthermore, the neck structure of the bidirectional weighted fusion pyramid module TWFP includes a polymerization network structure PAN, and a bidirectional jump connection is added to the middle layer of the polymerization network structure PAN, and the weight calculation of the input features is increased in the weighted fusion part of the polymerization network structure PAN, so that the high-level semantic information and the low-level detail information are fully integrated.

[0018] Furthermore, the calculation of the bidirectional weighted fusion pyramid module TWFP in the neck structure is as follows:

[0019] F′4=Conv1(I4)

[0020] F′3=Conv1{ELAN[I3+up(F′4)]}

[0021] F'2=Conv1{ELAN[I2+up(F3')]}

[0022] F1'=Conv1[I1+up(F2')]

[0023] F1=GELAN-SW(F1')

[0024] F2=GELAN-SW[F2'+I2+Conv2(F1)]

[0025] F3=GELAN-SW[F3'+I3+Conv2(F2)]

[0026] F4=GELAN-SW[F4'+Conv2(F3)]

[0027] Among them, F1'~F4' represent the feature maps obtained from the bottom-up path, corresponding to levels I1~I4, and F1~F4 represent the feature maps output from the top-down path; + represents concat weighted feature fusion, up represents upsampling, Conv1 represents 1×1 convolution operation; Conv2 represents 3×3 convolution operation; ELAN represents that the feature map has passed through the ELAN module; GELAN-SW represents that the feature map has passed through the detection head.

[0028] Furthermore, the detection head structure includes four layers. The first layer contains a detection head that combines a generalized and efficient layer aggregation network GELAN and a visual Swin Transformer model; the second to fourth layers each contain two detection heads that combine a generalized and efficient layer aggregation network GELAN and a visual Swin Transformer model.

[0029] The present invention also provides an aerial image target detection system based on an attention mechanism, comprising:

[0030] A data acquisition module is used to acquire aerial images taken by a drone;

[0031] The network module building module is used to build an aerial image target detection network model. The backbone network in the aerial image target detection network model is a YOLOv7 network containing a background suppression module BSAM with an attention mechanism, the neck structure is a bidirectional weighted fusion pyramid module TWFP, and the detection head structure is a combination of a generalized efficient layer aggregation network GELAN and a visual SwinTransformer model;

[0032] The target detection module is used to input the aerial images acquired by the UAV into the aerial image target detection network model to obtain the target detection results of the aerial images.

[0033] The present invention also provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of an aerial image target detection method based on an attention mechanism as described in any one of claims 1 to 7 are implemented.

[0034] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the aerial image target detection method based on the attention mechanism as described in any one of claims 1 to 7 are implemented.

[0035] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of an aerial image target detection method based on an attention mechanism as described in any one of claims 1 to 6.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] The present invention provides an aerial image target detection method based on an attention mechanism, and constructs an empty image target detection network model. First, a background suppression module BSAM containing an attention mechanism is introduced into the network backbone structure to extract image features, which effectively suppresses the interference of complex backgrounds in aerial images, makes target features more prominent, and improves the accuracy of target detection; secondly, the neck structure adopts a bidirectional weighted fusion pyramid module TWFP to replace the traditional PAN structure, and through bidirectional jump connections and weighted fusion strategies, it realizes the full fusion of high-level semantic information and low-level detail information, and improves the model's detection ability for targets of different scales. Finally, the detection head structure adopts the form of combining a generalized efficient layer aggregation network GELAN with a visual Swin Transformer model, which not only retains the local feature extraction ability of a convolutional neural network, but also introduces the global context perception ability of a Transformer, so that the model can more accurately locate and classify targets during the detection process; in summary, the method of the present invention is an end-to-end single-stage target detection algorithm, which simplifies the detection process, improves the detection speed, and maintains a high detection accuracy.

[0038] Furthermore, the backbone network of the aerial image target detection network model of the present invention adopts the YOLOv7 framework, combined with the BSAM module, to improve the feature extraction capability; the neck structure realizes the effective fusion of multi-scale features through the TWFP module; the detection head improves the detection accuracy by combining the generalized efficient layer aggregation network GELAN with the visual Swin Transformer model. Each module in the model can be adjusted and optimized according to specific task requirements to adapt to different application scenarios and data sets. By introducing the attention mechanism and advanced network structure, the model has a high detection speed while maintaining high accuracy, and is suitable for application scenarios with high real-time requirements.

[0039] The improved network structure of the present invention can improve accuracy and stability, which means that we can obtain accurate detection results more reliably. After the algorithm can obtain accurate detection results, we can make more accurate decisions and plans in the fields of public safety, urban planning, traffic monitoring, etc. In the field of public safety, the detection method of the present invention can monitor road conditions in real time, detect abnormal events and potential threats in a timely manner, and provide strong guarantees for the safety and stability of the city. In urban planning, by detecting buildings, roads, green spaces and other targets in the city, accurate data support can be provided for urban planning, which helps to optimize urban layout and improve urban management. In the field of traffic monitoring, the detection method of the present invention can quickly and accurately identify targets such as vehicles and pedestrians, which helps to achieve real-time monitoring and intelligent scheduling of traffic flow and improve traffic operation efficiency. Therefore, our improved method is of great significance for improving decision-making effects and social management levels in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The invention discloses an aerial image target detection method based on an attention mechanism.

[0041] Figure 2 It is a background suppression module BSAM with attention mechanism in the aerial image target detection network model.

[0042] Figure 3 It is the CBAM convolutional attention mechanism in the aerial image target detection network model.

[0043] Figure 4 It is the neck structure-bidirectional weighted fusion pyramid module TWFP in the aerial image target detection network model.

[0044] Figure 5 It is a detection head that combines the generalized efficient layer aggregation network GELAN and the visual SwinTransformer model in the aerial image target detection network model.

[0045] Figure 6 This is the structural diagram of the generalized efficient layer aggregation network GELAN in the aerial image target detection network model.

[0046] Figure 7 This is a visualization of the results of different algorithms in the aerial image target detection network model on the VisDrone dataset. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0048] The present invention provides an aerial image target detection method based on an attention mechanism, which can detect targets in images more accurately in complex aerial images by integrating the attention mechanism and the visual Swin Transformer model. The method of the present invention specifically includes the following steps:

[0049] Step 1: Use two public datasets as input images for training;

[0050] Images from two public datasets are used as input. These datasets are VisDrone and UAVDT. VisDrone2019 consists of 7,019 high-resolution aerial images (2,000×1,500) from 10 categories. The dataset is characterized by small object size, dense distribution, and possible partial occlusion. The UAVDT dataset is a dataset (1,024×540) designed for vehicle detection tasks from the perspective of unmanned aerial vehicles (UAVs). It contains images taken by drones at different heights and angles on urban roads. Each of these datasets has different scenes, density, and image quality to ensure that the model has good generalization performance in various environments.

[0051] The specific implementation process of the embodiment is described as follows:

[0052] First, the dataset needs to be converted into yolo format to meet the experimental requirements. In the VisDrone dataset, 6,471 images are used for training and 548 images are used for verification. The dataset image input is set to 1536×1536. The UAVDT dataset consists of 50 videos, containing a total of 40,376 images. In the UAVDT dataset, 24,778 images are used for training and 15,598 images are used for testing. The resolution of all images is 1024×540. According to the requirements of the dataset, all images from the same video are grouped into a training set or a test set. Among these 50 videos, images from 31 videos are placed in the training set, and images from the remaining 19 videos are placed in the test set. By using these two datasets, the invention can comprehensively cover different scenes and target categories, and can evaluate and compare the performance of target detection algorithms in different datasets.

[0053] Step 2: Build the ASA-YOLO network for aerial image target detection

[0054] The target detection network uses the YOLOv7 network. Most of the targets in aerial images are small, so when the backbone network performs deep feature extraction, small targets often lose a lot of semantic information and are more easily affected by the image background. Therefore, a background suppression module BSAM with an attention mechanism is added to the end of the backbone network of the YOLOv7 network to extract the feature information of the target in the image, provide rich information for the follow-up, and reduce the interference of the small target features caused by the complex background at the end of the backbone network. In the neck architecture of the YOLOv7 network, the bidirectional weighted fusion pyramid module TWFP is used to replace the original PAN structure, which can better ensure that the features of different levels can communicate with each other and enrich the feature representation. In the detection head of the YOLOv7 network, the original detection head containing only convolutional layers is replaced with a combination of a generalized efficient layer aggregation network GELAN and a visual Swin Transformer model. The generalized efficient layer aggregation network GELAN is used to extract local information in the target, and then the visual Swin Transformer model is used to capture the global context information of the image. The combination of the two effectively improves the detection accuracy.

[0055] Overall, the target detection network of the present invention brings about improvements in accuracy and stability, which means that the target detection network of the present invention can more reliably obtain accurate detection results. The method of the present invention can make more accurate decisions and planning in the fields of public safety, urban planning, traffic monitoring, etc., which is of great significance to improving the decision-making effect and social management level in practical applications.

[0056] Step 2.1 Build the backbone network

[0057] like Figure 1 As shown in the figure, the backbone network is the feature extraction part of the entire target detection network. The backbone network of the YOLOv7 network is adopted, in which the eleventh layer is the background suppression module BSAM fused with the attention mechanism, and the twelfth layer is the fast spatial pyramid pooling module SPPF. By adjusting the width and depth of the backbone network, the entire target detection network is made more efficient.

[0058] Specifically: First, the first ten layers of the backbone network use the backbone network of the YOLOv7 network. This backbone network is an improved version of CSPDarknet53. By introducing the CSPNet structure and further optimization, the feature extraction capability and computational efficiency of the network are improved. This makes YOLOv7 perform well in target detection tasks and become an important version of the modern target detection model.

[0059] In the eleventh layer of the backbone network, the present invention adopts a background suppression module BSAM fused with an attention mechanism, which incorporates the CBAM convolution attention mechanism, not only maintaining the feature information of small targets during the downsampling process, but also suppressing irrelevant background noise in the image, while taking into account channel and position information, being flexible and lightweight enough, and being able to improve detection accuracy with a relatively small computational cost. Immediately after the eleventh layer, a fast spatial pyramid pooling module SPPF is connected, which extracts and fuses high-level features, and repeatedly uses 3x3 maximum pooling operations during the fusion process to extract as many high-level semantic features as possible, while maintaining the multi-scale feature fusion effect, improving computational efficiency. Later, when entering the neck architecture, the 5th, 7th, 9th, and 12th layers in the backbone network enter the neck architecture with feature maps of different levels, providing strong support for the next step of multi-level feature fusion.

[0060] The specific implementation process of the embodiment is described as follows:

[0061] In the invention, the backbone network is mainly composed of 12 layers, and the input image is a given aerial image I = R W×H×3 (W, H and channel size 3 represent the basic features of a given crowd counting image. W represents the width of the image, H represents the height of the image, and channel size 3 means that the image has three channels, usually in RGB color mode). The first layer is a convolutional layer with a kernel size of 3 and a stride of 1. The second layer is a convolutional layer with a kernel size of 3 and a stride of 2. The third layer is a convolutional layer with a kernel size of 3 and a stride of 1. The fourth layer is a convolutional layer with a kernel size of 3 and a stride of 2. The fifth layer is an ELAN module composed of convolution and residual modules; the sixth layer is a convolutional layer with a kernel size of 3 and a stride of 2. The seventh to tenth layers repeat the fifth and sixth layers.

[0062] The ELAN module passes the input feature map through two 1x1 convolutional layers to reduce the number of channels and improve computational efficiency. Then, it processes and extracts the feature map layer by layer through multiple 3x3 convolutional layers. Finally, it selects specific feature maps through the index list, concatenates them together, and fuses them through a 1x1 convolutional layer to obtain the final output. This design can enhance the feature representation capability while maintaining computational efficiency, enabling the model to better extract and fuse features when processing complex tasks.

[0063] The eleventh layer is the BSAM module that integrates the CBAM convolutional attention mechanism, such as Figure 2As shown in the figure, the background suppression module BSAM with attention mechanism is formed by the fusion of C3 module and CBAMbottleneck with CBAM convolutional attention mechanism. The C3 module includes three convolution operations, the first two convolution operations process input features independently, and the third convolution operation is used to merge the outputs of the first two paths; and CBAMbottleneck is connected to one of the first two convolutions of the C3 module. CBAMbottleneck includes two convolution layers, CBAM convolutional block attention module and residual connection shortcut.

[0064] In the BSAM module, the input feature map is reduced in number of channels through two 1x1 convolution layers of the C3 module. The CBAMbottleneck performs feature processing on one of the first two convolutions, convolves the processed feature map with the other to achieve splicing, and finally converts the spliced ​​feature map into the required number of output channels through a 1x1 convolution layer. This design can reduce the amount of calculation while ensuring the feature expression capability.

[0065] In CBAMbottleneck, a 1x1 convolution layer is used to reduce the number of channels, and then a 3x3 convolution layer is used to increase the number of channels, and then the CBAM convolution block attention mechanism is entered. Figure 3 As shown in the figure, the CBAM attention mechanism consists of two parts: the channel attention module and the spatial attention module. When the feature map is input into the CBAM attention mechanism, it first enters the channel attention module for global pooling and multi-layer perceptron operations, adds the two outputs to generate a channel attention map, and then multiplies the channel attention map and the original feature map element by element by channel to obtain the weighted feature map F′. The weighted feature map input enters the spatial attention module again for feature map pooling and convolution operations, and multiplies the spatial attention map and the weighted feature map F′ element by element by spatial position to obtain the final weighted feature map F″. The final output F″ is residually connected to the input feature map of CBAMbottleneck. CBAMbottleneck combines the CBAM attention mechanism to reduce the interference of the background on the target, and the use of residual connection more effectively promotes gradient flow and convergence speed.

[0066] The twelfth layer is the fast spatial pyramid pooling module SPPF.

[0067] The fast spatial pyramid pooling module SPPF first performs a 1x1 convolution on the input, then performs three maximum poolings in sequence, concatenates the original feature map and the results of the three maximum poolings, and finally reduces the number of channels of the concatenated feature map to the desired number of output channels through another 1x1 convolution layer. This design can enhance the feature representation capability while ensuring computational efficiency, allowing the model to perform better when processing objects of different scales.

[0068] The input image is 1536×1536×3, and after passing through the twelve-layer backbone network, it becomes an output of 46×46×1024. The backbone network feature extraction process is expressed as:

[0069] F c =C zgwl (I) (1)

[0070] F c represents the features from CNN, C zgwl It represents the backbone network for feature extraction, and I represents the input image.

[0071] Step 2.2, construct the neck structure of the bidirectional weighted fusion pyramid module TWFP:

[0072] like Figure 4 As shown in the figure, the bidirectional weighted fusion pyramid module TWFP uses bidirectional weighted fusion and jump connection to improve the entire neck. In the feature fusion stage, it can better realize the multi-scale fusion of feature maps and provide more accurate prediction detection for subsequent target detection tasks.

[0073] The specific improvements are:

[0074] The aggregation network structure PAN structure adopts bidirectional connection, which includes both top-down feature propagation and bottom-up feature propagation. This ensures that features at different levels can communicate with each other and enrich feature representation. The weighted bidirectional feature pyramid network BiFPN introduces a learnable weight mechanism and jump connection, which can perform weighted averaging and fusion of features of different scales, thereby more effectively utilizing feature information. Under the influence of the original structure path aggregation network structure PAN and the weighted bidirectional feature pyramid network BiFPN structure, the present invention retains some structures of the two, improves and fuses them, and obtains the bidirectional weighted fusion pyramid module TWFP.

[0075] The specific implementation process of the embodiment is described as follows:

[0076] like Figure 4 As shown. Figure 4It is a TWFP structure. The model takes the four feature maps of layers 5, 7, 9, and 12 extracted from the backbone network as input, performs bottom-up feature propagation from F′4 to F′1, and then performs top-down feature propagation from F1 to F4. Bidirectional jump connections are added to the middle two layers of the original PAN structure. The bidirectional jump connections allow information to be directly transferred from the previous layer to the next layer, bypassing the middle layer and directly fusing feature maps. In this way, high-level semantic information and low-level detail information can be fully integrated. This multi-scale feature fusion can enhance the network's ability to handle targets of different sizes and improve the accuracy of target detection.

[0077] After the bidirectional jump connection, the feature maps are weighted fused. Learn the importance of different input features and fuse different input features differently. Traditional feature fusion is often just a simple superposition or addition of feature maps, first adjust them to the same resolution, and then add them together. Previous methods treat all input features equally without distinction. However, different input feature maps have different resolutions, and their contributions to the fused input feature maps are also different, so simply adding or superimposing them is not the best operation. So add an additional weight to each input to let the network learn the importance of each input feature.

[0078] Taking F′2 and F2 as examples, the calculation method of weighted fusion is as follows:

[0079]

[0080] Among them, I2, F2', F2 represent the feature maps output by the backbone network, obtained from the bottom up, and output from the top down, ω represents the weight, ε represents a small constant, Resize represents the upsampling or downsampling operation of resolution matching, and Conv represents the convolution operation for feature processing.

[0081] The feature maps of the input image are 384×384×256, 192×192×512, 96×96×1024, and 48×48×1024, among which the feature map resolution of I1 is the highest, and the resolution of subsequent feature maps is halved. In order to enhance the ability of the framework to fuse multi-scale features and reduce the complexity of the framework, the present invention chooses to add a bidirectional jump connection between the two middle layers of the module in the actual experiment. This enables the effective capture of feature information of different scales. Finally, F1, F2, F3, and F4 are output. The entire fusion process is as follows:

[0082] The I4 feature map is convolved with a 1×1 convolution kernel and a stride of 1 to obtain the F4' feature map.

[0083] The F4' feature map is first upsampled by the nearest neighbor interpolation to improve the resolution of the feature map. Then it is concat weighted feature fusion with the I3 feature map. The weighted fusion feature map enters the ELAN convolution module (reducing the number of channels) for feature processing, and then passes through a 1×1 convolution kernel with a stride of 1. The F'3 feature map is obtained after the operation.

[0084] The F3' feature map is first upsampled by the nearest neighbor interpolation to improve the resolution of the feature map. Then it is concat weighted with the I2 feature map. The weighted fused feature map enters the ELAN convolution module (reducing the number of channels) for feature processing, and then passes through a 1×1 convolution kernel with a stride of 1. The F'2 feature map is obtained after the operation.

[0085] The F′2 feature map is first upsampled by the nearest neighbor interpolation to improve the resolution of the feature map. Then it is concat weighted feature fusion with the I1 feature map. The weighted fusion feature map is convolved with a 1×1 convolution kernel and a stride of 1 to obtain the F′1 feature map.

[0086] The F′1 feature map is passed through the generalized efficient layer aggregation network GELAN + visual Swin Transformer model detection head to perform local and global feature extraction to obtain the F1 feature map.

[0087] The F1 feature map first undergoes a convolution operation with a 3×3 convolution kernel and a stride of 2, then I2, F2', and F1 are fused through concat weighted features (I2 skips the intermediate layer and directly performs weighted fusion with F2' and F1), and then undergoes a convolution operation with a 1×1 convolution kernel and a stride of 1, and finally input into the generalized efficient layer aggregation network GELAN + visual Swin Transformer model detection head for local and global feature extraction to obtain the F2 feature map.

[0088] The F2 feature map first undergoes a convolution operation with a 3×3 convolution kernel and a stride of 2, then F3', I3, and F2 undergo concat weighted feature fusion (I3 skips the intermediate layer and directly performs weighted fusion with F3' and F2), and then undergoes a convolution operation with a 1×1 convolution kernel and a stride of 1, and finally inputs the generalized efficient layer aggregation network GELAN+visual Swin Transformer model detection head for local and global feature extraction to obtain the F3 feature map.

[0089] The F3 feature map first undergoes a convolution operation with a 3×3 convolution kernel and a stride of 2, and then F3 and F′4 are concat weighted feature fused. Finally, it is input into the generalized efficient layer aggregation network GELAN + visual Swin Transformer model detection head for local and global feature extraction to obtain the F4 feature map.

[0090] F′4=Conv1(I4) (4)

[0091] F′3=Conv1{ELAN[I3+up(F′4)]} (5)

[0092] F2'=Conv1{ELAN[I2+up(F3')]} (6)

[0093] F1'=Conv1[I1+up(F2')] (7)

[0094] F1=GELAN-SW(F1') (8)

[0095] F2=GELAN-SW[F2'+I2+Conv2(F1)] (9)

[0096] F3=GELAN-SW[F3'+I3+Conv2(F2)] (10)

[0097] F4=GELAN-SW[F4'+Conv2(F3)] (11)

[0098] Among them, F′1~F′4 represent the feature maps obtained from the bottom-up path, corresponding to levels I1~I4, and F1~F4 represent the feature maps output from the top-down path. + represents concat weighted feature fusion, up represents upsampling, Conv1 represents 1×1 convolution operation. Conv2 represents 3×3 convolution operation. ELAN represents that the feature map has passed through the ELAN module. GELAN-SW represents that the feature map has passed through the generalized efficient layer aggregation network GELAN+visual Swin Transformer model detection head.

[0099] Step 2.3: Build the detection head structure

[0100] The structure of the new detection head designed by the invention is as follows Figure 5 As shown in the figure, the detection head in the model has four layers, of which the first layer contains a generalized efficient layer aggregation network GELAN + visual Swin Transformer model detection head. The second to fourth layers each contain two generalized efficient layer aggregation network GELAN + visual Swin Transformer models, and the generalized efficient layer aggregation network GELAN-visual Swin Transformer model is stacked twice in sequence to generate output. The feature map of the neck after multi-scale fusion is input into the generalized efficient layer aggregation network GELAN to extract the local information of the features in the feature map. The structure of the generalized efficient layer aggregation network GELAN is as follows: Figure 6As shown in the figure, by combining the characteristics of CSPNet and the convolution module ELAN, the efficiency and effect of the model in processing complex data patterns are improved. The extracted feature map is passed to the visual Swin Transformer model again for global feature extraction. The globally extracted feature map is passed to the detection module to generate a tensor containing bounding box coordinate information, confidence, and category classification. Then the tensor generated by each detection layer is decoded, and the bounding boxes, confidences, and category predictions of the four detection layers after decoding are integrated together to obtain the final detection result.

[0101] The specific implementation process of the embodiment is described as follows:

[0102] In the present invention, after the neck is subjected to multi-scale feature fusion, there will be 4 layers of feature maps of different scales as the input of the detection head and enter the generalized efficient layer aggregation network GELAN for processing and calculation. The generalized efficient layer aggregation network GELAN extracts features through multiple convolutional layers and RepNCSP modules. The input feature map first reduces the number of input channels through a 1x1 convolutional layer, and then uses chunk to divide the output into two tensors along the channel dimension, which pass through a RepNCSP module and a 3x3 convolutional layer respectively. Finally, all feature maps are spliced ​​together and output. The design of this module can enhance the ability of feature extraction while ensuring computational efficiency. The feature map output after the operation is passed into the visual Swin Transformer model.

[0103] The input feature map in the visual Swin Transformer model is represented as W×X×C. W represents the width of the image, H represents the height of the image, and the number of channels is C. The input feature map is divided into a series of fixed-size patches (small blocks) by Patch Partition. The size of each patch is P×P×C, where P is the side length of the patch. This process converts the input image from a two-dimensional structure to a one-dimensional sequence of patch tokens. Then the patch tokens are mapped to a high-dimensional space through a linear transformation through the fully connected layer Linear Embedding to obtain Patch Embedding.

[0104] Patch Embedding=Linear(Flatten(Patch)) (12)

[0105] The generated Patch Embedding enters Layer Normalization, which is usually applied to the input of each sub-layer in the visual Swin Transformer model. It mainly plays a role in improving the convergence speed and enhancing the robustness. Then, by introducing the Window Multi-Head Self-Attention (W-MHSA) module and the Shifted Window Multi-Head Self-Attention (SW-MHSA) module, the ability to capture contextual information and spatial information in the image is improved. The W-MHSA module uses a window attention mechanism to divide the input Patch Embedding into multiple windows, and effectively models local information by calculating the attention weights in each window. The SW-MHSA module uses a shift window attention mechanism to consider the relationship between adjacent windows and effectively capture local information. The MLP multi-layer perceptron is used in the visual Swin Transformer model to perform nonlinear transformations on the last part of each visual Swin Transformer model Layer. Its nonlinear transformation capability enables the network to learn richer feature representations. The calculation process is as follows:

[0106] F'=W-MHSA(LN(F))+F (13)

[0107] F'=MLP(LN(F'))+F' (14)

[0108] F"=SW-MHSA(LN(F'))+F' (15)

[0109] F"=MLP(LN(F"))+F" (16)

[0110] After being processed by the visual Swin Transformer model, the output feature map enters the Detect module again to generate a tensor containing bounding box coordinates, confidence, and category classification information. For the prediction of each detection layer, the bounding box needs to be decoded. The prediction in the present invention is given in the form of center coordinates and width and height, and the relative position and scale information usually need to be converted into actual image coordinates. The decoding of confidence and category prediction is calculated by the sigmoid function. The bounding boxes, confidence and category predictions of the four detection layers after decoding are integrated together to obtain the final detection result.

[0111] Step 3: Training and testing output results

[0112] The experiment was conducted using an NVIDIA A800-SXM4 graphics card with 80GB video memory and 16 CPU cores. The experimental software was configured with PyTorch 2.21 and CUDA 12.0. The training phase included 200 times, with the first two times used as warm-up. The initial learning rate was set to 3.2×10-4 and decayed to 0.12 times its value in the final epoch. The Adam optimization algorithm was used for parameter optimization. The performance of the target detection network ASA-YOLO was optimized by minimizing the loss function. The target detection network ASA-YOLO was iteratively trained, and the model was continuously optimized through backpropagation and gradient descent according to the error calculated by each forward propagation, and finally the trained network model was obtained. The trained target detection network model was used to test the test set and output the test results.

[0113] The specific implementation process of the embodiment is described as follows:

[0114] The experimental configuration and optimization strategy of the ASA-YOLO model in the present invention on the PyTorch platform. The present invention uses a hardware environment equipped with an 80G NVIDIA A800-SXM4 GPU to train and evaluate the model of the present invention, and uses a publicly accessible dataset to perform target detection tasks. The evaluation indicators of the present invention mainly include the average precision AP and the average precision AP50 calculated when the IoU threshold is 0.5 as evaluation indicators.

[0115] The proposed ASA-YOLO model consists of three parts: the backbone network for feature extraction, the neck for multi-scale fusion, and the detection head for outputting results. The backbone network extracts the feature information of the target in the image, and introduces the background suppression module BSAM with an attention mechanism to reduce the interference of small target features caused by complex background at the end of the backbone network. The neck structure uses the improved weighted bidirectional feature fusion architecture TWFP for multi-scale fusion, which can better ensure that features at different levels can communicate with each other and enrich feature representation. In the detection head part, the generalized efficient layer aggregation network GELAN and the visual Swin Transformer model are combined to extract local information in the target with the generalized efficient layer aggregation network GELAN, and then the visual Swin Transformer model captures the global context information of the image. The combination of the above three effectively improves the detection accuracy.

[0116] During the training process, the present invention uses 1536×1536 for the VisDrone dataset and 1024×1024 for the UAVDT dataset as model input, and sets the batch size to 4 according to the computing power of the GPU. The present invention uses the Adam optimizer for model training, and sets the learning rate to 3.2×10-4, and the weight decay to 0.12 times its value to minimize the loss function to optimize the network performance. These experimental configurations and optimization strategies are intended to ensure good performance and reliability of the model in target detection tasks. Figure 7 This is the visualization result after comparing with yolov7.

[0117] We conducted extensive experiments on two public datasets (VisDrone and UAVDT) to verify that our approach achieves excellent results.

[0118] Table 1

[0119]

[0120] Table 2

[0121]

[0122] The experiments conducted by the present invention on the VisDrone dataset, in order to evaluate the performance of ASA-YOLO, the present invention compares its experimental results with the experimental results of other eleven state-of-the-art methods on the VisDrone dataset, as shown in Table 1. The average precision (AP50) of the present invention at IoU=0.5 is 64.3%, which obtains the highest value of this indicator, while the average precision (AP) of the present invention reaches 41.8, which is only 0.3 less than the best SOTA. Compared with the baseline method, GSE-YOLO shows significant improvements in AP50 (increased from 61.5% to 64.3%) and AP (increased from 39.6% to 41.8%). These components significantly improve the feature extraction capability of detecting targets of various scales in drone captured images, while also reducing the occurrence of false alarms. In order to capture the comprehensive feature information of small targets in complex scenes, the present invention increases the depth and width of the network. In terms of model complexity, the computational complexity of the model of the present invention is 551.0G FLOPs. The computational complexity of TPH-YOLOv5 reaches 556.6GFLOPS, and the computational complexity of the method of the present invention is less than that of TPH-YOLOv5. When evaluated using the AP50 indicator, the method is 1.2% higher than TPH-YOLOV5. Although the AP is 0.2% lower, the AP50 of the present invention achieves SOTA performance. Its superior accuracy is highlighted. Experimental comparison shows that a slight increase in complexity can significantly improve performance.

[0123] Experiments conducted by the present invention on the UAVDT dataset. In order to further verify the present invention method, the present invention conducted experiments on the UAVDT dataset, as shown in Table 2. Compared with the existing methods, the ASA-YOLO of the present invention achieved the best SOTA effect on AP50. Among them, AP is only 0.2% less than SMFF-YOLO. In summary, the method has good detection accuracy in small targets and dense scenes, and overcomes the challenges of target detection in complex scenes to a certain extent.

[0124] The present invention also provides an aerial image target detection system based on an attention mechanism, which can be used to implement the above-mentioned aerial image target detection method based on an attention mechanism. Specifically, the aerial image target detection system based on an attention mechanism includes a data acquisition module, a network module construction module and a target detection module, wherein:

[0125] A data acquisition module is used to acquire aerial images taken by a drone;

[0126] The network module building module is used to build an aerial image target detection network model. The backbone network in the aerial image target detection network model is a YOLOv7 network containing a background suppression module BSAM with an attention mechanism, the neck structure is a bidirectional weighted fusion pyramid module TWFP, and the detection head structure is a combination of a generalized efficient layer aggregation network GELAN and a visual SwinTransformer model;

[0127] The target detection module is used to input the aerial images acquired by the UAV into the aerial image target detection network model to obtain the target detection results of the aerial images.

[0128] An embodiment of the present invention provides a terminal device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned aerial image target detection method based on the attention mechanism are implemented; or, when the processor executes the computer program, the functions of each module in the above-mentioned aerial image target detection system based on the attention mechanism are implemented.

[0129] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0130] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0131] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0132] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0133] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0134] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned aerial image target detection method based on the attention mechanism can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0135] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0136] In summary, the aerial image target detection method based on the attention mechanism proposed in the present invention is performed on an end-to-end single-stage target detection algorithm. The aerial image target detection network model of the invention solves the problems encountered by the three parts of the trunk end, the neck structure and the detection head. The invention has a great advantage in detection speed compared to the two-stage detection method, and the accuracy is higher than the two-stage detection methods such as ClusDet and DMNet. While improving efficiency, it also obtains high-precision returns. Compared with traditional target detection methods and two-stage detection methods, the method proposed by the invention has great advantages.

Claims

1. A method for aerial image target detection based on attention mechanism, characterized in that: The aerial images acquired by the UAV are input into the aerial image target detection network model to obtain the target detection results of the aerial images; Among them, the backbone network in the aerial image target detection network model is the YOLOv7 backbone network including the background suppression module BSAM with attention mechanism, the neck structure is the bidirectional weighted fusion pyramid module TWFP, and the detection head structure is the combination of the generalized efficient layer aggregation network GELAN and the visual Swin Transformer model.

2. The method for aerial image target detection based on attention mechanism according to claim 1, characterized in that: The YOLOv7 network backbone network consists of 12 layers. The background suppression module BSAM with attention mechanism is located in the eleventh layer of the YOLOv7 network backbone network, and the fast spatial pyramid pooling module SPPF is located in the twelfth layer of the YOLOv7 network backbone network.

3. The method for aerial image target detection based on attention mechanism according to claim 1 or 2, characterized in that: The background suppression module BSAM with attention mechanism is obtained by fusing the C3 module and the CBAMbottleneck with CBAM convolutional attention mechanism. The CBAMbottleneck with CBAM convolutional attention mechanism is connected to one of the first two convolutions of the C3 module. CBAMbottleneck includes 1x1 convolution layer, 3x3 convolution layer, and CBAM convolution block attention mechanism. The features after 1x1 convolution layer, 3x3 convolution layer, and CBAM convolution block attention mechanism are residually connected with the input feature map, and the obtained feature map is convolved with the other branch of the C3 module to obtain the output feature map.

4. The method for aerial image target detection based on attention mechanism according to claim 1, characterized in that: The neck structure of the bidirectional weighted fusion pyramid module TWFP includes a polymerization network structure PAN, and a bidirectional jump connection is added to the middle layer of the polymerization network structure PAN. The weight calculation of the input features is increased in the weighted fusion part of the polymerization network structure PAN, so that the high-level semantic information and the low-level detail information are fully integrated.

5. The method for aerial image target detection based on attention mechanism according to claim 4, characterized in that: The calculation of the bidirectional weighted fusion pyramid module TWFP in the neck structure is as follows: F4'=Conv1(I4) F3'=Conv1{ELAN[I3+up(F4')]} F2'=Conv1{ELAN[I2+up(F3')]} F1'=Conv1[I1+up(F2')] F1=GELAN-SW(F1') F2=GELAN-SW[F2'+I2+Conv2(F1)] F3=GELAN-SW[F3'+I3+Conv2(F2)] F4=GELAN-SW[F4'+Conv2(F3)] Among them, F1'~F4' represent the feature maps obtained from the bottom-up path, corresponding to levels I1~I4, and F1~F4 represent the feature maps output from the top-down path; + represents concat weighted feature fusion, up represents upsampling, Conv1 represents 1×1 convolution operation; Conv2 represents 3×3 convolution operation; ELAN represents that the feature map has passed through the ELAN module; GELAN-SW represents that the feature map has passed through the detection head.

6. The method for aerial image target detection based on attention mechanism according to claim 1, characterized in that: The detection head consists of four layers. The first layer contains a detection head that combines a generalized and efficient layer aggregation network GELAN and a visual Swin Transformer model. The second to fourth layers each contain two detection heads that combine a generalized and efficient layer aggregation network GELAN and a visual Swin Transformer model.

7. An aerial image target detection system based on attention mechanism, characterized in that: include: A data acquisition module is used to acquire aerial images taken by a drone; The network module building module is used to build an aerial image target detection network model. The backbone network in the aerial image target detection network model is a YOLOv7 network with a background suppression module BSAM that includes an attention mechanism, the neck structure is a bidirectional weighted fusion pyramid module TWFP, and the detection head structure is a combination of a generalized efficient layer aggregation network GELAN and a visual Swin Transformer model. The target detection module is used to input the aerial images acquired by the UAV into the aerial image target detection network model to obtain the target detection results of the aerial images.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the aerial image target detection method based on the attention mechanism as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of an aerial image target detection method based on an attention mechanism are implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of an aerial image target detection method based on an attention mechanism as described in any one of claims 1 to 6 are implemented.

Images