A vehicle detection method for multi-directional drone aerial photography based on attention mechanism

By introducing a hybrid attention mechanism and rotating RPN in Faster R-CNN, the problem of difficulty in detecting vehicles at small and medium-sized scales of aerial images of drones has been solved, and more accurate vehicle detection has been achieved, reducing false detection and missed detection, and improving detection performance.

CN112733691BActive Publication Date: 2024-11-05BEIJING UNIV OF TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110000536.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-04
Publication Date
2024-11-05
Estimated Expiration
2041-01-04

AI Technical Summary

Technical Problem

Existing object detection algorithms are difficult to accurately detect small-scale vehicles in aerial images of drones, especially in medium and low altitude images, where vehicle targets are small and background noise are complex, resulting in frequent false alarms and missed detection.

Method used

By introducing a feature pyramid network and a rotating area suggestion network (RPN) with a hybrid attention mechanism in Faster R-CNN, multi-directional suggestions with vehicle azimuth information are generated, and feature extraction and fusion are combined with the space-channel attention mechanism to enhance Semantic information reduces background noise impact and improves the accuracy of tilt vehicle detection through boundary box regression.

Benefits of technology

It significantly improves the accuracy and accuracy of vehicle detection in aerial images of drones, reduces missed and missed detection, and improves the performance of vehicle detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112733691B_ABST
    Figure CN112733691B_ABST
Patent Text Reader

Abstract

This invention discloses a multi-directional vehicle detection method for UAV aerial photography based on an attention mechanism. It designs a Feature Pyramid Network (RPN) that integrates the attention mechanism; based on vehicle characteristics, the RPN is designed and directional attributes are added; and the Faster R-CNN is improved by incorporating the above two steps. Vehicles are then detected, and results are obtained. In the Faster R-CNN design, by adding an attention mechanism, each channel and region of the FPN receives adaptive weights, enabling accurate detection of vehicles from multiple directions, thus improving vehicle counting and reducing false positives and false negatives.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of target detection in drone aerial images, and involves currently advanced target detection algorithms, attention mechanisms, and a multi-directional vehicle detection module for aerial images based on FasterR-CNN. Background Art

[0002] With its high mobility, rapid deployment and wide-range monitoring, drones play an important role in traffic management, parking management, urban planning, emergency rescue, etc. In recent years, with the increase in the number of vehicles, traffic problems have become increasingly prominent. With the help of vehicle detection, traffic flow calculations and traffic congestion predictions can be performed to better monitor and manage traffic. The premise of all this is to be able to accurately detect all vehicles.

[0003] In recent years, with the improvement of graphics processing technology, deep learning and artificial intelligence have achieved great success in computer vision and other fields. Many researchers have proposed target detection algorithms based on deep learning and achieved good results. They are generally divided into two modules: One-Stage and Two-Stage. Representative algorithms of One-Stage include SSD, Yolo, Yolov2, and Yolov3. Two-Stage mainly includes Faster-RCNN. However, they are developed for very different data sets. The images of these data sets, such as daily vehicle images such as Pascal VOC2007 and MSCOCO, only contain one object or occupy a large part of the image, while drone images lose the "face" of the vehicle, and the vehicle presents a typical straight line structure. The existence of non-vehicle straight line objects, such as power units and air conditioning units on the top of buildings, will complicate the task and cause many false alarms. In addition, the target objects in medium and low altitude images are relatively small, generally only 15-20 pixels, so it is a great challenge to accurately detect vehicles photographed by drones.

[0004] In summary, this invention extends the Faster R-CNN target detection framework. To address the current problem of low-altitude vehicle detection, we add the attention mechanism to FPN and cross-fuse its features to generate feature maps with richer semantic information, effectively alleviating the small-scale loss of small aerial targets, and enabling better image classification and regression. At the same time, in order to detect vehicles in different directions, we propose a rotating region proposal network (RPN) to generate multi-directional proposals with vehicle azimuth information. In RPN, the azimuth of the bounding box is also regressed to make the generation of tilted vehicle region proposals more accurate. Summary of the invention

[0005] The purpose of the present invention is to provide a high-performance vehicle detection module for images taken by unmanned aerial vehicles to solve the problems mentioned in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a vehicle detection method for multi-directional drone aerial photography based on an attention mechanism, comprising the following steps:

[0007] Step S1: Design a feature pyramid network that integrates the attention mechanism, namely the RPN network;

[0008] Step S2: Design the RPN network according to the vehicle characteristics and add direction attributes;

[0009] Step S3: Improve Faster R-CNN and add the above two steps to detect the vehicle and get the result.

[0010] Figure 1 The structural flow chart of the present invention. The present invention improves the original Faster R-CNN according to the characteristics of low-altitude and medium-altitude vehicles, wherein the ImageNet image data set is selected for pre-training to obtain the VGG16 convolution layer as the backbone network to extract feature values, and other layers are initialized in a random manner. In order to obtain the multi-scale feature expression of the image, the convolutional neural network is used to extract the hierarchical features, and then the feature pyramid is established; then the target candidate area is generated according to the RPN network, and the corresponding target area is found in each level of the feature pyramid, so as to extract the multi-scale features of the target. The pooled features are fused in the ROI Polling layer. The entire network structure adopts multi-task joint learning to simultaneously realize the detection, positioning and type recognition of the target.

[0011] Like FasterR-CNN, the present invention also uses two stages to detect the network. First, candidate regions are used to generate candidate targets, then the targets are classified, and finally the location and type of the targets are obtained. Unlike the conventional Faster R-CNN, the backbone network extracts multi-layer feature layers containing edge information and semantic information and fuses them as the input of the RPN network. By introducing the context information of the candidate regions and fusing the context information with the multi-scale features, the detection and recognition of aerial vehicles can be achieved.

[0012] As a further embodiment of the present invention:

[0013] The step S1 mainly adds a specially designed feature pyramid (FPN) to the FasterR-CNN backbone network, that is, adds a spatial-channel mixed attention mechanism to perform feature extraction to obtain higher-level semantic information. Figure 3The overall structure diagram of the FPN network with the hybrid attention mechanism added.

[0014] Aerial images often contain complex scenes and usually have strong background noise, which results in poor shooting effects. The attention mechanism can effectively process the background noise, focus on the area of ​​interest, and reduce the background reasoning in the image. In the FPN design, the attention mechanism is used to enable the feature pyramid to obtain connectivity through spatial and channel attention during the feature fusion process, so that the feature maps are merged in a weighted form.

[0015] This paper describes a feature pyramid network with a hybrid attention mechanism. The hybrid attention module consists of lateral channel attention and top-down spatial attention. The purpose is to allow each channel and region of FPN to obtain adaptive weights.

[0016] Since the FPN lateral path converts the dimensional map from a high-dimensional feature map to a low-dimensional feature, it is equivalent to compressing the channel and is considered to be a channel attention operation. Since the lateral path also requires the regional interdependence of the feature map, an autoencoder is introduced to implement the spatial attention operation, encode the feature map into a feature space, and then restore the compressed feature map to its original size and divide it to extract the feature interdependence. Finally, the output is normalized by sigmoid to [0,1]. The spatial attention formula can be expressed as

[0017] S(X r )=σ(AE(X a ), W AE )

[0018] Where W AE is the parameter to be learned by the encoder, σ is the sigmoid, X a is the channel aggregation feature map.

[0019] Similar to the lateral path, the top-down path is considered to be a spatial attention operation because the feature map space keeps getting smaller. In order to obtain the correlation dependency of the channels of the feature map, channel attention is introduced to obtain the spatial-channel feature map. For channel attention, the original feature map is first compressed to 1*1, then a 1*1 convolution kernel is used to reduce the feature dimension from C to c / 4, then the Relu function is activated, and finally the feature dimension is restored. While obtaining the channel weight, the correlation of the feature map at the channel level is obtained. The channel attention formula is expressed as

[0020] C(X)=σ(W,θ(W C g(X)))

[0021] Among them, W C Belongs to the original feature map power, g is the global pooling function, θ is the ReLU function, and σ is the sigmoid function.

[0022] Similar to the lateral path, the top-down path is considered to be a spatial attention operation because the feature map space keeps getting smaller. In order to further obtain the correlation dependency of the channels of the feature map, channel attention is introduced to obtain the spatial-channel feature map. For channel attention, the original feature map is first compressed to 1*1, then a 1*1 convolution kernel is used to reduce the feature dimension from C to c / 4, then the Relu function is activated, and finally the feature dimension is restored. While obtaining the channel weight, the correlation of the feature map at the channel level is obtained. The channel attention formula is expressed as

[0023] C(X)=σ(W,θ(W C g(X)))

[0024] Among them, W C Belongs to the original feature map power, g is the global pooling function, θ is the ReLU function, and σ is the sigmoid function.

[0025] For the kth feature map of FPN, the hybrid attention first obtains the basic fusion features The formula is as follows

[0026]

[0027] in It is a top-down feature map starting from the upper layer of the feature pyramid, Y l is the corresponding lateral feature map from the backbone network, τ(·) represents the upsampling process, which is achieved by interpolation algorithm or transposed convolution, ∈(·) represents the convolution-based dimensionality mapping, which is achieved by 1×1 convolution kernel, is the element-wise sum.

[0028] Next, the module generates a spatial attention mask on the lateral path and a channel attention mask on the top-down path, with the formulas being

[0029]

[0030]

[0031] Among them, S(·) is the spatial attention operation, C(·) is the channel attention operation, is a product operation. The final feature map is composed of and Generate, the formula is as follows

[0032]

[0033] The features obtained by dual-channel attention have more image information and semantic information than the feature maps obtained by the original FPN. Since each channel and region of the feature map has its own different objects, it is difficult to distinguish the foreground and background. The hybrid attention module can increase the network's attention to information channels and regions during feature fusion. The hybrid attention module of each layer of the feature pyramid has its own attention feature, which is dedicated to different scale attributes.

[0034] As a further embodiment of the present invention:

[0035] Since the direction of the vehicle in the aerial image is not fixed, the detection result may be inaccurate, resulting in a decrease in detection accuracy. In step S2 of the present invention, the RPN network of Faster R-CNN is improved.

[0036] The Faster-RCNN algorithm abandons the traditional sliding window and ss (Selective Search) method and directly uses RPN to generate detection boxes, which can greatly improve the speed of generating detection boxes. The RPN network is used to generate candidate boxes (region proposals). Its specific process is shown in the figure below. This layer uses softmax to determine whether the anchors belong to the foreground (foreground) or the background (background), and then uses bounding box regression to correct the anchors to obtain accurate candidate boxes.

[0037] In order to obtain vehicle candidate regions in different directions, the parameters of the anchor box are not just the scale and aspect ratio, but the size, shape and direction. Six different orientations are set, 0, π / 6, π / 3, π / 2, 2π / 3 and 5π / 6 are used to rotate the anchor box to different directions. The anchor box has five parameters t = (x, y, w, l, θ). This anchoring strategy can cover vehicle targets of almost all shapes. In this way, vehicle candidate regions of arbitrary directions are generated, and bounding box regression is further performed to make the proposal better fit the vehicle region with the candidate region as input.

[0038] As a further embodiment of the present invention:

[0039] In step S3, by expanding Faster R-CNN, an FPN network based on a hybrid attention mechanism is added for feature extraction, and an RPN network with directional attributes is added to better fit the vehicle area, thereby finally obtaining the vehicle detection result of the drone aerial photography.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In the design of Faster R-CNN, by adding the attention mechanism, each channel and area of ​​FPN can obtain adaptive weights, and vehicles in multiple directions can be accurately detected, so as to better count vehicles and reduce missed detections and false detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The overall flow chart of the present invention is

[0043] Figure 2 Feature pyramid

[0044] Figure 3 FPN architecture diagram with attention mechanism added to the present invention DETAILED DESCRIPTION

[0045] The technical solution of the present invention is clearly and completely described below in conjunction with the architecture diagram of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] The implementation plan of this experiment is as follows:

[0047] Step S1: Construct the experimental data set. The experiment is mainly conducted on the VEDIA data set, which is a data set for aerial image vehicle detection. The data set contains a large number of vehicles and other attributes, including vehicle direction, occlusion, etc. At the same time, some data from the COWC data set are added. It is an aerial image collected from various places such as the United States and Germany for vehicle detection. And some data are rotated at any angle and moved about 10 pixels in any direction to expand the data. The data set is generally divided into two categories: cars and trucks. The data set is divided into a training set and a test set in an 8:2 ratio.

[0048] Step S2: Construct a feature extraction network. In the selection of the Faster R-CNN backbone network, the fast and lightweight VGG16 is continued to be used as the feature extraction network for feature extraction, making the detection effect more accurate and faster. First, the convolution layer (Conv) + activation function (Relu) + pooling layer (Pooling) of the first 13 layers of the VGG16 network are used to extract the feature map of the image. From the figure, we can see that for an image of any size P×Q, the network first scales it to a fixed size M×N, and then sends the M×N image to the VGG16 network layer convolution operation. The convolution layer contains 13 Conv layers + 13 Relu layers + 4 Pooling layers. The parameters of all Conv layers are: the convolution kernel (kernel_size) size is equal to 3, pad is equal to 1, and all Pooling layer parameter settings are: kernel_size is equal to 2, stride is equal to 2. In Convlayers, all convolutions are expanded (pad=l, i.e. filled with a circle of 0s), causing the original image to become (M+2) x(N+2) in size, and then a 3x3 convolution is performed to output M×N.

[0049] Step S3: Construct a feature pyramid that introduces a hybrid attention mechanism. Since aerial images have strong background noise, the attention mechanism can effectively process the background noise, focus on the area of ​​interest, and weaken the background reasoning ability in the image. Figure 3 Spatial attention is added to the lateral channels of FPN, and channel attention is added to the top-down channels, so that FPN uses spatial attention and channel attention in the feature fusion process to obtain connections, so that feature maps are merged in a weighted form.

[0050] Step S4: Build the designed RPN network. In order to obtain vehicle candidate regions in different directions, the parameters of the anchor box are not just the scale and aspect ratio, but the size, shape and direction. Six different orientations are set, 0, π / 6, π / 3, π / 2, 2π / 3 and 5π / 6 are used to rotate the anchor box to different directions. The anchor box has five parameters t = (x, y, w, l, θ). This anchoring strategy covers almost all shapes of vehicle targets. In this way, vehicle candidate regions of arbitrary directions are generated, and bounding box regression is further performed to make the proposal better fit the vehicle region with the candidate region as input.

[0051] Step S5: Network training. Use pytorch to build and train on NVIDIA 1080TI. The computer processor is Intel(R) Xeon(R) CPU. E5-1650v2@3.5GHz, GPU memory is 16G, the initialization parameters of the improved Faster R-CNN are the parameters trained on ImageNet, the initial learning rate is 1e-4, the weight decay is 0.0001, the momentum of 0.9, the batch-size of each batch of input data for this model is 64, each training iteration has 16 groups, each group contains 4 pictures. 70 iterations are performed. The learning strategy is changed from Step mode to Inv mode;

[0052] L=L0(1+γt) -ε

[0053] Where L is the learning rate, L0 is the basic learning rate, γ is the learning change rate, and ε is the decay exponent.

[0054] Step S6: Network testing. For the evaluation of the model, standard accuracy, recall rate, average precision and F1 indicators are used. The evaluation formula is:

[0055]

[0056]

[0057]

[0058] TP is the target positive sample, i.e., the vehicle target; FP is the target negative sample, i.e., the non-vehicle target; and FN is the sample that is actually a positive sample but is classified as a negative sample. The distinction between the three is determined by the IOU (intersection over union) used by Pascal VOC or COCO. In this paper, an IOU greater than 0.5 is defined as true, otherwise it is false.

[0059] Table 1 shows the evaluation results of several advanced methods with existing public codes and the method proposed in this paper. Table 2 shows the detailed results of the method proposed in this paper, including recall rate, precision, F1 score, mAP, and processing time. mAP corresponds to the average value of the maximum precision under different recall values. To ensure fairness, all methods are performed on a set of unseen test sets.

[0060] Table 1 Comparison of mAP of different methods

[0061] Method mAP AVDNet 2019 52.34 R3-Net(R+F)2019 69.3 Faster-RCNN 2017 70.9 Improved Faster RCNN 2017 72.9 This method 75.3

[0062] Table 2 Specific performance values ​​of this method

[0063] frame FasterRER-CNN This method R 80.1 81.7 P 60.3 64.1 mAP 72.9 75.3 F1-score 69.6 71.8 Test Time(Per Pic) 0.048s 0.051s

[0064] By comparing the performance of VEDIA with the current excellent methods, this method is 4.4 percentage points higher than the traditional FasterR-CNN. Faster RER-CNN uses ResNet-101 instead of VGG16, the initial learning rate and momentum values ​​are set to 0.001 and 0.9, and a certain level of multi-scale representation is introduced into the network. Compared with Faster RER-CNN, there is an mAP increment of 2.4, which is a solid improvement. It can be seen that the network performance of the proposed improved hybrid attention and feature fusion feature pyramid has been greatly improved.

[0065] In drone images, since vehicle targets are relatively small, the pixels are relatively low, and the background is relatively complex, false detection and missed detection are inevitable. There are many problems that cause missed detection, such as the vehicle is in the shadow of the image or the vehicle is blocked by a building, and only a part of the vehicle is exposed in the image; it is also possible that some boxes are mistakenly detected as vehicles. The following table shows a comparison between Faster-RERCNN and the improved method.

[0066] Table 3 Comparison of detection results between FasterRER-CNN and the invention of this paper

[0067] Detection Methods Correct detection number Number of false positives Number of missed detections Faster RER-CNN 75 31 19 Our inventions 88 30 8

[0068] This method found some of the same images and conducted experimental comparisons on the two models. Compared with Faster RER-CNN, the number of correct detections by our proposed method has increased, and the most obvious is that the number of missed detections has been significantly improved. The reason may be that although Faster RER-CNN has multi-scale feature fusion representation, it still cannot prevent the loss of low-level features that are crucial to small object detection algorithms.

[0069] The present invention mainly uses deep convolutional deep networks to detect vehicles photographed by drones. Specifically, we extended Faster R-CNN for aerial vehicle detection and introduced a feature pyramid with a hybrid attention mechanism to further enhance the semantic information of features. We also redesigned the RPN network and added directional attributes so that it can detect vehicles in multiple directions. Tests on VEDIA show that the present invention has better detection performance for vehicles photographed by drones.

[0070] The above examples are only used to describe the present invention, rather than to limit the technical solutions described in the present invention. Therefore, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A vehicle detection method for multi-directional drone aerial photography based on attention mechanism, characterized by: The following steps are included: Step S1: Design a feature pyramid network that integrates the attention mechanism, namely the RPN network; Step S2: Design the RPN network according to the vehicle characteristics and add direction attributes; Step S3: Improve Faster R-CNN and add the above two steps to detect the vehicle and obtain the result; According to the characteristics of low-altitude and medium-altitude vehicles, the original Faster R-CNN was improved. The VGG16 convolutional layer obtained by pre-training with the ImageNet image dataset was selected as the backbone network to extract feature values. Other layers were initialized in a random manner, and the convolutional neural network was used to extract hierarchical features, and then a feature pyramid was established. Then, the target candidate region was generated according to the RPN network, and the corresponding target region was found in each level of the feature pyramid, thereby extracting the multi-scale features of the target. The pooled features are fused in the ROIPooling layer; The entire network structure adopts multi-task joint learning to achieve target detection, positioning and type recognition at the same time; The detection network is tested in two stages. First, candidate regions are used to generate candidate targets, the targets are classified, and finally the location and type of the targets are obtained. The backbone network extracts multi-layer feature layers containing edge information and semantic information and fuses them as the input of the RPN network. By introducing the context information of the candidate regions and fusing the context information with multi-scale features, the detection and recognition of aerial vehicles can be achieved. S2 improves the RPN network of Faster R-CNN as follows: RPN is used to generate detection boxes. The RPN network is used to generate candidate boxes. Softmax is used to determine whether the anchors belong to the foreground or background. Then, the bounding box regression is used to correct the anchors to obtain accurate candidate boxes. In order to obtain vehicle candidate regions in different directions, the parameters of the anchor box are not just the scale and aspect ratio, but the size, shape and direction; 6 different orientations are set, 0, π / 6, π / 3, π / 2, 2π / 3 and 5π / 6 are used to rotate the anchor box to different directions; the anchor box has five parameters t = (x, y, w, l, θ) to cover vehicle targets of all shapes; generate vehicle candidate regions in any direction, perform bounding box regression, and fit the vehicle region with the candidate region as input.

2. According to claim 1, a vehicle detection method for multi-directional drone aerial photography based on attention mechanism is characterized by: S1 adds feature pyramid FPN to the Faster R-CNN backbone network, that is, adds a spatial-channel hybrid attention mechanism to perform feature extraction to obtain higher-level semantic information; the reference attention mechanism enables the feature pyramid to obtain connectivity with spatial and channel attention during feature fusion, so that feature maps are merged in a weighted form; a comprehensive description of the feature pyramid network with a hybrid attention mechanism; The hybrid attention module consists of lateral channel attention and top-down spatial attention, allowing each channel and region of FPN to obtain adaptive weights; an autoencoder is introduced to implement the spatial attention operation, encode the feature map into a feature space, restore the compressed feature map to its original size, and divide it to extract feature interdependencies; Finally, the output is normalized to [0,1] by sigmoid; the spatial attention formula is expressed as S(X)=σ(AE(X a ),W AE ) Where W AE is the parameter to be learned by the encoder, σ is the sigmoid, X a is the channel aggregation feature map; In order to obtain the correlation dependency of the channels of the feature map, channel attention is introduced to obtain the spatial-channel feature map; for channel attention, the original feature map is first compressed to 1*1, and then a 1*1 convolution kernel is used to reduce the feature dimension from C to C / 4, and then the Relu function is activated, and finally the feature dimension is restored. While obtaining the channel weight, the correlation of the feature map at the channel level is obtained; the channel attention formula is expressed as C(X)=σ(W,θ(W C g(X))) Among them, W C Belongs to the original feature map power, g is the global pooling function, θ is the Relu function, and σ is the sigmoid function; For the kth feature map of FPN, the hybrid attention first obtains the basic fusion features The formula is as follows in It is a top-down feature map starting from the upper layer of the feature pyramid, Y l is the corresponding lateral feature map from the backbone network, τ(·) represents the upsampling process, which is achieved by interpolation algorithm or transposed convolution, ∈(·) represents the convolution-based dimension mapping, which is achieved by 1×1 convolution kernel, is the sum of the elements; Generate a spatial attention mask on the lateral path and a channel attention mask on the top-down path. The formulas are Among them, S(·) is the spatial attention operation, C(·) is the channel attention operation, is a product operation; the final feature map is composed of and Generate, the formula is as follows: The features obtained by dual-channel attention have more image information and semantic information than the feature maps obtained by the original FPN; the hybrid attention module increases the network's attention to information channels and regions during feature fusion; the hybrid attention module at each layer of the feature pyramid has attention features that are dedicated to different scale attributes.

3. According to claim 1, a vehicle detection method for multi-directional drone aerial photography based on attention mechanism is characterized by: In S3, by expanding Faster R-CNN, adding an FPN network based on a hybrid attention mechanism for feature extraction, and adding an RPN network with directional attributes to better fit the vehicle area, we finally get the vehicle detection results of drone aerial photography.

Citation Information

Patent Citations

  • Multi-scale target detection method based on self-attention mechanism

    CN110533084A

  • Aerial image multi-scale target detection method based on spatial pyramid attention driving

    CN111401201A

  • Multi-scale vehicle pedestrian detection method and device based on attention mechanism

    CN111814621A

  • Spatiotemporal action detection method

    US20210248378A1

  • A computer-implemented method and system for detecting small objects on an image using convolutional neural networks

    WO2020020472A1