Highway traffic monitoring system and method based on unmanned aerial vehicle aerial photography

Abstract

The application discloses a highway traffic monitoring system based on unmanned aerial vehicle aerial photography, which comprises a camera arranged on a holder, the holder being arranged on an unmanned aerial vehicle, the unmanned aerial vehicle flying at a low speed above a designated highway lane, and the unmanned aerial vehicle driving towards vehicles on the highway lane; the camera is used for photographing a road surface image of the designated highway lane based on the unmanned aerial vehicle and sending the road surface image to an image recognition unit; the image recognition unit recognizes the vehicles in the road surface image; and a road surface monitoring unit determines whether traffic congestion occurs on the designated highway lane based on the positions of the vehicles in adjacent frames.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and more particularly, the present application relates to a highway traffic monitoring system and method based on unmanned aerial vehicle aerial photography. BACKGROUND

[0002] With the continuous development and progress of society, the number of motor vehicles in China is constantly improving. According to statistics, as of August 2022, the number of motor vehicles in China reached 408 million. As a result, serious traffic jams occur during morning and evening rush hours in major cities, which puts more stringent requirements on urban traffic management departments. How to better manage and monitor road motor vehicles and timely detect traffic jams to quickly handle them to avoid larger-scale congestion is a pressing problem.

[0003] Using unmanned aerial vehicle aerial photography to detect road vehicles is a solution to improve traffic congestion problems, but unmanned aerial vehicle aerial photography for road vehicle detection faces some difficulties. During the feature extraction process of the image through the convolutional neural network, the network model undergoes a large number of convolution, pooling, and downsampling operations, and the image size gradually decreases, and the feature information in the image is also highly abstracted, resulting in the gradual loss of feature information in the image. Since most of the unmanned aerial vehicle aerial photography images are small targets, the feature processing of small targets will lose more feature information, and the detection effect of aerial small targets is poor, resulting in reduced vehicle detection accuracy in aerial images, which in turn affects the accuracy of traffic monitoring. SUMMARY

[0004] The present application provides a highway traffic monitoring system based on unmanned aerial vehicle aerial photography, aiming to improve the detection accuracy of vehicle targets in road images.

[0005] The present application is implemented as follows: a highway traffic monitoring system based on unmanned aerial vehicle aerial photography, the system comprises:

[0006] A camera mounted on a gimbal, the gimbal is located on an unmanned aerial vehicle, the unmanned aerial vehicle flies at a low speed above a designated high-speed lane, and the unmanned aerial vehicle travels towards the vehicles on the high-speed lane;

[0007] The camera captures road images of the designated high-speed lane and sends them to an image recognition unit, which identifies vehicle targets in the road images; a road monitoring unit determines whether traffic congestion occurs on the high-speed lane based on the number of vehicles and their relative positions in consecutive adjacent frames.

[0008] Further, the image recognition unit identifies vehicle targets in the road images based on the YOLOv5 network structure.

[0009] Furthermore, the YOLOv5 network structure consists of a Backbone layer, a Neck layer, and a Head layer connected in sequence. The last layer of the Backbone layer contains an MSPPF module, which includes:

[0010] SPPF1 and SPPF2 are connected in series; two Conv convolutional layers are connected in parallel with SPPF1 and SPPF2 respectively; the output of SPPF1 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer and SPPF2 respectively; the output of SPPF2 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer for fusion; the Concat fusion layer is connected to the Neck layer.

[0011] Furthermore, SPPF1 includes:

[0012] A standard convolutional layer (CBL) is connected in series with three max pooling layers (MaxPool). The outputs of the standard convolutional layer (CBL) and the three max pooling layers (MaxPool) are fused through a Concat fusion layer and then input into another standard convolutional layer (CBL).

[0013] Furthermore, SPPF2 includes:

[0014] A standard convolutional layer CBL, and three average pooling layers AvgPool connected in series with the standard convolutional layer CBL; the outputs of the standard convolutional layer CBL and the three average pooling layers AvgPool are fused through a Concat fusion layer and then input into another standard convolutional layer CBL.

[0015] Furthermore, the Neck layer consists of an FPN structure and a PAN structure. An attention module CBAM is added at the fusion point of the FPN and PAN structures to introduce the feature image output from the Backbone layer.

[0016] Furthermore, an attention module CBAM is set at the end of the FPN structure's Concat fusion layer to introduce the feature image output from the Backbone layer.

[0017] Furthermore, the road surface monitoring unit includes: a detection frame determination module and a road surface monitoring module.

[0018] The detection box determination module constructs a detection box based on the center of the image output by the image recognition unit, moves the detection box based on the sampling period and flight speed of the UAV, and then determines the position of the detection box in the next frame image.

[0019] If the number of vehicles in the detection box is the same in several adjacent frames and the relative positions between the vehicles remain basically unchanged, the road monitoring module determines that there is a traffic jam in the current highway lane; otherwise, it determines that there is no traffic jam in the current highway lane.

[0020] This invention is implemented as follows: a method for monitoring highway traffic based on drone aerial photography, the method specifically includes the following steps:

[0021] S1. Real-time capture of road surface images of the designated highway lane;

[0022] S2. Detect vehicle targets in the road surface image;

[0023] S3. Form a detection box in the middle area of ​​the road surface image, and determine whether the highway lane is congested based on the number of vehicles in the detection box and their relative positions in adjacent road surface images.

[0024] This invention improves the accuracy of vehicle target recognition in road images and the accuracy of traffic condition recognition on highways by improving the YOLOv5 network structure. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of a road vehicle detection system based on drone aerial photography provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the YOLOv5 network structure provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of the MSPPF module provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of SPPF1 and SPPF2 provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram illustrating the introduction of an attention channel according to an embodiment of the present invention;

[0030] Figure 6 A flowchart illustrating a highway traffic monitoring method based on drone aerial photography, provided in an embodiment of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0032] Figure 1 This is a schematic diagram of a highway traffic monitoring system based on UAV aerial photography provided in an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown. The system includes:

[0033] A camera is mounted on a gimbal, which is located on a drone. The drone flies at low speed over a designated highway lane, and the drone travels towards vehicles in that highway lane.

[0034] The camera captures real-time images of the road surface in the designated highway lane and sends them to the image recognition unit;

[0035] The image recognition unit identifies vehicles in road surface images;

[0036] The road surface monitoring unit determines whether traffic congestion has occurred in a designated highway lane based on the relative positions of vehicles in adjacent frames.

[0037] In this invention, the image recognition unit performs vehicle target recognition in road images based on an improved YOLOv5 network structure. The road images captured in real time by the camera are input into the improved YOLOv5 network structure, and the improved YOLOv5 network structure outputs vehicle targets, target detection boxes, and confidence scores in the road images.

[0038] In this embodiment of the invention, combined with Figure 2 A detailed description of the YOLOv5 network structure is provided. The YOLOv5 network structure consists of a Backbone layer, a Neck layer, and a Head layer connected sequentially. The last layer of the Backbone layer is set as an MSPPF module. The structure of the MSPPF module is as follows: Figure 3 As shown, it includes: SPPF1 and SPPF2 connected in series; two Conv convolutional layers connected in parallel with SPPF1 and SPPF2 respectively; the output of SPPF1 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer and SPPF2 respectively; the output of SPPF2 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer for fusion; the Concat fusion layer is connected to the Neck layer.

[0039] The MSPPF model is placed in the last layer of the backbone, replacing the SPPF in the original network. This fuses the feature information from shallower layers, which contains more fine-grained features, thereby improving the accuracy of small object detection. Furthermore, SPPF enhances the representation of deep feature information through multiple pooling and feature stacking methods. This invention uses SPPF1 and SPPF2 in series, avoiding excessive computation compared to parallel processing. Multiple convolutions followed by stacked pooling in a serial manner increase the receptive field of the image and enrich feature information. Residual connections are also made between SPPF1 and SPPF2, fully utilizing the advantage of ResNet networks in continuously increasing the number of convolutional layers while improving model accuracy, and reducing gradient vanishing caused by the series processing of SPPF1 and SPPF2.

[0040] In the embodiments of the present invention, the SPPF1 and SPPF2 described above are as follows: Figure 3 As shown, SPPF1 includes: a standard convolutional layer CBL, and three max pooling layers MaxPool connected in series with the standard convolutional layer CBL; the outputs of the standard convolutional layer CBL and the three max pooling layers MaxPool are fused through a Concat fusion layer and then input into another standard convolutional layer CBL.

[0041] SPPF2 includes: a standard convolutional layer CBL, and three average pooling layers AvgPool connected in series with the standard convolutional layer CBL; the outputs of the standard convolutional layer CBL and the three average pooling layers AvgPool are fused through a Concat fusion layer and then input into another standard convolutional layer CBL.

[0042] The standard convolution described above consists of three operations: Conv (convolution), BN (normalization), and activate (linear activation function).

[0043] Compared to SPPF1, SPPF2 replaces max pooling in SPPF1 with average pooling. Global average pooling is commonly used in image detection to obtain global contextual relationships. After SPPF1, image feature information is processed by SPPF2, allowing the network to acquire global information and preventing overfitting. Ultimately, this enables the network to obtain more feature information about small aerial targets. Compared to the original SPPF module, the MSPPF module provides an additional average pooling operation and fuses the results of the two pooling methods, allowing the model to extract more feature information about small targets.

[0044] In this embodiment of the invention, the Neck layer consists of an FPN structure and a PAN structure. An attention module CBAM is added at the fusion point of the FPN and PAN structures to introduce the feature image output from the Backbone layer. Furthermore, an attention module CBAM is also placed at the Concat fusion layer at the end of the FPN structure to introduce the feature image output from the Backbone layer.

[0045] By connecting feature information from more primitive layers and passing it to subsequent feature representations, feature fusion across different layers is enhanced, resulting in better feature representation. For example... Figure 5 As shown, P = {P1, P2, P3} represents feature maps of different scales from the feature extraction network, and the resolution of these feature maps decreases progressively. i ,i∈{1,2,3} represents the feature map of layer i extracted from the backbone feature network. The proposed scheme of this invention performs feature fusion in the following manner:

[0046] O i =M(q(P) i ),h(P i ),c(Pi i∈{1,2,3}

[0047] Among them, O i q(P) represents the feature map output after feature fusion. i This indicates bottom-up transmission; the FPN structure outputs P. i Feature map, h(P) i The function represents top-down transmission, and the PAN structure outputs P. i Feature map, c(P) i ) represents P i After the layer feature information is processed by attention CBAM, the residuals are directly connected to the end. M represents the fusion of feature information processed by the three methods, thereby generating a new feature map O. i This allows it to integrate more feature information, thus possessing a more powerful feature representation capability. Each layer contains a large amount of feature information from the original features. Therefore, it is more conducive to the detection of small targets in aerial photography.

[0048] Feature maps from the backbone of the network are extracted and fused with the terminal blocks after adding an attention mechanism module. The feature maps processed by the attention module possess multiple layers of feature map channels. By using the attention module to assign different weights to feature information based on weights, the weight information of small targets is increased, thereby enhancing the neural network's ability to fuse feature information of small targets. After combining the attention module with the original YOLOv5 network model, the prediction results are output. This makes the improved model more sensitive to the effective feature information of small targets and suppresses irrelevant background and medium-to-large scale object feature weights, further improving the prediction accuracy for small targets.

[0049] In this embodiment of the invention, the road surface monitoring unit includes a detection frame determination module and a road surface monitoring module. The detection frame determination module constructs a detection frame based on the center of the image output by the image recognition unit, moves the detection frame based on the sampling period and flight speed of the UAV, and determines the position of the detection frame in the next frame image. The road surface monitoring module determines that if the number of vehicles in the detection frames of several adjacent frames is the same and the relative positions of the vehicles in the detection frames remain basically unchanged, then it is determined that there is a traffic jam in the current highway lane; otherwise, it is determined that there is no traffic jam in the current highway lane.

[0050] Figure 6 The flowchart of the highway traffic monitoring method based on UAV aerial photography provided in this embodiment of the invention includes the following steps:

[0051] S1. Real-time capture of road surface images of the designated highway lane;

[0052] S2. Detect vehicles in road surface images. The vehicle detection method in road surface images refers to the image recognition unit.

[0053] S3. Form a detection box in the middle area of ​​the road image. Based on whether the number of vehicles and their relative positions in the detection box change in several consecutive adjacent frames of the road image, if there is no change, the highway lane is congested. If the number of vehicles or their relative positions change, the highway lane is considered not congested.

[0054] The present invention has been described by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A highway traffic monitoring system based on unmanned aerial vehicle (UAV) aerial photography, characterized in that, The system includes: A camera is mounted on a gimbal, which is located on a drone. The drone flies at low speed over a designated highway lane and travels towards vehicles in that highway lane. The camera captures images of the road surface of a designated highway lane and sends them to the image recognition unit, which identifies vehicle targets in the road surface images. The road surface monitoring unit determines whether traffic congestion has occurred in the highway lane based on the number and relative positions of vehicles in several consecutive adjacent frames. The image recognition unit identifies vehicle targets in road surface images based on the YOLOv5 network structure; The YOLOv5 network architecture consists of a Backbone layer, a Neck layer, and a Head layer connected sequentially. The last layer of the Backbone layer contains an MSPPF module, which includes: SPPF1 and SPPF2 are connected in series; two Conv convolutional layers are connected in parallel with SPPF1 and SPPF2 respectively; the output of SPPF1 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer and SPPF2 respectively; the output of SPPF2 is fused with the output of the parallel convolutional layers and input into the Concat fusion layer for fusion; the Concat fusion layer is connected to the Neck layer. SPPF1 includes: a standard convolutional layer CBL, and three max pooling layers MaxPool connected in series with the standard convolutional layer CBL; the outputs of the standard convolutional layer CBL and the three max pooling layers MaxPool are fused through a Concat fusion layer and then input into another standard convolutional layer CBL. SPPF2 includes: a standard convolutional layer CBL, and three average pooling layers AvgPool connected in series with the standard convolutional layer CBL; the outputs of the standard convolutional layer CBL and the three average pooling layers AvgPool are fused through a Concat fusion layer and then input into another standard convolutional layer CBL.

2. The highway traffic monitoring system based on UAV aerial photography as described in claim 1, characterized in that, The Neck layer consists of an FPN structure and a PAN structure. An attention module CBAM is added at the fusion point of the FPN and PAN structures to introduce the feature image output from the Backbone layer.

3. The high-speed road traffic monitoring system based on UAV aerial photography as described in claim 2, characterized in that, An attention module CBAM is set at the end of the FPN structure's Concat fusion layer to introduce the feature image output from the Backbone layer.

4. The high-speed road traffic monitoring system based on UAV aerial photography as described in claim 1, characterized in that, The road surface monitoring unit includes: a detection frame determination module and a road surface monitoring module. The detection box determination module constructs a detection box based on the center of the image output by the image recognition unit, moves the detection box based on the sampling period and flight speed of the UAV, and then determines the position of the detection box in the next frame image. If the number of vehicles in the detection frame is the same in several consecutive adjacent frames and the relative positions between the vehicles remain basically unchanged, the road surface monitoring module determines that there is a traffic jam in the current highway lane; otherwise, it determines that there is no traffic jam in the current highway lane.

Images