Non-motor vehicle illegal behavior detection system and method based on unmanned aerial vehicle aerial photography

Abstract

The invention relates to the technical field of traffic control, in particular to a non-motor vehicle illegal behavior detection system and method based on unmanned aerial vehicle aerial photography, and aims to optimize and integrate flight control, target detection, image processing and information integration and transmission processes of unmanned aerial vehicle detection in an application scene so as to realize detection of non-motor vehicle illegal behaviors. In the flight control part, a mode of multi-sensor fusion navigation and optimization and debugging of a flight control algorithm is adopted to achieve the purpose of flight stability augmentation. In the target detection part, a YOLOV8 algorithm is used and improved, a FaceNet face recognition algorithm and a CRNN license plate recognition algorithm are fused, and non-motor vehicle illegal behavior categories and key information of law breakers in aerial images are detected. In the image processing part, a proper image processing method is selected according to different types of image information. In the information integration and transmission part, the extracted picture information and text information of illegal behaviors and law breakers are integrated and transmitted to a ground terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and in particular to a non-motor vehicle illegal behavior detection system and method based on unmanned aerial vehicle aerial photography. Background Art

[0002] At present, the mainstream non-motor vehicle violation detection mainly relies on two methods: manual inspection and fixed road monitoring. Traditional manual inspection is limited by the number of inspectors and the inspection coverage, and often cannot conduct comprehensive and timely supervision of all traffic violations. Especially during peak traffic hours or on a wide range of roads, limited human resources make it difficult to take care of every intersection and road section, resulting in many violations not being discovered and handled in a timely manner. Although the traditional fixed road monitoring system can assist in inspection to a certain extent, its coverage and monitoring angle are limited. Due to the fixed installation position of the camera, they can often only cover a limited section of the road, resulting in a large number of monitoring blind spots. In some narrow alleys, remote sections of the road or areas with more obstructions, it is difficult for cameras to capture traffic violations in real time, which makes it easy for offenders to avoid the monitoring area and escape supervision.

[0003] In addition, traditional detection methods are relatively lacking in intelligence in terms of technical application, and often rely on manual analysis and judgment. This method is not only inefficient, but also prone to underreporting violations due to human negligence, and is unable to flexibly respond to the increasingly complex and changing traffic environment. For example, in the face of highly mobile non-motorized vehicles and diverse forms of traffic violations, traditional fixed monitoring and manual inspections are difficult to adapt. These factors combined have led to the current non-motorized vehicle violation detection methods being significantly insufficient in terms of coverage, real-time performance, and intelligence. Summary of the invention

[0004] The purpose of the present invention is to provide a non-motor vehicle violation detection system and method based on drone aerial photography, which detects non-motor vehicle traffic violations based on drone aerial photography technology, and monitors, records and reminds related violations.

[0005] To achieve the above object, the present invention provides a non-motor vehicle illegal behavior detection method based on drone aerial photography, comprising the following steps:

[0006] Step 1: Design and optimize the UAV flight control so that the UAV can rely on sensor-assisted flight and set a route for autonomous flight;

[0007] Step 2: Capture ground traffic conditions through an onboard camera;

[0008] Step 3: Integrate the Yolov8, FaceNet, CRNN algorithms and image processing methods on the airborne development board to process the images of non-motor vehicle traffic violations captured by the airborne camera, and then detect and extract information;

[0009] Step 4: Integrate the extracted image information and text information of the violations and violators, and transmit them to the ground end; Use digital image transmission to return the images captured by the drone to the ground end in real time;

[0010] Step 5: Evaluate the detection results of non-motor vehicle violations captured by the drone using confusion matrix, F1 score, accuracy-confidence relationship curve, accuracy-recall curve, face recognition accuracy, and license plate information extraction accuracy.

[0011] Optionally, in Step 1, three navigation methods, namely lidar navigation, GPS navigation, and inertial navigation, are integrated for navigation. In a densely built area at low altitude with good lighting, lidar positioning navigation is used for flight; In an open area at high altitude with poor lighting, a combination of GPS navigation and inertial navigation is used for flight; And the flight attitude of the drone is controlled by debugging the PID algorithm.

[0012] Optionally, in Step 3, the defined non-motor vehicle violations are detected by the Yolov8 algorithm, including not wearing a helmet and overloading. Then, the Yolov8 algorithm is used to detect the faces and license plates in the violation pictures, and the overall violation pictures, violator face pictures, and violator license plate pictures are intercepted;

[0013] For the overall violation pictures, two image processing methods, namely image correction and image enhancement, are used for processing;

[0014] For license plate images, edge detection and feature extraction methods are used to process blurred license plates;

[0015] For face images, the GFPGAN algorithm is used to repair blurred faces.

[0016] Optionally, in Step 4, the process of integrating the extracted image information and text information of the violations and violators is specifically to integrate the intercepted overall violation pictures, violator face pictures, violator license plate pictures, text information recognized from faces, text information of license plates, and time information of the occurrence of violations into a folder on the airborne development board for storage.

[0017] Optionally, during the process of debugging the PID algorithm, by changing the PID values, test flights are carried out to find that the horizontal plane error does not exceed 2 cm within 1 minute of hovering.

[0018] Furthermore, the present invention also proposes a non-motor vehicle illegal behavior detection system based on UAV aerial photography for the non-motor vehicle illegal behavior detection method described above, including a UAV, an airborne camera, a lidar module, an inertial navigation module, a GPS module, an airborne development board, and a ground terminal. The airborne camera, the lidar module, the inertial navigation module, the GPS module, and the airborne development board are all installed on the UAV, and the UAV, the airborne camera, the lidar module, the inertial navigation module, the GPS module, the airborne development board, and the ground terminal are electrically connected;

[0019] The UAV is a quadcopter UAV using the PX4 2.4.8 open-source flight controller; the airborne camera is a high-definition variable-focus camera; the airborne development board is a JETSON development board.

[0020] The present invention provides a non-motor vehicle illegal behavior detection system and method based on UAV aerial photography, which optimally integrates the processes of flight control, target detection, image processing, and information integration and transmission of UAV detection for the application scenario to achieve the detection of non-motor vehicle illegal behaviors. In the flight control part, the method of multi-sensor fusion navigation and optimization and debugging of the flight control algorithm is adopted to achieve the purpose of stable flight. In the target detection part, the YOLOV8 algorithm is used and improved, and the FaceNet face recognition algorithm and the CRNN license plate recognition algorithm are fused to detect the categories of non-motor vehicle illegal behaviors and the key information of the violators in the aerial photography images. In the image processing part, appropriate image processing methods are selected according to different types of image information. In the information integration and transmission part, the extracted illegal behavior and the picture information and text information of the violator are integrated and transmitted to the ground terminal. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0022] Figure 1 It is a schematic flow chart of the steps of a non-motor vehicle illegal behavior detection method based on UAV aerial photography of the present invention.

[0023] Figure 2 It is a confusion matrix diagram of the method of the present invention.

[0024] Figure 3 It is an F1 score diagram of the method of the present invention.

[0025] Figure 4It is a curve graph showing the relationship between the accuracy rate and confidence of the method of the present invention.

[0026] Figure 5 It is a curve graph of the accuracy rate and recall rate of the method of the present invention.

[0027] Figure 6 It is a schematic diagram of the actual measurement effect of the road in a specific embodiment of the present invention. Detailed implementation manners

[0028] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] Please refer to Figure 1 , the present invention provides a non-motor vehicle illegal behavior detection method based on UAV aerial photography, including the following steps:

[0030] S1: Design and optimize the UAV flight control so that the UAV can fly stably relying on sensors and fly autonomously along a set route;

[0031] S2: Capture the ground traffic conditions through an on-board camera;

[0032] S3: Integrate the Yolov8, FaceNet, CRNN algorithms and image processing methods on the on-board development board to detect and extract information after processing the non-motor vehicle traffic illegal behavior images captured by the on-board camera;

[0033] S4: Integrate the captured illegal behavior and the picture information and text information of the violator, and transmit them to the ground end; use digital video transmission to return the pictures taken by the UAV to the ground end in real time;

[0034] S5: Evaluate the detection results of the non-motor vehicle illegal behavior detected by the UAV aerial photography by using confusion matrix, F1 score, the curve of the relationship between accuracy rate and confidence, the curve of accuracy rate and recall rate, and the face recognition accuracy rate and license plate information extraction accuracy rate.

[0035] Specifically, in step S1, the UAV flight control is designed and optimized so that the UAV can fly stably relying on sensors and fly autonomously along a set route. The specific processing method is to perform high-precision stable flight in areas with dense obstacles through lidar, perform long-distance flight in open areas through inertial navigation and GPS, and fly autonomously along a set route relying on the open-source ground station QGC and the self-set program code.

[0036] Among them, the method of three-sensor fusion navigation and optimizing and debugging the flight control algorithm is adopted. The fusion of lidar navigation, GPS navigation, and inertial navigation is utilized to meet the requirements of different environmental conditions and application scenarios. In the low-altitude, densely built-up areas with good lighting, lidar positioning navigation is used for flight. In the high-altitude, open areas with poor lighting, the combination of GPS and inertial navigation is used for flight. By debugging the PID algorithm, the UAV can achieve rapid dynamic response, without overshoot or deficiency.

[0037] Furthermore, the specific steps of debugging the PID algorithm include: by changing the values of PID and conducting test flights, find that the horizontal plane error within 1 minute of hovering does not exceed 2 cm, which means the debugging of the PID algorithm is completed.

[0038] In step S2, the ground traffic situation is captured by the on-board camera. The specific method is to connect the on-board camera to the on-board development board, obtain and store the video captured by the on-board camera based on Opencv and call it to the detection and recognition algorithms for information acquisition.

[0039] In step S3, the Yolov8, FaceNet, CRNN algorithms, and image processing methods are integrated. The specific method is to detect the defined non-motor vehicle illegal behaviors through the Yolov8 algorithm, including not wearing a helmet and overloading, and then use the Yolov8 algorithm to detect the faces and license plates in the illegal behavior pictures, and cut out the overall illegal behavior pictures, the pictures of the violators' faces, and the pictures of the violators' license plates. For the overall illegal behavior pictures, two image processing methods, image correction and image enhancement, are used to improve the image quality. Among them, the image correction part includes geometric correction and color correction, and the image enhancement part includes sharpening, denoising, and contrast enhancement. For license plate images, edge detection and feature extraction methods are used to process blurred license plates. For face images, the GFPGAN algorithm is used to repair blurred faces.

[0040] In step S4, digital image transmission is used to return the pictures taken by the UAV to the ground end in real time. The specific method is to integrate the intercepted overall illegal behavior pictures, the pictures of the violators' faces, the pictures of the violators' license plates, the text information recognized by face recognition, the text information of the license plates, and the time information of the occurrence of illegal behaviors into a folder on the on-board development board for storage.

[0041] In step S5, the confusion matrix, F1 score, accuracy-confidence relationship curve, accuracy-recall curve, as well as the face recognition accuracy and license plate information extraction accuracy are used to evaluate the detection results of non-motor vehicle illegal behaviors photographed by the UAV. As Figures 2 to 5As shown, there is a confusion matrix diagram. The vertical axis represents the true detected objects, and the horizontal axis represents the predicted detected objects. The detection accuracy can be seen through the confusion matrix diagram. The F1 score can show the performance in terms of balanced precision and recall, and its value range is between 0 and 1. 1 represents the best performance, while 0 represents the worst performance. The accuracy-confidence relationship curve reflects the performance of the detector at different confidence levels. The accuracy-recall curve provides a more detailed insight into the performance of the model under different tasks. The face recognition accuracy and license plate information extraction accuracy can describe the probability of successful recognition of faces and license plate information.

[0042] Furthermore, the present invention also proposes a non-motor vehicle illegal behavior detection system based on UAV aerial photography for the non-motor vehicle illegal behavior detection method based on UAV aerial photography, including a UAV, an airborne camera, a lidar module, an inertial navigation module, a GPS module, an airborne development board, and a ground end. The airborne camera, the lidar module, the inertial navigation module, the GPS module, and the airborne development board are all installed on the UAV, and the UAV, the airborne camera, the lidar module, the inertial navigation module, the GPS module, the airborne development board, and the ground end are electrically connected.

[0043] The UAV is a quadcopter UAV using the PX4 2.4.8 open-source flight controller. The airborne camera is a high-definition variable-focus camera. The airborne development board is a JETSON development board.

[0044] In addition, to verify the effectiveness of the system proposed by the present invention, flight tests are carried out in a test site and actual tests are carried out on a typical measured road section to analyze the actual effect of the system.

[0045] Experiment 1 is for the UAV to perform fixed-point hovering in the test site. After 10 actual tests, the average horizontal plane error within 1 minute of hovering does not exceed 2 cm. Table 1 shows the data obtained from the above ten experiments.

[0046] Table 1 Fixed-point hovering data

[0047]

[0048] Experiment 2 is for autonomous flight according to a pre-set route. 24 points (represented by detection codes) are set in three-dimensional space, and the set route traverses the 24 points. If the detection code information captured by the camera is obtained, it is considered that the point traversal is successful. Table 2 shows the test results that all ten experiments on the same route are successful after the above ten experiments.

[0049] Table 2 Autonomous flight data

[0050]

[0051] Figure 6 It shows the actual road test results in the non-motor vehicle illegal behavior detection system based on drone aerial photography. It can be seen that the required non-motor vehicle illegal behaviors can be detected relatively accurately, and the flight is stable and meets the image requirements. The average detection accuracy of various non-motor vehicle traffic illegal behaviors in the present invention is above 80%.

[0052] In summary, the present invention has the following beneficial effects:

[0053] 1. By constructing a three-sensor fusion navigation and optimizing and debugging the flight control algorithm, compared with the existing related technologies, the present invention adaptively switches different sensor combinations under different environmental conditions and application scenarios, so that each sensor combination can obtain the best drone stabilization and autonomous flight effects under the current conditions.

[0054] 2. By integrating the Yolov8, FaceNet, CRNN algorithms and image processing methods, the present invention can actually solve this problem. According to the effects obtained according to the actual problem needs, the Yolov8 algorithm is used to distinguish various non-motor vehicle traffic illegal behaviors and intercept the corresponding pictures; the FaceNet algorithm is used to identify the face information; the CRNN algorithm is used to convert the license plate information in the pictures into text information for subsequent information statistics and screening. In view of the limitations of the shooting equipment cost and weight during the aerial photography process, methods for various types of pictures are adopted to optimize the images to obtain an image that meets the requirements.

[0055] 3. The present invention integrates and extracts the picture information and text information of illegal behaviors and violators, which can reduce the time of manual screening and the amount of information transmission. Integrate the illegal-related information of the corresponding violators into a folder for classified storage, reducing the subsequent manual information processing time; by only transmitting the picture information and text information, compared with transmitting videos, it reduces the amount of information transmission and the real-time requirements for information and can obtain more accurate information.

[0056] The above-disclosed is only a preferred embodiment of the present invention. Of course, the scope of the rights of the present invention cannot be limited by this. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A method for detecting non-motor vehicle illegal behaviors based on UAV aerial photography, characterized in that, Including the following steps: Step 1: Design and optimize the flight control of the drone so that the drone can fly stably relying on sensors and fly autonomously along a set route; Step 2: Capture the ground traffic situation through the on-board camera; Step 3: Integrate the Yolov8, FaceNet, CRNN algorithms and image processing methods on the on-board development board to detect and extract information after processing the images of non-motor vehicle traffic violations captured by the on-board camera; Step 4: Integrate the extracted violation and offender's picture information and text information and transmit it to the ground end; Use digital video transmission to return the images captured by the drone to the ground end in real time; Step 5: Evaluate the detection results of non-motor vehicle violations captured by the drone using confusion matrix, F1 score, accuracy-confidence relationship curve, accuracy-recall curve, and face recognition accuracy and license plate information extraction accuracy.

2. The method for detecting non-motor vehicle violations based on drone aerial photography according to claim 1, characterized in that, In step 1, three navigation methods of lidar navigation, GPS navigation and inertial navigation are integrated for navigation. In a densely built area with low altitude and good lighting, lidar positioning navigation is used for flight; in an open area with high altitude and poor lighting, GPS navigation and inertial navigation are combined for flight; and the flight attitude of the drone is controlled by debugging the PID algorithm.

3. The method for detecting non-motor vehicle violations based on drone aerial photography according to claim 2, characterized in that, In step 3, the defined non-motor vehicle violations, including not wearing a helmet and overloading, are detected by the Yolov8 algorithm, and then the face and license plate in the violation picture are detected by the Yolov8 algorithm, and the overall violation picture, the offender's face picture, and the offender's license plate picture are intercepted; For the overall violation picture, two image processing methods of image correction and image enhancement are used for processing; For the license plate image, edge detection and feature extraction methods are used to process the blurred license plate; For the face image, the GFPGAN algorithm is used to repair the blurred face.

4. The method for detecting non-motor vehicle violations based on drone aerial photography according to claim 3, characterized in that, In step 4, the process of integrating the extracted violation and offender's picture information and text information is specifically to integrate the intercepted overall violation picture, the offender's face picture, the offender's license plate picture, the text information recognized by face recognition, the text information of the license plate, and the time information of the violation occurrence into a folder on the on-board development board for storage.

5. The method for detecting non-motor vehicle violations based on drone aerial photography according to claim 2, characterized in that, During the process of debugging the PID algorithm, by changing the values of PID, conduct test flights to find that the horizontal error within 1 minute of hovering does not exceed 2 cm.

6. A system for detecting non-motor vehicle violations based on drone aerial photography, used for the method for detecting non-motor vehicle violations based on drone aerial photography according to any one of claims 1 to 5, characterized in that, It includes a drone, an airborne camera, a lidar module, an inertial navigation module, a GPS module, an airborne development board and a ground terminal. The airborne camera, the lidar module, the inertial navigation module, the GPS module and the airborne development board are all installed on the drone, and the drone, the airborne camera, the lidar module, the inertial navigation module, the GPS module, the airborne development board and the ground terminal are electrically connected to each other; The drone is a quadcopter drone that uses the PX4 2.4.8 open-source flight controller; the airborne camera is a high-definition zoom camera; the airborne development board is a JETSON development board.

Images