Vision-based intelligent unmanned aerial vehicle single target tracking method and system
By improving the YOLOV5 and NanoTrack model combined with Kalman filtering trajectory prediction, the problem of environmental factors in drone target tracking is solved, the tracking accuracy and real-time performance are improved, and stability and continuity are achieved in complex environments.
Patent Information
- Application Number
- CN202411534916.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-31
Smart Images

Figure CN119048566A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a vision-based intelligent unmanned aerial vehicle single target tracking method and system. Background Art
[0002] With the development and increasing maturity of drone technology, drone technology has been widely used in recent years, covering many fields, including military, logistics, agriculture, film and television production, and security monitoring. In these applications, drone target tracking is one of the key technologies. However, when existing drone systems track targets, they are generally based on multi-target tracking, and only a few are single-target tracking, which also leads to differences in their application scenarios and data processing methods. From the perspective of drones, there are problems such as too large a field of view and too small a target, resulting in fewer pixels of ground targets in the drone camera and difficulty in extracting target features. In addition, during the tracking process, there will be environmental factors such as color, background, occlusion, etc., which make it easy to lose the target. Therefore, the use of advanced technology and reasonable tracking strategies to track the target for a long time, in real time, and stably is of great significance to promote the further development and application of drone technology.
[0003] The patent with application number CN202311732901.5 discloses a method for multi-target tracking of pedestrians based on the perspective of a drone. This patent is for multi-target tracking of pedestrians. By training and optimizing the detection and tracking network, it is used to solve the problem of low detection accuracy in complex scenes with high-altitude perspective, low resolution, and small-scale targets, which leads to easy failure in multi-target tracking of pedestrians. However, this method uses the detection and tracking network in a serial manner, which makes it too time-consuming to process an image and cannot be tracked in real time. The patent with application number CN202311544566.6 discloses a night tracking method, system and related devices based on a thermal infrared camera of a drone. This patent sends the image obtained by the thermal infrared camera on the drone to the ground station; then uses the Faster-RCNN detection model and the SiamRPN single target tracking model to process the received image to obtain the target motion trajectory information; finally, the reinforcement learning algorithm is used to learn the target trajectory information, so as to determine the flight path of the drone to track the target. This method uses a two-stage model to communicate with the ground station and detect the model. There are time-consuming calculations and communications, which will cause the tracking target to deviate from the final flight path of the drone and make it impossible to accurately track the target.
[0004] It can be seen that the existing drone target tracking technology has the following shortcomings: when processing target feature information, the evaluation criteria are often single, and the influence of environmental factors such as lighting changes, background complexity and target occlusion are not fully considered. This limitation makes these methods may not be accurate enough when determining the key state of the target, resulting in insufficient target tracking accuracy; some detection and tracking methods require complex calculation and analysis processes, which will cause processing delays and cannot provide real-time target location information. This delay makes it impossible for the system to accurately track the target in a dynamic environment, affecting the real-time and reliability of tracking.
[0005] Therefore, how to provide a vision-based intelligent UAV single target tracking method and system that can effectively improve the influence of environmental factors such as lighting changes, background complexity and target occlusion during UAV target tracking, improve the accuracy of UAV target tracking, and effectively simplify the calculation and analysis process during UAV target tracking, and truly achieve real-time and reliability of tracking is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0006] In view of this, the present invention proposes a vision-based intelligent UAV single target tracking method and system.
[0007] In order to achieve the above object, the present invention adopts the following technical solution: A vision-based intelligent UAV single target tracking method, comprising: Step 1: Obtain PTZ high-definition data and perform data preprocessing to obtain target calibration data and continuously moving target image data; Step 2: Based on the target calibration data, train the target detection model based on the improved YOLOV5 to obtain the target detection result; the improvement of YOLOV5 includes: using Focaler-IoU to reconstruct the IoU loss calculation method of YOLOV5; Step 3: Based on the continuously moving target image data, train the NanoTrack-based target tracking model, perform target tracking on the target detection results, and introduce Kalman filter trajectory prediction for auxiliary tracking.
[0008] Optionally, in step 2, the IoU loss calculation method of YOLOV5 is reconstructed using Focaler-IoU, specifically: Assume the coordinates of the true bounding box A are , the coordinates of the predicted bounding box B are , then the width of the intersection of A and B is , the height is , the intersection area is ; The area of A is , the area of B is , the area of the union of A and B is ; in, is the calculated value of Focaler-IoU; , is the threshold value.
[0009] Optionally, in step 2, the improvement to YOLOV5 further includes: adding a target detection head for detecting 4×4 pixel size targets to the head network of YOLOV5.
[0010] Optionally, in step 2, the improvement to YOLOV5 also includes: in the first convolutional layer network of YOLOV5, using a large step size to reduce the size of image data.
[0011] Optionally, in step 2, the improvement to YOLOV5 further includes: based on YOLOV5, introducing C_BIoU to make the target continuous and selectable.
[0012] Optionally, in step 3, Kalman filter trajectory prediction is introduced to assist tracking, specifically: According to the target confidence output by the target tracking model , using different matching methods for calculation, specifically: when When , the L2 paradigm is used to match the predicted track target frame and the tracked target frame, as follows: in, is the target confidence threshold; , are the width and height of the true bounding box respectively; are the coordinates of the true bounding box; , are the width and height of the predicted bounding box respectively; is the coordinate of the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; is the L2 paradigm threshold; when When , IoU is used for matching as follows: in, is the target confidence threshold; is the intersection area of the real bounding box and the predicted bounding box; is the union area of the true bounding box and the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; The loU threshold of the intersection-over-union ratio between the target box and the predicted box; when When , Kalman filter trajectory prediction is performed.
[0013] Optionally, after performing Kalman filter trajectory prediction, the method also includes: calling the gimbal to rotate and track the target according to the trajectory prediction result of the Kalman filter, and controlling the lens to perform real-time zoom processing according to the target information obtained by the target tracking model, so that the target remains in the center of the picture and is presented at a preset ratio.
[0014] Optionally, it also includes: counting the number of matching failures in the Kalman filter trajectory prediction, and when a threshold of the number of matching failures is reached, it is determined that the tracking target has failed, and the Kalman filter is initialized for auxiliary tracking.
[0015] The present invention also provides a vision-based intelligent unmanned aerial vehicle single target tracking system using a vision-based intelligent unmanned aerial vehicle single target tracking method, comprising: Data acquisition module: used to acquire high-definition data of the PTZ and perform data preprocessing to obtain target calibration data and continuously moving target image data; Target detection module: used to train the target detection model based on the improved YOLOV5 based on the target calibration data to obtain the target detection results; the improvement of YOLOV5 includes: using Focaler-IoU to reconstruct the IoU loss calculation method of YOLOV5; Target tracking module: It is used to train the NanoTrack-based target tracking model based on the continuously moving target image data, perform target tracking on the target detection results, and introduce Kalman filter trajectory prediction for auxiliary tracking.
[0016] It can be seen from the above technical solutions that compared with the prior art, the present invention proposes a vision-based intelligent unmanned aerial vehicle single target tracking method and system. By combining the YOLOV5 target detection model and the NanoTrack target tracking model, efficient target detection and tracking are achieved, overcoming the problem of poor real-time performance caused by serial processing in the prior art. In addition, by optimizing the head network, model calculation amount, target boundary regression box loss calculation of the YOLOV5 target detection model and introducing the target continuous selectability brought by C_BIoU, at the same time, on the basis of the NanoTrack target tracking model, Kalman filter trajectory prediction is used for auxiliary tracking, which truly realizes the continuity and stability of the target in a complex environment, enhances the long-term tracking capability of the target, and further improves the robustness and accuracy of tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0018] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION
[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] Embodiment 1: Embodiment 1 of the present invention discloses a vision-based intelligent unmanned aerial vehicle single target tracking method, such as Figure 1 As shown, including: Step 1: Use a PTZ HD camera to obtain PTZ HD data at different viewing angles and focal lengths, and perform data preprocessing to obtain target calibration data and continuously moving target image data.
[0021] Data preprocessing includes: using video segmentation tools to extract video data, decomposing it into 30 frames of image data per second, and removing blurred and unclear images of the target; using the autolabelme image automatic labeling tool to label the image to obtain target calibration data; and using the cropping program to crop the labeled image to obtain continuously moving target image data.
[0022] Step 2: Based on the target calibration data, train the target detection model based on the improved YOLOV5 to obtain the target detection result. The YOLOV5 target detection model is used to detect targets from the perspective of the drone, providing users with a choice of which target to track. Therefore, real-time detection must be achieved while ensuring its high accuracy. The YOLOV5 detection model is a one-stage detection model, which is faster than the two-stage detection model. Its structure includes three parts: backbone network, neck network and head network. However, from the perspective of the drone, there are problems such as too large field of view and too small target, which leads to fewer pixels of ground targets in the drone camera and difficulty in target feature extraction. The use of the YOLOV5 detection model is not ideal in this scenario. For this reason, the YOLOV5 detection model is improved. Among them, the improvement of YOLOV5 includes: using Focaler-IoU to reconstruct the IoU loss calculation method of YOLOV5.
[0023] The IoU loss calculation method of YOLOV5 is reconstructed using Focaler-IoU, specifically: Assume the coordinates of the true bounding box A are , the coordinates of the predicted bounding box B are , then the width of the intersection of A and B is , the height is , the intersection area is ; The area of A is , the area of B is , the area of the union of A and B is ; in, is the calculated value of Focaler-IoU; , is the threshold value, and the range is .
[0024] Improvements to YOLOV5 also include: optimizing the head network, adding a very small target detection head for detecting 4×4 pixel targets to the head network of YOLOV5, changing the number of detection heads from three to four.
[0025] Improvements to YOLOV5 also include: reducing the amount of model calculations. In the first convolutional layer network of YOLOV5, a large step size is used to reduce the size of image data, so that the model can achieve real-time calculation.
[0026] Improvements to YOLOV5 also include: Based on YOLOV5, C_BIoU is introduced to make the target continuous and selectable. After the target detection model detects the target position and target category information, it is available for users to click. However, in the process of selecting the target, since the target detected by the target detection model changes each time, there may be a situation where the corresponding target cannot be found in the background after clicking the target on the screen. For this reason, the C_BIoU method is introduced to cooperate with the target detection model to make the target continuous and selectable. C_BIoU (also known as C-BIoU Tracker) is a multi-target tracking method that does not consider the motion state of the target. According to the information of the target detection model, the search area is expanded to associate the target of the current frame with the previous frame, so that the target can be continuous and not lost. And when the user clicks the target, in order to ensure real-time performance, the target detection model is closed until the tracked target is lost.
[0027] Step 3: Based on the continuously moving target image data, train the NanoTrack-based target tracking model, perform target tracking on the target detection results, and introduce Kalman filter trajectory prediction for auxiliary tracking.
[0028] The NanoTrack target tracking model takes a given target as a target template and performs a range search on subsequent image frames to find the area that best matches the target template. In addition, the model is a lightweight tracking network with very small network size and parameters, which is very suitable for real-time tracking. When the target tracking model receives the tracking target information sent by the user, it will extract important features of the current target as a template and store it; then, the image of the next frame is searched and matched with the template to obtain the best target position and target confidence. Since the target tracking model searches and matches by expanding the area at the center of the target template, there may be similar targets or targets may be obscured, resulting in the loss of the real target. For this reason, Kalman filter (KF) trajectory prediction is introduced for auxiliary tracking.
[0029] Kalman filter trajectory prediction is introduced to assist tracking, specifically: According to the target confidence output by the target tracking model , using different matching methods for calculation, specifically: when When , the L2 paradigm is used to match the predicted track target frame and the tracked target frame, as follows: in, is the target confidence threshold; , are the width and height of the true bounding box respectively; are the coordinates of the true bounding box; , are the width and height of the predicted bounding box respectively; is the coordinate of the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; is the L2 paradigm threshold; when , use IoU for matching, as follows: in, is the target confidence threshold; is the intersection area of the real bounding box and the predicted bounding box; is the union area of the true bounding box and the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; The loU threshold of the intersection-over-union ratio between the target box and the predicted box; when When , Kalman filter trajectory prediction is performed.
[0030] After performing Kalman filter trajectory prediction, the process also includes: calling the gimbal to rotate and track the target based on the trajectory prediction result of the Kalman filter, and controlling the lens to perform real-time zoom processing based on the target information obtained by the target tracking model, so that the target remains in the center of the picture and is presented at a preset ratio.
[0031] It also includes: counting the number of matching failures in the Kalman filter trajectory prediction. When the matching failure number threshold is reached, it is determined that the tracking target has failed, and the Kalman filter is initialized for auxiliary tracking, that is, the Kalman filter is initialized for the received tracking target information, and the Kalman filter trajectory prediction is continued for auxiliary tracking.
[0032] Embodiment 2: Embodiment 2 of the present invention discloses a vision-based intelligent unmanned aerial vehicle single target tracking system using a vision-based intelligent unmanned aerial vehicle single target tracking method, comprising: Data acquisition module: used to acquire high-definition data of the PTZ and perform data preprocessing to obtain target calibration data and continuously moving target image data; Target detection module: used to train the target detection model based on the improved YOLOV5 based on the target calibration data to obtain the target detection results; the improvement of YOLOV5 includes: using Focaler-IoU to reconstruct the IoU loss calculation method of YOLOV5; Target tracking module: It is used to train the NanoTrack-based target tracking model based on the continuously moving target image data, perform target tracking on the target detection results, and introduce Kalman filter trajectory prediction for auxiliary tracking.
[0033] The embodiment of the present invention discloses a vision-based intelligent unmanned aerial vehicle single target tracking method and system. By combining the YOLOV5 target detection model and the NanoTrack target tracking model, efficient target detection and tracking are achieved, overcoming the problem of poor real-time performance caused by serial processing in the prior art. In addition, by optimizing the head network, model calculation amount, target boundary regression box loss calculation of the YOLOV5 target detection model and introducing the target continuous selectable neutrality brought by C_BIoU, at the same time, on the basis of the NanoTrack target tracking model, Kalman filter trajectory prediction is used for auxiliary tracking, which truly realizes the continuity and stability of the target in a complex environment, enhances the long-term tracking capability of the target, and further improves the robustness and accuracy of tracking.
[0034] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0035] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A vision-based intelligent UAV single target tracking method, characterized in that: include: Step 1: Obtain PTZ high-definition data and perform data preprocessing to obtain target calibration data and continuously moving target image data; Step 2: Based on the target calibration data, train a target detection model based on the improved YOLOV5 to obtain a target detection result; wherein the improvement of the YOLOV5 includes: reconstructing the IoU loss calculation method of the YOLOV5 using Focaler-IoU; Step 3: Based on the continuously moving target image data, a target tracking model based on NanoTrack is trained, target tracking is performed on the target detection results, and Kalman filter trajectory prediction is introduced for auxiliary tracking.
2. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 1, characterized in that: In step 2, the IoU loss calculation method of YOLOV5 is reconstructed using Focaler-IoU, specifically: Assume the coordinates of the true bounding box A are , the coordinates of the predicted bounding box B are , then the width of the intersection of A and B is , the height is , the intersection area is ; The area of A is , the area of B is , the area of the union of A and B is ; in, is the calculated value of the Focaler-IoU; , is the threshold value.
3. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 1, characterized in that: In step 2, the improvement of the YOLOV5 also includes: adding a target detection head for detecting 4×4 pixel size targets to the head network of the YOLOV5.
4. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 1, characterized in that: In step 2, the improvement of the YOLOV5 further includes: in the first convolutional layer network of the YOLOV5, using a large step size to reduce the size of image data.
5. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 1, characterized in that: In step 2, the improvement of the YOLOV5 further includes: on the basis of the YOLOV5, introducing C_BIoU to make the target continuous and selectable.
6. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 1, characterized in that: In step 3, Kalman filter trajectory prediction is introduced to assist tracking, specifically: According to the target confidence output by the target tracking model , using different matching methods for calculation, specifically: when When , the L2 paradigm is used to match the predicted track target frame and the tracked target frame, as follows: in, is the target confidence threshold; , are the width and height of the true bounding box respectively; are the coordinates of the true bounding box; , are the width and height of the predicted bounding box respectively; is the coordinate of the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; is the L2 paradigm threshold; when When , IoU is used for matching as follows: in, is the target confidence threshold; is the intersection area of the real bounding box and the predicted bounding box; is the union area of the true bounding box and the predicted bounding box; when When , it indicates that the target match is clear, and the Kalman filter is called to perform trajectory prediction and update operations, otherwise the Kalman filter trajectory prediction is performed; The loU threshold of the intersection of the target box and the predicted box; when When , Kalman filter trajectory prediction is performed.
7. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 6, characterized in that: After performing the Kalman filter trajectory prediction, it also includes: calling the gimbal to rotate and track the target according to the trajectory prediction result of the Kalman filter, and controlling the lens to perform real-time zoom processing according to the target information obtained by the target tracking model, so that the target remains in the center of the picture and is presented at a preset ratio.
8. The method for tracking a single target of an intelligent unmanned aerial vehicle based on vision according to claim 6, characterized in that: Also includes: The number of matching failures in the Kalman filter trajectory prediction is counted. When the matching failure number threshold is reached, it is determined that the tracking target has failed, and the Kalman filter is initialized for auxiliary tracking.
9. A vision-based intelligent unmanned aerial vehicle single target tracking system using the vision-based intelligent unmanned aerial vehicle single target tracking method according to any one of claims 1 to 8, characterized in that: include: Data acquisition module: used to acquire high-definition data of the PTZ and perform data preprocessing to obtain target calibration data and continuously moving target image data; Target detection module: used to train a target detection model based on improved YOLOV5 based on the target calibration data to obtain a target detection result; wherein the improvement of YOLOV5 includes: reconstructing the IoU loss calculation method of YOLOV5 using Focaler-IoU; Target tracking module: used to train a NanoTrack-based target tracking model based on the continuously moving target image data, perform target tracking on the target detection results, and introduce Kalman filter trajectory prediction for auxiliary tracking.
Citation Information
Patent Citations
Unmanned aerial vehicle video vehicle speed correction method based on multi-target tracking framework
CN110738690A
Vision-based target tracking method and system in intelligent network connection bus scene
CN113724293A
Aircraft multi-target tracking method based on improved YOLOV5 algorithm
CN114529799A
Pedestrian tracking method based on improved YOLOv5 and DeepSORT
CN114724082A
Unmanned aerial vehicle target tracking method and simulation system based on deep learning algorithm
CN116309727A
