A visual-based intelligent unmanned aerial vehicle single-target tracking method and system

Abstract

The application discloses a kind of intelligent unmanned aerial vehicle single target tracking method and system based on vision, applied to image processing technical field, comprising: obtaining gimbal high-definition data, and carrying out data preprocessing, obtain target calibration data and the target image data of continuous motion;Based on target calibration data, train the target detection model based on improved YOLOV5, obtain target detection result;Wherein, the improvement of YOLOV5 includes: reconstruct the IoU loss calculation mode of YOLOV5 using Focaler-IoU;Based on the target image data of continuous motion, train the target tracking model based on NanoTrack, track target to target detection result, and introduce Kalman filter trajectory prediction to assist tracking.The application effectively improves the real-time, accuracy and stability of unmanned aerial vehicle target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an intelligent unmanned aerial vehicle single target tracking method and system based on vision. BACKGROUND

[0002] With the development and maturation of unmanned aerial vehicle technology, unmanned aerial vehicle technology has been widely applied in recent years, covering multiple fields, including military, logistics, agriculture, film production and security monitoring. In these applications, target tracking of unmanned aerial vehicles is one of the key technologies. However, existing unmanned aerial vehicle systems generally track multiple targets when tracking targets, and only a few track single targets, which also leads to differences in application scenarios and data processing methods. Under the perspective of an unmanned aerial vehicle, there is a problem of too large a field of view and too small a target, which results in fewer pixels of the ground target in the unmanned aerial vehicle camera and difficulty in extracting target features. In addition, during tracking, environmental factors such as color, background and occlusion may affect the target and make it easy to lose track. Therefore, it is of great significance to promote the further development and application of unmanned aerial vehicle technology by using advanced technology and reasonable tracking strategies to track targets for a long time, in real time and stably.

[0003] The patent with the application number CN202311732901.5 discloses a pedestrian multi-target tracking method based on the perspective of an unmanned aerial vehicle. This patent tracks multiple targets of pedestrians by using a detection and tracking network optimized through training, to solve the problem of easy failure in pedestrian multi-target tracking caused by low detection accuracy in the complex scene of high-altitude perspective, low resolution and small-scale targets. However, this method uses a serial method of detection and tracking network, which results in a large time consumption when processing an image, and cannot perform real-time tracking. The patent with the application number CN202311544566.6 discloses a night tracking method and system based on an unmanned aerial vehicle thermal infrared camera and related devices. This patent sends images obtained by a thermal infrared camera on an unmanned aerial vehicle to a ground station, then processes the received images using a Faster-RCNN detection model and a SiamRPN single target tracking model to obtain target motion trajectory information, and finally uses a reinforcement learning algorithm to learn the target trajectory information to determine the flight path of the unmanned aerial vehicle to track the target. This method uses communication between the unmanned aerial vehicle and the ground station and a two-stage detection model, which results in time consumption in calculation and communication, and causes deviation of the final flight path of the unmanned aerial vehicle from the tracked target, and cannot accurately track the target.

[0004] It can be seen that the existing unmanned aerial vehicle target tracking technology has the following shortcomings: when processing target feature information, the evaluation standard is often single, and the influence of environmental factors such as light changes, background complexity and target occlusion is not fully considered. This limitation makes these methods not accurate enough when determining the key state of the target, resulting in insufficient accuracy of target tracking; some detection and tracking methods need to go through a complex calculation and analysis process, which will cause processing delay and cannot provide real-time target position information. This delay makes the system unable to accurately track the target in a dynamic environment, affecting the real-time and reliability of tracking.

[0005] Therefore, how to provide an intelligent unmanned aerial vehicle single target tracking method and system based on vision which can effectively improve the influence of environmental factors such as light changes, background complexity and target occlusion in the unmanned aerial vehicle target tracking process, improve the accuracy of unmanned aerial vehicle target tracking, effectively simplify the calculation and analysis process in the unmanned aerial vehicle target tracking process, and truly realize the real-time and reliability of tracking is a problem that those skilled in the art need to solve. SUMMARY

[0006] Therefore, the present application provides an intelligent unmanned aerial vehicle single target tracking method and system based on vision.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] An intelligent unmanned aerial vehicle single target tracking method based on vision, comprising:

[0009] Step 1: Obtain gimbal high-definition data and perform data preprocessing to obtain target calibration data and continuous motion target image data;

[0010] Step 2: Based on the target calibration data, train a target detection model based on improved YOLOV5 to obtain a target detection result; wherein the improvement of YOLOV5 includes: reconstructing the IoU loss calculation method of YOLOV5 using Focaler-IoU;

[0011] Step 3: Based on the continuous motion target image data, train a target tracking model based on NanoTrack to track the target detection result, and introduce Kalman filter trajectory prediction for auxiliary tracking.

[0012] Optionally, in step 2, the IoU loss calculation method of YOLOV5 is reconstructed using Focaler-IoU, specifically:

[0013] Let the coordinates of the real bounding box A be , the coordinates of the predicted bounding box B be , the width of the intersection of A and B be , and the height be , the intersection area is ; the area of A is , the area of B is , the union area of A and B is ;

[0014]

[0015]

[0016]

[0017] wherein, is the calculated value of Focaler-IoU; , is a threshold value.

[0018] Optionally, in step 2, the improvement of YOLOV5 further includes: adding a target detection head for detecting 4x4 pixel size targets in the head network of YOLOV5.

[0019] Optionally, in step 2, the improvement of YOLOV5 further includes: using a large stride to reduce the image data size in the first convolutional layer network of YOLOV5.

[0020] Optionally, in step 2, the improvement of YOLOV5 further includes: introducing C_BIoU to make the target continuous and selectable based on YOLOV5.

[0021] Optionally, in step 3, Kalman filter trajectory prediction is introduced for auxiliary tracking, specifically:

[0022] According to the target confidence output by the target tracking model , different matching methods are used for calculation, specifically:

[0023] When , the L2 norm matching is used to match the predicted track target box and the tracking target box, as follows:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] wherein, is a target confidence threshold value; , are width and height of the real bounding box respectively; is a coordinate of the real bounding box; , are width and height of the predicted bounding box respectively; is a coordinate of the predicted bounding box; when , it indicates that the target matching is clear, and Kalman filtering is called for trajectory prediction and updating operation, otherwise Kalman filtering trajectory prediction is performed; is an L2 norm threshold value;

[0038] when , IoU is used for matching, as follows:

[0039]

[0040] wherein, is a target confidence threshold value; is an intersection area of the real bounding box and the predicted bounding box; is a union area of the real bounding box and the predicted bounding box; when , it indicates that the target matching is clear, and Kalman filtering is called for trajectory prediction and updating operation, otherwise Kalman filtering trajectory prediction is performed; is an intersection and union ratio loU threshold value of the target box and the predicted box;

[0041] when , Kalman filtering trajectory prediction is performed.

[0042] Optionally, after Kalman filtering trajectory prediction, it further comprises: calling a pan-tilt rotation to track the target according to the trajectory prediction result of Kalman filtering, and controlling the lens to perform real-time zoom processing according to the target information obtained by the target tracking model, so that the target is kept in the center of the picture and presented at a preset scale.

[0043] Optionally, the method further comprises: counting the number of matching failures in the Kalman filter trajectory prediction, and when the number of matching failures reaches a threshold, determining that the tracking target is invalid, and initializing the Kalman filter for auxiliary tracking.

[0044] The application further provides a visual-based intelligent unmanned aerial vehicle single-target tracking system using the visual-based intelligent unmanned aerial vehicle single-target tracking method.

[0045] The data acquisition module is configured to acquire gimbal high-definition data, and perform data preprocessing to obtain target calibration data and continuously moving target image data.

[0046] The target detection module is configured to train a target detection model based on improved YOLOV5 based on the target calibration data, to obtain a target detection result, wherein the improvement of YOLOV5 comprises reconstructing the IoU loss calculation method of YOLOV5 using Focaler-IoU.

[0047] The target tracking module is configured to train a target tracking model based on NanoTrack based on the continuously moving target image data, to perform target tracking on the target detection result, and to introduce Kalman filter trajectory prediction for auxiliary tracking.

[0048] Compared with the prior art, the application provides a visual-based intelligent unmanned aerial vehicle single-target tracking method and system. Through the combination of the YOLOV5 target detection model and the NanoTrack target tracking model, efficient target detection and tracking are realized, and the problem of poor real-time performance caused by serial processing in the prior art is overcome. In addition, through the optimization of the YOLOV5 target detection model in the head network, model calculation amount, and target boundary regression frame loss calculation, and the introduction of C_BIoU for target continuous selectivity, and based on the NanoTrack target tracking model, Kalman filter trajectory prediction is used for auxiliary tracking, which truly realizes the continuity and stability of the target in a complex environment, enhances the long-time tracking ability of the target, and further improves the robustness and accuracy of the tracking. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0050] Figure 1 The method flowchart of the application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0052] Embodiment 1

[0053] Embodiment 1 of the present application discloses a visual-based intelligent unmanned aerial vehicle single-target tracking method, as shown in the following figure, which comprises the following steps: Figure 1

[0054] Step 1: Obtain gimbal high-definition data at different angles of view and different focal lengths by using a gimbal high-definition camera, and perform data preprocessing to obtain target calibration data and continuously moving target image data.

[0055] The data preprocessing comprises: extracting video data by using a video segmentation tool, decomposing the video data into 30 frames of image data per second, and eliminating target blurred and unclear images; labeling the images by using an autolabelme image automatic labeling tool to obtain target calibration data; and cropping the images by using a cropping program based on the labeled images to obtain continuously moving target image data.

[0056] Step 2: Based on the target calibration data, a target detection model based on an improved YOLOV5 is trained to obtain a target detection result. The YOLOV5 target detection model is used to detect targets under the view of an unmanned aerial vehicle, providing a selection for a user to track which target. Therefore, while ensuring high precision, real-time detection is also realized. The YOLOV5 detection model is a one-stage detection model, which is faster than a two-stage detection model, and its structure comprises three parts: a backbone network, a neck network and a head network. However, under the view of an unmanned aerial vehicle, there is a problem of too large a field of view and too small a target, which leads to fewer pixels of a ground target in an unmanned aerial vehicle camera, and difficulty in target feature extraction, and the use of the YOLOV5 detection model is not ideal in this scenario. Therefore, the YOLOV5 detection model is improved. The improvement of the YOLOV5 comprises: reconstructing the IoU loss calculation method of the YOLOV5 by using Focaler-IoU.

[0057] The IoU loss calculation method of the YOLOV5 is reconstructed by using Focaler-IoU, specifically as follows:

[0058] Let the coordinates of a real boundary box A be , and the coordinates of a predicted boundary box B be , then the width of the intersection of A and B is​ , the height is , the intersection area is ; the area of A is , the area of B is , the union area of A and B is ;

[0059]

[0060]

[0061]

[0062] wherein, is the calculated value of Focaler-IoU; , is a threshold value, ranging from .

[0063] The improvement of YOLOV5 also includes: optimizing the head network, adding a small target detection head for detecting 4x4 pixel size targets to the head network of YOLOV5, so that the number of detection heads changes from three to four.

[0064] The improvement of YOLOV5 also includes: reducing the model calculation amount, using a large stride to reduce the image data size in the first convolutional layer network of YOLOV5, so that the model achieves real-time calculation.

[0065] The improvement of YOLOV5 also includes: introducing C_BIoU to make the target continuous and selectable based on YOLOV5. After the target detection model detects the target position and target category information, it can be selected by the user. However, during the process of selecting the target, the target detected by the target detection model changes every time, and there may be a situation that the corresponding target cannot be found in the background after selecting the target on the screen. Therefore, the C_BIoU method is introduced to make the target continuous and selectable in cooperation with the target detection model. C_BIoU, also known as C-BIoU Tracker, is a multi-target tracking method that does not consider the motion state of the target. According to the information of the target detection model, the search area is expanded to associate the target of the current frame with the previous frame, so that the target can be continuous and not lost. And after the user selects the target, in order to ensure real-time performance, the target detection model is closed until the tracking target is lost.

[0066] Step 3: Based on the continuous motion of the target image data, train the target tracking model based on NanoTrack, track the target detection result, and introduce Kalman filter trajectory prediction for auxiliary tracking.

[0067] The NanoTrack target tracking model is to take the target as a target template after a given target, and search the subsequent image frames to find the region most matched with the target template. In addition, the model is a lightweight tracking network, and the network size and parameter amount are very small, which is very suitable for real-time tracking. When the target tracking model receives the tracking target information sent by the user, the important features of the current target are extracted as a template and stored. Then, the next frame of image is searched and matched with the template to obtain the best target position and target confidence. Since the target tracking model searches and matches in the expanded region with the center of the target template, similar targets and occluded targets may exist, which may lead to the loss of the real target. Therefore, Kalman filter (KF) trajectory prediction is introduced for auxiliary tracking.

[0068] The Kalman filter trajectory prediction is introduced for auxiliary tracking, specifically as follows.

[0069] According to the target confidence output by the target tracking model , different matching methods are used for calculation, specifically as follows.

[0070] When , the L2 norm matching is used to predict the track target frame and the tracking target frame, as follows.

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] wherein, The target confidence threshold; , These are the width and height of the actual bounding box, respectively; These are the coordinates of the actual bounding box; , These are the width and height of the predicted bounding box, respectively; To predict the coordinates of the bounding box; when If the target match is clear, Kalman filtering is invoked for trajectory prediction and update; otherwise, Kalman filtering trajectory prediction is performed. The threshold is the L2 norm.

[0085] when Matching is performed using IoU, as follows:

[0086]

[0087] in, The target confidence threshold; The area of ​​the intersection between the true bounding box and the predicted bounding box; The area of ​​the union of the true bounding box and the predicted bounding box; when If the target match is clear, Kalman filtering is invoked for trajectory prediction and update; otherwise, Kalman filtering trajectory prediction is performed. The threshold for the Intersection over Union (IoU) of the target bounding box and the predicted bounding box;

[0088] when At that time, Kalman filter trajectory prediction is performed.

[0089] After performing Kalman filter trajectory prediction, the process also includes: based on the trajectory prediction results of the Kalman filter, calling the gimbal to rotate and track the target, and controlling the lens to perform real-time zoom processing based on the target information obtained from the target tracking model, so that the target is kept in the center of the image and presented at a preset ratio.

[0090] It also includes: counting the number of matching failures in Kalman filter trajectory prediction; when the number of matching failures reaches the threshold, it is considered that the tracking target has failed, and the Kalman filter is initialized for auxiliary tracking. That is, for the received tracking target information, the Kalman filter is initialized, and the Kalman filter trajectory prediction is used for auxiliary tracking.

[0091] Example 2:

[0092] Embodiment 2 of the present invention discloses a vision-based intelligent UAV single-target tracking system utilizing a vision-based intelligent UAV single-target tracking method, comprising:

[0093] Data acquisition module: used to acquire high-definition data from the PTZ camera and perform data preprocessing to obtain target calibration data and continuously moving target image data;

[0094] The target detection module is configured to train a target detection model based on improved YOLOV5 based on target calibration data to obtain a target detection result.

[0095] The target tracking module is configured to train a target tracking model based on NanoTrack based on continuous motion target image data, track the target detection result, and introduce Kalman filter trajectory prediction for auxiliary tracking.

[0096] The embodiments of the present application disclose a visual-based intelligent unmanned aerial vehicle single-target tracking method and system. Through the combination of the YOLOV5 target detection model and the NanoTrack target tracking model, efficient target detection and tracking are realized, and the problem of poor real-time performance caused by serial processing in the prior art is overcome. In addition, through the optimization of the YOLOV5 target detection model in the head network, model calculation amount, target boundary regression frame loss calculation, and the introduction of the target continuous selectivity brought by C_BIoU, at the same time, on the basis of the NanoTrack target tracking model, Kalman filter trajectory prediction is used for auxiliary tracking, the continuity and stability of the target in the complex environment are truly realized, the long-time tracking ability of the target is enhanced, and the robustness and accuracy of the tracking are further improved.

[0097] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0098] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual-based intelligent unmanned aerial vehicle single-target tracking method, characterized in that, The application relates to a cloud platform-based target tracking method and device. Step 1: Obtain gimbal high-definition data, and perform data preprocessing to obtain target calibration data and continuously moving target image data; Step 2: Train a target detection model based on improved YOLOV5 based on the target calibration data to obtain a target detection result; wherein the improvement of the YOLOV5 comprises reconstructing an IoU loss calculation mode of the YOLOV5 by using a Focaler-IoU; Step 3: Train a target tracking model based on NanoTrack based on the continuously moving target image data, track the target detection result, and introduce Kalman filter trajectory prediction for auxiliary tracking. In step 2, the improvement of the YOLOV5 further comprises adding a minimum target detection head for detecting 4*4 pixel size targets to the head network of the YOLOV5, so that the number of detection heads is changed from three to four; In step 2, the improvement of the YOLOV5 further comprises reducing the size of image data by using a large step in the first convolutional layer network of the YOLOV5; In step 2, the improvement of the YOLOV5 further comprises introducing C_BIoU to make the target continuous and selectable based on the YOLOV5; In step 3, the Kalman filter trajectory prediction is introduced for auxiliary tracking, and the specific process is as follows: According to the target confidence degree delta output by the target tracking model, different matching methods are used for calculation, and the specific process is as follows: When delta is greater than or equal to alpha, the L2 norm matching is used to predict the track target frame and the tracking target frame, and the specific process is as follows: d w =|W A -W B |; d h =|h A -h B |; wherein a is a target confidence threshold; W A , h A are width and height of the real bounding box, respectively; are coordinates of the real bounding box;W B , h B are width and height of the predicted bounding box, respectively; are coordinates of the predicted bounding box; when L2≤ε, it indicates that the target matching is clear, and Kalman filtering is called for trajectory prediction and updating operation, otherwise Kalman filtering trajectory prediction is performed; ε is an L2 norm threshold. When beta is less than delta and delta is less than alpha, the IoU is used for matching, and the specific process is as follows: wherein β is a target confidence threshold; Intersection Area o is an intersection area of the real bounding box and the predicted bounding box; Union Area o is an union area of the real bounding box and the predicted bounding box; when loU>θ, it indicates that the target matching is clear, and Kalman filtering is called for trajectory prediction and updating operation, otherwise Kalman filtering trajectory prediction is performed; θ is a target box and predicted box intersection over union loU threshold; When delta is less than or equal to beta, Kalman filter trajectory prediction is performed. 2.The visual-based intelligent unmanned aerial vehicle single-target tracking method according to claim 1, characterized in that, In step 2, the IoU loss calculation mode of the YOLOV5 is reconstructed by using the Focaler-IoU, and the specific process is as follows: Let the coordinates of the real bounding box A be Let the coordinates of the predicted bounding box B be The width of the intersection of A and B is The height of the intersection of A and B is The area of the intersection of A and B is The area of A is The area of B is The area of the union of A and B is Union·Area o = |A| + |B| - Intersection·Area o ; L Focaler-IoU = 1 - IoU F ; wherein L Focaler-IoU is the calculated value of the Focaler-IoU; d, u are threshold values. 3.The visual-based intelligent unmanned aerial vehicle single-target tracking method according to claim 1, wherein, After the Kalman filter trajectory prediction is performed, the specific process is as follows: according to the trajectory prediction result of the Kalman filter, the tracking target is called by rotating the gimbal, and according to the target information obtained by the target tracking model, the lens is controlled to perform real-time zooming processing, so that the target is kept in the center of the picture and presented at a preset scale. 4.The visual-based intelligent unmanned aerial vehicle single-target tracking method according to claim 1, wherein, Further comprising: The matching failure number in the Kalman filter trajectory prediction is counted, and when the matching failure number threshold is reached, it is determined that the tracking target is invalid, and the Kalman filter is initialized for auxiliary tracking.

5. A visual-based intelligent unmanned aerial vehicle single target tracking system using the visual-based intelligent unmanned aerial vehicle single target tracking method according to any one of claims 1-4. Further comprising: The data acquisition module is used for acquiring gimbal high-definition data, and performing data preprocessing to obtain target calibration data and continuously moving target image data; The target detection module is used for training a target detection model based on improved YOLOV5 based on the target calibration data to obtain a target detection result; wherein the improvement of the YOLOV5 comprises reconstructing an IoU loss calculation mode of the YOLOV5 by using a Focaler-IoU; The target tracking module is used for training a target tracking model based on NanoTrack based on the continuously moving target image data, tracking the target detection result, and introducing Kalman filter trajectory prediction for auxiliary tracking.

Images