A method for tracking multiple target vehicles in a rotating frame of reference from a drone perspective

Abstract

The application designs a multi-target vehicle rotating frame tracking method under the visual angle of a UAV, increases a CA attention module and a Transformer prediction head in an original YOLOv5 detection model, simultaneously proposes a rotating frame representation method based on a bottom point coordinate, adopts a rotating frame instead of a horizontal frame to better identify a vehicle target, greatly improves the visual effect, obtains detection frame information and re-identification features based on a trained vehicle detection network and a re-identification network, realizes track prediction and data association through Kalman filtering and the Hungarian algorithm, and thus realizes multi-target tracking; in view of the limited computing resources of the UAV, the model is compressed under the premise of ensuring tracking accuracy and speed through FPGM pruning technology. The application can well detect and track vehicle targets under the visual angle of the UAV, well solves the problem of the limited computing resources of the UAV by using the FPGM pruning technology, and has good application potential in an intelligent traffic detection system.

Description

Technical Field

[0001] This invention relates to the field of vehicle target tracking, and more specifically to a method for tracking multiple target vehicles from the perspective of an unmanned aerial vehicle (UAV) with a rotating frame. Background Technology

[0002] Compared to manned aircraft, drones possess advantages such as small size, high flexibility, low cost, simple operation, low operational environment requirements, and strong battlefield survivability, making them crucial for future air warfare in the military field. Beyond the military, drones also have extremely wide applications in the civilian sector. Currently, they play a significant role in aerial photography, agriculture, plant protection, express delivery, power line inspection, and disaster relief. Intelligent traffic detection systems have become essential for ensuring urban traffic safety and improving the operational efficiency of traffic facilities. Vehicle detection and tracking are the primary tasks of intelligent traffic detection systems. Applying drone technology to vehicle tracking tasks can leverage the unique advantages of drones; however, vehicle tracking from the drone's perspective remains a critical issue that urgently needs to be addressed.

[0003] Tracking is primarily applied to any target in videos or sequentially semantically related images. It models the target's appearance features and motion information, predicting its motion state to determine its spatial location, shape, and size. Depending on the target being tracked, it can be categorized into single-target tracking and multi-target tracking; and depending on the tracking method, it can be divided into generative tracking methods and discriminative tracking methods. Generative tracking methods focus more on the target itself; classic methods include Kalman filtering, particle filtering, and mean shift. Discriminative filtering can be based on correlation filtering or deep learning methods, which can address the problem of insufficient sample size to some extent.

[0004] Unlike the common ground vehicle target tracking from the perspective of a camera, vehicle target tracking from the perspective of a drone faces the following challenges.

[0005] 1) When taking photos with a drone, the drone is positioned at a high altitude, offering a wide field of view and a complex background. The vehicle of interest can interfere with other objects, and with the environmental background, making it difficult to construct an accurate target detector.

[0006] 2) When the drone flies to a certain altitude, the field of view increases, the image resolution decreases, and the outline and texture features of vehicle targets become sparse, which brings new challenges to target detection and tracking;

[0007] 3) During the filming process, the drone may experience camera shake, motion blur, and rapid changes in vehicle direction due to factors such as light, wind, and the operator's flight control.

[0008] 4) Due to the inherent structural characteristics of UAVs, they cannot provide sufficient computing resources. Therefore, reducing algorithm complexity and compressing model size while ensuring accuracy becomes the top priority. Summary of the Invention

[0009] Vehicle targets viewed from the perspective of drones are characterized by complex backgrounds, small size, arbitrary orientation, and dense arrangement, posing challenges to vehicle detection and tracking. Furthermore, the limited computing resources available to drones place higher demands on algorithm optimization and model lightweighting.

[0010] To address the aforementioned issues and better achieve vehicle target tracking from a UAV perspective, this invention aims to provide a method for tracking multi-target vehicles with rotating bounding boxes from a UAV perspective, as detailed below:

[0011] A method for tracking rotating bounding boxes of multiple targets from the perspective of an unmanned aerial vehicle (UAV) includes the following steps:

[0012] Step 1), Vehicle bounding box detection: Create a vehicle dataset from the perspective of the drone, build and train a detection network model to achieve bounding box detection of vehicle targets.

[0013] Step 2), FPGM pruning: Compressing the detection network model by pruning redundant filters based on geometric median convolutional neural network filters;

[0014] Step 3), Re-identification Network Training: Construct a vehicle re-identification dataset, train the re-identification network model, obtain the appearance features and motion features of the target, and realize the re-identification of vehicle targets;

[0015] Step 4), Multi-target vehicle tracking: Input the video to be detected into the detection network model to obtain the target vehicle detection box. Extract the appearance and motion features of the detection box through the re-identification network, and calculate the similarity between the detection result and the Kalman filter prediction result. Finally, perform data association through the Hungarian algorithm to achieve multi-target tracking.

[0016] As a further optimization of the multi-target vehicle rotating frame tracking method from the perspective of an unmanned aerial vehicle (UAV) according to the present invention, the specific steps of step 1) are as follows:

[0017] Step 1.1): Using a drone, photograph vehicle targets at several key traffic routes in different flight states to obtain raw data; then, manually annotate each vehicle target with a rotated bounding box using annotation software; the annotation format is (x1, y1, x2, y2, x3, y3, x4, y4, class), where (x1, y1, x2, y2, x3, y3, x4, y4, class). i ,y iLet be the coordinates of the i-th vertex, 1≤i≤4, and class be the vehicle category, which includes three categories: car, bus, and truck.

[0018] Step 1.2): For N original images, repeat step 1.1) to obtain N corresponding label data, and then convert the (x1,y1,x2,y2,x3,y3,x4,y4,class) label format to (x center ,y center ,x bottom ,y bottom The training format is (x, w, h, class), where (x, w, h, class) center ,y center (x) represents the coordinates of the center point of the rotating frame. bottom ,y bottom ) represents the coordinates of the bottom point of the rotated box, and w, h, and class represent the width, height, and class of the rotated box, respectively. Finally, it is divided into training set and test set.

[0019] Step 1.3) Add the CA attention module and Transformer prediction head to the original YOLOv5 detection model, modify the model parameters, and set up the model's running environment;

[0020] Step 1.4): On a workstation with Ubuntu 16.04 operating system and NVIDIA RTX 2080Ti graphics card, set the training batch size to 4, the initial learning rate to 0.001, and train for 120 epochs.

[0021] As a further optimization of the multi-target vehicle rotation frame tracking method from the perspective of an unmanned aerial vehicle (UAV) according to the present invention, the specific steps of step 2) are as follows:

[0022] Step 2.1) Load the model parameters into the detection network model. For each convolutional layer, calculate the sum of the Euclidean distances between each convolutional kernel and all other convolutional kernels.

[0023] Step 2.2): Sort all the sums of Euclidean distances obtained in ascending order and prune the convolution kernels corresponding to those below a predetermined threshold μ.

[0024] Step 2.3) Force the gradient of the pruned convolutional kernel to zero, retrain the pruned detection network model, and repeat the pruning and training process until the detection network model converges.

[0025] Step 2.4) Remove all-zero convolutional kernels and redundant channels in the convolutional kernels, and also remove redundant values ​​of BN layer parameters to obtain a pruned and zero-removed compressed detection network model.

[0026] As a further optimization of the multi-target vehicle rotation frame tracking method from the perspective of an unmanned aerial vehicle (UAV) according to the present invention, the specific steps of step 3) are as follows:

[0027] Step 3.1) Construct a vehicle re-identification dataset to complete data augmentation and capacity expansion of the original VeRi dataset;

[0028] Step 3.2) Train the re-identification network model based on cosine metric learning, so that the re-identification network model has good classification ability and can accurately re-identify vehicle targets that have disappeared briefly and then reappeared.

[0029] As a further optimization of the multi-target vehicle rotation frame tracking method from the perspective of an unmanned aerial vehicle (UAV) according to the present invention, the specific steps of step 4) are as follows:

[0030] Step 4.1), Single-frame image vehicle target detection: Obtain a single-frame image from the video and feed it into the detection network model to obtain the position information (x, y, w, h, theta) and confidence scores of all vehicles in the image. Where x and y are the abscissa and ordinate of the center point of the vehicle rotation frame, respectively; w and h are the width and height of the vehicle rotation frame, respectively; and theta is the angle between the long side of the vehicle rotation frame and the horizontal line.

[0031] Step 4.2): For the detection box obtained in Step 4.1), its appearance features and motion features are obtained through the trained re-identification network model.

[0032] Step 4.3), Kalman filter prediction: (x,y,w,h,theta,dx,dy,dw,dh,dtheta) is used as the state vector of the state estimation model. The Kalman filter of the uniform motion model is used to predict the trajectory and predict the target trajectory of the next frame. Here, dx,dy,dw,dh,dtheta are the changes of x,y,w,h, andtheta, respectively.

[0033] Step 4.4) involves cascading and matching the confirmed state trajectory predicted by the Kalman filter with the detection box obtained by the target detector.

[0034] Step 4.5) Perform IOU matching between the unconfirmed trajectory predicted by Kalman filter and the detection box mismatch and trajectory mismatch results generated by cascade matching, and obtain the final matching result through the Hungarian algorithm;

[0035] Step 4.6): Update the Kalman filter parameters;

[0036] Step 4.7) Repeat steps 4.1) to 4.6) until the video ends, and obtain the final tracking result.

[0037] This invention employs the above technical solutions to achieve rotating bounding box tracking of vehicle targets from the perspective of an unmanned aerial vehicle (UAV). Addressing the challenges of small, densely packed, arbitrarily oriented, and complex background environments of vehicle targets from the UAV's perspective, an improved YOLOv5 target detector is used. A rotating bounding box representation method based on base coordinates is proposed, replacing the horizontal bounding box with a rotating one to achieve more accurate vehicle target detection and improve visual recognition performance. Real-time tracking of vehicle targets is achieved through the DeepSort target tracker. Simultaneously, FPGM pruning is performed on the network model to further compress the model while maintaining tracking accuracy and speed, aiming to solve the problem of limited computing resources in UAVs. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a schematic diagram of a rotating frame representation method based on the coordinates of the base point;

[0040] Figure 3 Diagram of the improved YOLOv5 network structure;

[0041] Figure 4 Flowchart for DeepSort target tracking. Detailed Implementation

[0042] Specific embodiments of the present invention are given below. These specific examples are only used to further illustrate the present invention and do not limit the scope of protection of the claims of this application.

[0043] The invention will be further described with reference to the accompanying drawings. See references. Figure 1 This invention provides a method for tracking multi-target vehicles with rotating frames from the perspective of an unmanned aerial vehicle (UAV), and the specific implementation steps are as follows:

[0044] Step 1), Vehicle bounding box detection: Create a vehicle dataset from the perspective of the drone, build a detection network model for training, and realize the bounding box detection of vehicle targets.

[0045] Step 1.1): Using a drone, photograph vehicle targets at several key traffic routes in different flight states to obtain raw data; then, using annotation software, manually annotate each vehicle target with a rotated bounding box, in the format (x1, y1, x2, y2, x3, y3, x4, y4, class), where (x1, y1, x2, y2, x3, y3, x4, y4, class). i ,y i (1≤i≤4) represents the coordinates of the i-th vertex; class represents the vehicle category, including cars, buses, and trucks; then the annotation format is converted to (x center ,y center ,xbottom ,y bottom The training format is (x, w, h, class), where (x, w, h, class) center ,y center (x) represents the coordinates of the center point of the rotating frame. bottom ,y bottom The coordinates of the center point, bottom point, and width and height of the rotated box are represented by , and w, h, and class represent the width, height, and class of the rotated box, respectively. A complete rotated box can be represented using its center point coordinates, bottom point coordinates, and width and height, as shown below. Figure 2 As shown;

[0046] Step 1.2), repeat step 1.1) to obtain 1015 sets of data, and split them into training set and test set, with a total of 828 sets of data in training set and 187 sets of data in test set;

[0047] Step 1.3): Add a CA attention module and a Transformer prediction head to the original YOLOv5 detection model, modify the model parameters, and set up the model's runtime environment, as shown in the reference. Figure 3 As shown;

[0048] Step 1.4): On a workstation with an operating system of Ubuntu 16.04 and a graphics card of NVIDIA RTX 2080Ti, set the training batch size to 4, the initial learning rate to 0.001, and train for 120 epochs.

[0049] Step 2), FPGM pruning: Compressing the detection network model by pruning redundant filters based on geometric median convolutional neural network filters;

[0050] Step 2.1) Load the model parameters into the detection network model. For each convolutional layer, calculate the sum of the Euclidean distances between each convolutional kernel and all other convolutional kernels.

[0051] Step 2.2): Sort all the sums of Euclidean distances obtained in ascending order and prune the convolution kernels corresponding to those below a predetermined threshold μ.

[0052] Step 2.3) Force the gradient of the pruned convolutional kernel to zero, retrain the pruned detection network model, and repeat the pruning and training process until the detection network model converges.

[0053] Step 2.4) Remove all-zero convolutional kernels and redundant channels in the convolutional kernels, and remove redundant values ​​of BN layer parameters to obtain a pruned and zero-removed compressed detection network model.

[0054] Step 3), Re-identification Network Training: Construct a vehicle re-identification dataset, train the re-identification network model, obtain the appearance features and motion features of the target, and realize the re-identification of vehicle targets;

[0055] Step 3.1) Construct a training dataset for the vehicle re-identification network. By writing scripts, enhance the distinction between vehicles and backgrounds, and randomly flip and adjust the brightness of images to complete the data augmentation and capacity expansion of the original VeRi dataset.

[0056] Step 3.2) The re-identification network model is trained based on cosine metric learning. The batch size is set to 32 and the learning rate is 0.001. After 400,000 iterations, the classification accuracy can reach 94.5%. At this time, the network has good classification ability and can accurately re-identify vehicle targets that have disappeared for a short time and then reappeared.

[0057] Step 4), Multi-target vehicle tracking: Input the video to be detected into the detection network model to obtain target vehicle detection boxes. Extract the appearance and motion features of the detection boxes through a re-identification network to obtain the similarity between the detection results and the Kalman filter prediction results. Finally, perform data association using the Hungarian algorithm to achieve multi-target tracking. The specific process is as follows: Figure 4 As shown;

[0058] Step 4.1), Single-frame image vehicle target detection: Obtain a single-frame image from the video and feed it into the detection network model to obtain the position information (x, y, w, h, theta) and confidence scores of all vehicles in the image. Where x and y are the abscissa and ordinate of the center point of the vehicle rotation frame, respectively; w and h are the width and height of the vehicle rotation frame, respectively; and theta is the angle between the long side of the vehicle rotation frame and the horizontal line.

[0059] Step 4.2): For the detection box obtained in Step 4.1), its appearance features and motion features are obtained through the trained re-identification network model.

[0060] Step 4.3), Kalman filter prediction: (x,y,w,h,theta,dx,dy,dw,dh,dtheta) is used as the state vector of the state estimation model. The Kalman filter of the uniform motion model is used to predict the trajectory and predict the target trajectory of the next frame. Here, dx,dy,dw,dh,dtheta are the changes of x,y,w,h, andtheta, respectively.

[0061] Step 4.4) involves cascading the confirmed trajectory predicted by the Kalman filter with the detection boxes obtained by the target detector; three results are obtained: Tracks mismatched (Unmatched Tracks), Detections mismatched (UnmatchedDetections), and Tracks matched (Tracks Matched).

[0062] Step 4.5): For cascaded matching results that are Tracks matching, directly perform trajectory update and ID maintenance; for cascaded matching results that are Detections mismatch, Tracks mismatch, and uncertain trajectories predicted by Kalman filter, perform IOU matching again, calculate the cost matrix, and obtain three matching results: Tracks mismatch, Detections mismatch, and Tracks matching through the Hungarian algorithm.

[0063] When the IOU matching result is a Detections mismatch, a new trajectory information and ID will be added and updated. For unconfirmed trajectories with IOU matching results that are Tracks mismatches, the trajectory will be deleted directly, and the trajectory and ID will no longer be updated or maintained. For confirmed trajectories with IOU matching results that are Tracks mismatches, if the number of Tracks mismatches exceeds the predetermined maximum lifespan, the trajectory will be deleted directly. If the number of mismatches does not exceed the maximum lifespan, the trajectory parameters and ID information will continue to be updated and maintained.

[0064] Step 4.6): Update the Kalman filter parameters;

[0065] Step 4.7) Repeat steps 4.1) to 4.6) until the video ends, and obtain the final tracking result.

[0066] The above description is merely a preferred embodiment of the present invention, intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for tracking multi-target vehicles with rotating frames from the perspective of an unmanned aerial vehicle (UAV), characterized in that, It includes the following steps: Step 1), Vehicle bounding box detection: Create a vehicle dataset from the perspective of the drone, build and train a detection network model to achieve bounding box detection of vehicle targets. Step 2), FPGM pruning: Compressing the detection network model by pruning redundant filters based on geometric median convolutional neural network filters; Step 3), Re-identification Network Training: Construct a vehicle re-identification dataset, train the re-identification network model, obtain the appearance features and motion features of the target, and realize the re-identification of vehicle targets; Step 4), Multi-target vehicle tracking: Input the video to be detected into the detection network model to obtain the target vehicle detection box. Extract the appearance and motion features of the detection box through the re-identification network, thereby calculating the similarity between the detection result and the Kalman filter prediction result. Finally, perform data association through the Hungarian algorithm to achieve multi-target tracking. Step 4.1), Single-frame image vehicle target detection: Obtain a single-frame image from the video and feed it into the detection network model to obtain the position information (x, y, w, h, theta) and confidence scores of all vehicles in the image. Where x and y are the abscissa and ordinate of the center point of the vehicle rotation frame, respectively; w and h are the width and height of the vehicle rotation frame, respectively; and theta is the angle between the long side of the vehicle rotation frame and the horizontal line. Step 4.2): For the detection box obtained in Step 4.1), its appearance features and motion features are obtained through the trained re-identification network model. Step 4.3), Kalman filter prediction: (x,y,w,h,theta,dx,dy,dw,dh,dtheta) is used as the state vector of the state estimation model. The Kalman filter of the uniform motion model is used to predict the trajectory and predict the target trajectory of the next frame. Here, dx,dy,dw,dh,dtheta are the changes of x,y,w,h, andtheta, respectively. Step 4.4) involves cascading and matching the confirmed state trajectory predicted by the Kalman filter with the detection box obtained by the target detector. Step 4.5) Perform IOU matching between the unconfirmed trajectory predicted by Kalman filter and the detection box mismatch and trajectory mismatch results generated by cascade matching, and obtain the final matching result through the Hungarian algorithm; Step 4.6): Update the Kalman filter parameters; Step 4.7) Repeat steps 4.1) to 4.6) until the video ends, and obtain the final tracking result.

2. The method for tracking multiple target vehicles with rotating frames from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that, The specific steps of step 1) are as follows: Step 1.1): Using a drone, photograph vehicle targets at several key traffic routes in different flight states to obtain raw data; then, manually annotate each vehicle target with a rotated bounding box using annotation software; the annotation format is (x1, y1, x2, y2, x3, y3, x4, y4, class), where (x1, y1, x2, y2, x3, y3, x4, y4, class). i ,y i Let be the coordinates of the i-th vertex, 1≤i≤4, and class be the vehicle category, which includes three categories: car, bus, and truck. Step 1.2): For N original images, repeat step 1.1) to obtain N corresponding label data, and then convert the (x1,y1,x2,y2,x3,y3,x4,y4,class) label format to (x center ,y center ,x bottom ,y bottom The training format is (x, w, h, class), where (x, w, h, class) center ,y center (x) represents the coordinates of the center point of the rotating frame. bottom ,y bottom ) represents the coordinates of the bottom point of the rotated box, and w, h, and class represent the width, height, and class of the rotated box, respectively. Finally, it is divided into training set and test set. Step 1.3) Add the CA attention module and Transformer prediction head to the original YOLOv5 detection model, modify the model parameters, and set up the model's running environment; Step 1.4): On a workstation with Ubuntu 16.04 operating system and NVIDIA RTX 2080Ti graphics card, set the training batch size to 4, the initial learning rate to 0.001, and train for 120 epochs.

3. The method for tracking multiple target vehicles with rotating frames from the perspective of an unmanned aerial vehicle according to claim 2, characterized in that, The specific steps of step 2) are as follows: Step 2.1) Load the model parameters into the detection network model. For each convolutional layer, calculate the sum of the Euclidean distances between each convolutional kernel and all other convolutional kernels. Step 2.2): Sort all the sums of Euclidean distances obtained in ascending order and prune the convolution kernels corresponding to those below a predetermined threshold μ. Step 2.3) Force the gradient of the pruned convolutional kernel to zero, retrain the pruned detection network model, and repeat the pruning and training process until the detection network model converges. Step 2.4) Remove all-zero convolutional kernels and redundant channels in the convolutional kernels, and also remove redundant values ​​of BN layer parameters to obtain a pruned and zero-removed compressed detection network model.

4. The method for tracking multiple target vehicles with rotating frames from the perspective of an unmanned aerial vehicle according to claim 3, characterized in that, The specific steps of step 3) are as follows: Step 3.1) Construct a vehicle re-identification dataset to complete data augmentation and capacity expansion of the original VeRi dataset; Step 3.2) Train the re-identification network model based on cosine metric learning, so that the re-identification network model has good classification ability and can accurately re-identify vehicle targets that have disappeared and reappeared briefly.

Images