A multi-target tracking method and device for unmanned aerial vehicles using a divide-and-conquer approach.

Abstract

This invention discloses a multi-target tracking method for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach: Two consecutive frames of images and a motion query heatmap from the previous frame are input into a backbone network, simultaneously predicting the target's position, appearance features, and motion between adjacent frames. A divide-and-conquer target association algorithm is designed to handle online and lost trajectories during tracking. Motion information is used to associate online trajectories with detected targets in adjacent frames, while appearance features are used to associate lost trajectories with detected targets across multiple frames. To ensure the reliability of position association, the estimated position of the target's bounding box in the previous frame is calculated using the predicted target center point motion, and a similarity calculation (motion matching) is performed with the bounding box of the tracked trajectory in the previous frame. This invention can run in real-time at an image resolution of 608×1088. This invention also provides a corresponding multi-target tracking device for UAVs using a divide-and-conquer approach.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision technology, and more specifically, relates to a method and apparatus for multi-target tracking of unmanned aerial vehicles through divide-and-conquer correlation. Background Technology

[0002] Target tracking is one of the fundamental tasks in computer vision, requiring the localization and identification of targets across consecutive video frames. Achieving this often necessitates reliable detectors and robust tracking strategies. For detectors, multi-task training designs are required, while for trackers, specific scene-specific tracking and association strategies must be designed. Due to the characteristics of drone scenarios, such as rapid camera motion, camera blur, and sparse targets in the image, typical tracking algorithms based on fixed cameras struggle to perform effectively in moving scenes. Appearance features, as a relatively reliable association criterion in unobstructed (sparse) scenes, can be used for long-range association of targets in drone scenarios. However, in some scenarios, rapid camera motion can drastically alter the target's appearance due to camera blur, affecting the reliability of appearance features as an association criterion. Furthermore, traditional Kalman filter-based trackers, due to their prior assumption of linear motion, often fail to accurately reflect the target's precise location in drone scenarios due to the bounding boxes obtained through Kalman filtering. Summary of the Invention

[0003] This invention addresses the impact of rapid camera movement and motion blur on tracking performance in UAV scenarios. It proposes a robust multi-target tracking scheme using a divide-and-conquer association algorithm, enabling the tracker to reliably associate targets even under intense motion. Furthermore, its performance and speed outperform industry-leading traditional tracking frameworks in UAV scenarios. This invention primarily combines the advantages of motion displacement and appearance features in the tracking process, leveraging the complementarity of motion and appearance to perform robust target association in UAV scenarios.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for multi-target tracking of unmanned aerial vehicles (UAVs) using a divide-and-conquer approach is provided, comprising the following steps:

[0005] (1) Training detection, motion prediction and appearance feature tasks through deep learning models;

[0006] (2) Input the previous frame tracking result query heatmap, the previous frame image and the current frame image into the DCTrack backbone network framework at the same time, and extract the detection box of the current frame, the appearance features of the target and the motion displacement of the target between the previous and next frames.

[0007] (3) A divide-and-conquer strategy is adopted, using predicted motion displacement and appearance features to process online and lost trajectories respectively, so as to make full use of the complementary effect of location information and appearance information.

[0008] In one embodiment of the present invention, step (3) specifically includes:

[0009] (3.1) The position of the detection bounding box in the current frame in the previous frame is estimated by using the predicted target center point's motion displacement between adjacent frames, and the similarity is matched with the online trajectory bounding box in the previous frame by intersection and union comparison. The matched online trajectory remains online, and the unmatched online trajectory is marked as a lost trajectory.

[0010] (3.2) For the detection targets that did not obtain a match in (3.1), the appearance features corresponding to them are matched with the appearance features of the lost trajectory. The lost trajectory that obtains a match is considered as a re-identified trajectory and is marked as an online trajectory. The lost trajectory that did not obtain a match is examined for the number of lost frames. If it is greater than N frames, it will be determined that the trajectory has left the scene and will be deleted, where N is a preset value.

[0011] (3.3) For the detected targets that did not obtain a match in (3.2), perform bounding box intersection and union matching with the uncertain new trajectory in the previous frame. Uncertain trajectories that did not obtain an association will be judged as false positive trajectories and deleted. Uncertain trajectories that obtained an association will have their trajectory status marked as online trajectories.

[0012] (3.4) For the detected targets that did not obtain a match in (3.3), examine their confidence score. If the confidence score is greater than the threshold A, initialize it as a new trajectory and mark the new trajectory as an uncertain state so that it can be examined in the tracking of the next frame, where A is a preset threshold.

[0013] (3.5) Output the bounding boxes of all tracked online trajectories from steps (3.1) to (3.3) and their corresponding target IDs as the accurate tracking results of the tracker.

[0014] In one embodiment of the present invention, in step (1), the DCTrack backbone network is trained on the UAVDT training set. First, the current frame image, a random frame image near the current frame in time sequence, and a heatmap of the target marker center point rendered in the random frame are used as inputs to the DCtrack backbone network to predict the bounding box, appearance features, and motion displacement of the target in the current frame between the current frame and the random frame. Then, the marker box of the current frame, the target marker ID, and the motion displacement marker of the target center point between the current frame and the random frame are used to supervise the detection branch, appearance prediction branch, and motion prediction branch, respectively.

[0015] In one embodiment of the present invention, the loss function for training the detection branch is the intersection-over-union loss function and the L1 loss function, the loss function for training the motion prediction branch is the L1 loss function, and the loss function for training the appearance features is the cross-entropy loss function.

[0016] In one embodiment of the present invention, in step (2), the DCTrack backbone network framework consists of three parts: a backbone network, a multi-scale feature fusion network, and a multi-task branch prediction. The backbone network is composed of a convolutional neural network DLA-34. The multi-scale feature fusion network uses deformable convolution operators (DCN) and deconvolution operators to upsample feature maps at each stage. The multi-task branch consists of a detection branch, a motion prediction branch, and an appearance feature branch, which correspond to the bounding box prediction task, the motion prediction task between adjacent frames, and the appearance feature prediction task of the target, respectively.

[0017] In one embodiment of the present invention, the input of inference information in step (2) specifically includes: for the initial stage of video frame inference, the present invention uses the image of the first frame as the current frame and the previous frame, and initializes the tracking result query heatmap as a blank heatmap; in the inference of subsequent frames, the center point of the tracking target in the previous frame is rendered on the heatmap in the form of Gaussian dots, and the heatmap is used as the tracking result query heatmap and input together with the previous frame image and the current frame image into the DCTrack backbone network.

[0018] In one embodiment of the present invention, the reasoning prediction performed in step (2) specifically includes:

[0019] The DCTrack backbone network simultaneously predicts target bounding box information including the target center point, target motion displacement between adjacent frames, and target appearance features. Specifically, the bounding box is predicted as the target center point coordinates (cx, cy) and left, top, right, and bottom offsets (Δl, Δt, Δr, Δb) from the target boundary; the motion displacement is predicted as the motion displacement (Δx, Δy) of the same target center point between adjacent frames; and the appearance features are predicted as the appearance description feature vector L∈R for each target. d d represents the dimension of the appearance feature vector.

[0020] In one embodiment of the present invention, N is set to 30, A is set to 0.65, the crossover ratio matching threshold for motion matching is 0.5, the threshold for appearance feature similarity matching is 0.17, the trajectory confirmation crossover ratio matching threshold is 0.6, and the trajectory appearance feature is updated using an exponential moving average method, wherein the update parameter alpha is set to 0.9.

[0021] In one embodiment of the present invention, multi-target tracking inference in a drone scenario is achieved at a real-time running speed of no less than 25 frames per second at a resolution of 608×1088.

[0022] According to another aspect of the present invention, a multi-target tracking device for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach is also provided, comprising at least one processor and a memory connected via a data bus, wherein the at least one processor and the memory store instructions executable by the at least one processor, the instructions being executed by the processor to perform the aforementioned multi-target tracking method for UAVs using a divide-and-conquer approach.

[0023] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0024] (1) This invention addresses the issue of inaccurate target position estimation by using a deep learning model to model the online motion of the target in the context of intense camera movement in UAV scenarios.

[0025] (2) This invention proposes a divide-and-conquer association strategy for online trajectories and lost trajectories. It uses reliable location to associate online trajectories in consecutive frames and uses robust appearance features to associate lost trajectories across multiple frames. Compared with existing methods dominated by appearance features, this invention improves the utilization rate of appearance features in re-identification tasks, reduces the impact of too many appearance features (including blurred appearances) on association performance, and uses reliable location based on motion prediction to supplement the unreliability of appearance features for blurred targets during tracking, thereby improving the accuracy and speed of tracking.

[0026] (3) This invention designs an efficient and concise multi-target tracking framework. By inputting the tracking result of the previous frame, querying the heatmap, the image of the previous frame, and the image of the current frame, it simultaneously infers the detection information of the current frame, the motion displacement between adjacent frames, and the appearance features of the corresponding detected targets within a single stage. It then fully utilizes this information to execute a robust multi-target tracking process using a divide-and-conquer association strategy. Compared to existing tracking frameworks, this invention can provide richer association information in a single segment and execute a more robust association process, demonstrating the high efficiency of the entire framework. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a multi-target tracking method for unmanned aerial vehicles (UAVs) based on a divide-and-conquer approach provided by the present invention.

[0028] Figure 2 This is a schematic diagram of a simple and efficient multi-task detection framework structure provided by an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram comparing the performance of the tracker provided by this invention with other trackers. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] like Figure 1 As shown, this invention provides a multi-target tracking method for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach, comprising:

[0032] (1) Training detection, motion prediction, and appearance feature tasks through deep learning models, specifically including:

[0033] (1.1) The DCTrack backbone network is trained on the UAVDT training set. Specifically, during the training phase, the present invention first uses the current frame image, a random frame image near the current frame in time sequence, and a heatmap of the target marker center point rendered in the random frame as input to the DCtrack backbone network, and predicts the bounding box, appearance features, and motion displacement of the target in the current frame between the current frame and the random frame. Then, the marker box, target marker ID, and motion displacement marker of the target center point between the current frame and the random frame are used to supervise the detection branch, appearance prediction branch, and motion prediction branch, respectively.

[0034] (1.2) The loss function for training the detection branch is the intersection-union loss function and the L1 loss function, the loss function for training the motion prediction branch is the L1 loss function, and the loss function for training the appearance features is the cross-entropy loss function.

[0035] (2) Input inference information, that is, simultaneously input the heatmap of the tracking results of the previous frame, the image of the previous frame, and the image of the current frame into the DCTrack backbone network framework for inference and prediction, and extract the detection box of the current frame, the appearance features of the target, and the motion displacement of the target between the previous and next frames, specifically including:

[0036] (2.1) Reasoning framework: such as Figure 2 As shown, the DCTrack backbone network framework consists of three parts: the backbone network, the multi-scale feature fusion network, and the multi-task branch prediction. The backbone network is composed of a convolutional neural network DLA-34. The multi-scale feature fusion network mainly uses deformable convolution operators (DCN) and deconvolution operators to upsample feature maps at each stage. The multi-task branch consists of a detection branch, a motion prediction branch, and an appearance feature branch, which correspond to the bounding box prediction task, the motion prediction task between adjacent frames, and the appearance feature prediction task of the target, respectively.

[0037] (2.2) Inference Input Information: This invention inputs the previous frame tracking result query heatmap, the previous frame image, and the current frame image into the DCTrack backbone network for inference prediction. Specifically, in the initial stage of video frame inference, this invention uses the image of the first frame as the current frame and the previous frame, and initializes the tracking result query heatmap as a blank heatmap. In the inference of subsequent frames, the center point of the tracked target in the previous frame is rendered on the heatmap in the form of Gaussian dots, and this heatmap is used as the tracking result query heatmap and input into the DCTrack backbone network together with the previous frame image and the current frame image. Providing the previous frame tracking result query heatmap in the input can serve as a motion cue for the target, effectively improving the accuracy of DCTrack in predicting the motion of the target between adjacent frames.

[0038] (2.3) Inference and prediction information: such as Figure 2 As shown, the DCTrack backbone network simultaneously predicts target bounding box information (including target center point), target motion displacement between adjacent frames, and target appearance features. Specifically, the bounding box prediction includes the target center point coordinates (cx, cy) and left, top, right, and bottom offsets (Δl, Δt, Δr, Δb) from the target boundary; the motion displacement prediction includes the motion displacement (Δx, Δy) of the same target center point between adjacent frames; and the appearance features prediction includes the appearance description feature vector L∈R for each target. d d is the dimension of the appearance feature vector;

[0039] (3) A divide-and-conquer strategy is adopted, using predicted motion displacement and appearance features to process online and lost trajectories respectively, in order to fully utilize the complementary effect of location and appearance information. Specifically, this includes the following steps:

[0040] (3.1) The position of the detection bounding box in the current frame in the previous frame is estimated by using the predicted target center point's motion displacement between adjacent frames, and the similarity is matched with the online trajectory bounding box in the previous frame by intersection and union comparison. The matched online trajectory remains online, and the unmatched online trajectory is marked as a lost trajectory.

[0041] (3.2) For the detection targets that did not obtain a match in (3.1), the appearance features corresponding to them are matched with the appearance features of the lost trajectory. The lost trajectory that obtains a match is considered as a re-identified trajectory and is marked as an online trajectory. The lost trajectory that did not obtain a match is examined for the number of lost frames. If it is greater than N frames, it will be determined that the trajectory has left the scene and will be deleted, where N is a preset value.

[0042] (3.3) For the detected targets that did not obtain a match in (3.2), perform bounding box intersection and union matching with the uncertain new trajectory in the previous frame. Uncertain trajectories that did not obtain an association will be judged as false positive trajectories and deleted. Uncertain trajectories that obtained an association will have their trajectory status marked as online trajectories.

[0043] (3.4) For the detected targets that did not obtain a match in (3.3), examine their confidence score. If the confidence score is greater than the threshold A, initialize it as a new trajectory and mark the new trajectory as an uncertain state so that it can be examined in the tracking of the next frame, where A is a preset threshold.

[0044] (3.5) Output the bounding boxes of all tracked online trajectories from steps (3.1) to (3.3) and their corresponding target IDs as the accurate tracking results of the tracker.

[0045] The divide-and-conquer association method designed in this invention utilizes motion displacement and appearance features to process online trajectory targets and lost trajectory targets separately. Appearance features are processed only for lost trajectory targets, rather than all trajectories. This aims to reduce the involvement of appearance features in tracking, avoid the impact of blurred appearance on tracking performance, and simultaneously improve the running speed of the tracking algorithm (the more appearance feature vectors involved in the calculation, the slower the tracking algorithm runs).

[0046] Furthermore, in the default settings of this invention, N is set to 30, A to 0.65, d to 128, the crossover ratio (CRO) threshold for motion matching is 0.5, the threshold for appearance feature similarity matching is 0.17, the trajectory confirmation CRO threshold is 0.6, and the trajectory appearance feature update adopts the exponential moving average method, wherein the update parameter alpha is set to 0.9. At a resolution of 608×1088, multi-target tracking inference in UAV scenarios is achieved at a real-time running speed of no less than 25 frames per second.

[0047] like Figure 3 As shown, the UAV multi-target tracking framework DCTrack, based on the divide-and-conquer correlation algorithm of this invention, outperforms existing methods in both speed and accuracy, achieving a leading level. On the UAV multi-target tracking dataset UAVDT, using a CenterNet detector based on the DLA-34 backbone network, DCTrack achieves 49.1 MOTA (multi-target tracking accuracy) and 68.7 IDF1 (ID tracking accuracy) at a speed of 27 FPS. Under the same conditions, DCTrack achieves the best trade-off between speed and accuracy compared to other methods.

[0048] Furthermore, the present invention also provides a multi-target tracking device for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions executable by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the aforementioned multi-target tracking method for UAVs using a divide-and-conquer approach.

[0049] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-target tracking method for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach, characterized in that, Includes the following steps: (1) Detection, motion prediction and appearance feature tasks are trained through deep learning models; In step (1), the DCTrack backbone network is trained on the UAVDT training set. First, the current frame image, a random frame image near the current frame in time sequence, and the heatmap of the target marker center point of the random frame are used as inputs to the DCtrack backbone network to predict the bounding box, appearance features, and motion displacement of the target in the current frame between the current frame and the random frame. Then, the marker box of the current frame, the target marker ID, and the motion displacement marker of the target center point between the current frame and the random frame are used to supervise the detection branch, appearance prediction branch, and motion prediction branch, respectively. (2) Input the previous frame tracking result query heatmap, the previous frame image and the current frame image into the DCTrack backbone network framework at the same time, and extract the detection box of the current frame, the appearance features of the target and the motion displacement of the target between the previous and next frames. (3) A divide-and-conquer association strategy is adopted, using predicted motion displacement and appearance features to process online and lost trajectories respectively, so as to make full use of the complementary role of position information and appearance information; the specific steps (3) include: (3.1) using the predicted motion displacement of the target center point between adjacent frames to estimate the position of the detection bounding box in the current frame in the previous frame, and performing intersection-union similarity matching with the online trajectory bounding box in the previous frame. The online trajectory that is matched is still in the online state, and the online trajectory that is not matched is marked as a lost trajectory; (3.2) for the detection target that is not matched in (3.1), the appearance features corresponding to it are matched with the appearance features of the lost trajectory. The lost trajectory that is matched is considered as a re-identified trajectory and is marked as an online trajectory. The lost trajectory that is not matched is examined for the number of lost frames. If the number of frames exceeds N, the trajectory will be determined to have left the scene and deleted, where N is a preset value; (3.3) For the detected target that did not get a match in (3.2), the bounding box intersection-union matching is performed with the uncertain new trajectory in the previous frame. Uncertain trajectories that do not get an association will be determined as false positive trajectories and deleted. Uncertain trajectories that get an association will have their trajectory status marked as online trajectories; (3.4) For the detected target that did not get a match in (3.3), its confidence score is examined. If the confidence score is greater than the threshold A, it is initialized as a new trajectory and the new trajectory is marked as uncertain so that it can be examined in the tracking of the next frame, where A is a preset threshold; (3.5) Output the bounding boxes of all tracked online trajectories in steps (3.1) to (3.3) and their corresponding target IDs as the accurate tracking results of the tracker.

2. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, The loss functions for training the detection branch are the intersection-over-union loss function and the L1 loss function, the loss function for training the motion prediction branch is the L1 loss function, and the loss function for training the appearance features is the cross-entropy loss function.

3. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, In step (2), the DCTrack backbone network framework consists of three parts: the backbone network, the multi-scale feature fusion network, and the multi-task branch prediction. The backbone network is composed of the convolutional neural network DLA-34. The multi-scale feature fusion network mainly uses deformable convolutional operators DCN and deconvolution operators to upsample the feature maps at each stage. The multi-task branch consists of a detection branch, a motion prediction branch, and an appearance feature branch, which correspond to the bounding box prediction task, the motion prediction task between adjacent frames, and the appearance feature prediction task of the target, respectively.

4. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, The input of inference information in step (2) specifically includes: for the initial stage of video frame inference, the image of the first frame is used as the current frame and the previous frame, and the tracking result query heatmap is initialized as a blank heatmap; in the inference of subsequent frames, the center point of the tracking target in the previous frame is rendered on the heatmap in the form of Gaussian dots, and this heatmap is used as the tracking result query heatmap and input together with the previous frame image and the current frame image into the DCTrack backbone network.

5. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, In step (2), the inference and prediction are performed, specifically including: The DCTrack backbone network simultaneously predicts target bounding box information including the target center point, target motion displacement between adjacent frames, and target appearance features; specifically, the bounding box is predicted as the coordinates of the target center point. And information on the left, top, right, and bottom offsets from the target boundary. Motion displacement prediction refers to the motion displacement of the same target center point between adjacent frames. Appearance features predict the appearance description feature vector for each target. , The dimension of the appearance feature vector.

6. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, The value of N is set to 30, A is set to 0.65, the crossover ratio (CRO) threshold for motion matching is 0.5, the threshold for appearance feature similarity matching is 0.17, the CRO threshold for trajectory confirmation is 0.6, and the update of trajectory appearance features adopts the exponential moving average method, wherein the update parameter alpha is set to 0.

9.

7. The UAV multi-target tracking method based on divide-and-conquer as described in claim 1, characterized in that, exist At a resolution of no less than 25 frames per second, multi-target tracking inference in drone scenarios is achieved.

8. A multi-target tracking device for unmanned aerial vehicles (UAVs) using a divide-and-conquer approach, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the UAV multi-target tracking method by divide and conquer as described in any one of claims 1-7.

Images