Multi-target tracking method based on adaptive association threshold
By combining adaptive association threshold and cross-union ratio (CUI) two-dimensional cost matrix, the association threshold is dynamically adjusted, solving the problems of mismatch and lack of association in multi-target tracking, and achieving more efficient target tracking accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGBING INTELLIGENT INNOVATION RES INST CO LTD
- Filing Date
- 2024-06-07
- Publication Date
- 2026-05-22
AI Technical Summary
Existing multi-target tracking methods suffer from issues such as lack of correlation or failed correlation, leading to false matching and multiple detections, and insufficient tracking accuracy, especially in different scenarios.
An adaptive association thresholding method is adopted, which dynamically adjusts the association threshold based on the density information of the target detection box. Target matching is performed by constructing a two-dimensional cost matrix based on the intersection-union ratio (IU). The trajectory information is updated by combining the Kalman filter algorithm to achieve adaptive target association.
It effectively avoids false associations and association failures, improves the accuracy and robustness of multi-target tracking, is suitable for real-time processing and large-scale datasets, and can dynamically adjust according to the scenario to optimize tracking performance.
Smart Images
Figure CN118587628B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to computer vision target recognition and tracking, and more particularly to a multi-target tracking method based on adaptive association threshold. Background Technology
[0002] Multi-object tracking is one of the fundamental research problems in computer vision, and it is widely used in autonomous driving, intelligent surveillance, and behavior recognition. It involves detecting multiple targets such as pedestrians, cars, and animals in a video and assigning them IDs for trajectory tracking, even when the number of targets is unknown beforehand. Different targets have different IDs to enable subsequent trajectory prediction and accurate location.
[0003] The main problem that object tracking solves is the detection, recognition, prediction, and tracking of moving objects in an image sequence. Currently, a common tracking strategy involves a detector detecting objects such as pedestrians and vehicles in the image, and then using the resulting bounding boxes for further tracking. After obtaining the object information in the current frame, trajectory prediction is performed on the existing trajectories of the objects. Algorithms such as Kalman filtering are typically used to obtain the estimated position of the target at the next moment. Then, the estimated positions of all targets are correlated with the positions of the bounding boxes obtained in the current frame, thus ensuring the continuation of the object's trajectory.
[0004] Matching algorithms for association typically employ either a greedy algorithm or the Hungarian algorithm. The greedy algorithm selects the trajectory with the lowest association cost (less than an association threshold) for each object in the current frame, while the Hungarian algorithm calculates the association method with the lowest total association cost. When using a greedy algorithm, an association threshold is usually set. When the minimum association cost exceeds the threshold, it is considered that no trajectory matches the current detection box, and a new ID is assigned to the current detection box. In practice, however, the association accuracy may be insufficient, leading to false associations or failed associations. Summary of the Invention
[0005] Based on the above analysis, the present invention aims to provide a multi-target tracking method based on an adaptive association threshold, in order to solve the problem of multiple detections or false matches caused by lack of association or association failure in existing multi-target tracking methods.
[0006] The objective of this invention is mainly achieved through the following technical solutions:
[0007] This invention provides a multi-target tracking method based on adaptive correlation threshold, characterized by the following steps:
[0008] Acquire video images containing multiple targets to be tracked, and process each frame of the video images as follows:
[0009] Extract the target detection bounding box of the current frame image, and obtain the target prediction bounding box of the active trajectory in the current frame;
[0010] A two-dimensional cost matrix is constructed based on the association cost between any target detection box and any target prediction box;
[0011] Based on the density of each target detection box, set the corresponding association threshold;
[0012] Based on the two-dimensional cost matrix and the association threshold of each target detection box, trajectory association is performed on each target detection box and each active trajectory in the current frame image to complete target tracking.
[0013] Furthermore, based on any object detection box and any object prediction box, the association cost is obtained using the following formula:
[0014]
[0015] 0<i≤N det ; 0 < j ≤ N pre
[0016] Where R2[i,j] represents the association cost between the i-th object detection box and the j-th object prediction box; IoU() represents the intersection-union ratio of the detection boxes; class(i) represents the class of the i-th object detection box; class(j) represents the class of the j-th object prediction box; N det N represents the number of object detection boxes; pre This indicates the number of target prediction boxes.
[0017] Furthermore, the step of associating each target detection box and each active trajectory in the current frame image based on the two-dimensional cost matrix and the association threshold of each target detection box includes:
[0018] Based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box and the target prediction box are matched;
[0019] When multiple target prediction boxes match the same target detection box, the target prediction box closest to the target detection box is selected as the matching object;
[0020] For successfully matched target detection boxes and target prediction boxes, the information of the active trajectory is updated based on the location information of the target detection boxes;
[0021] For unmatched bounding boxes, initialize a new motion trajectory as the active trajectory;
[0022] For unmatched target prediction boxes, their corresponding active trajectories are set as invalid trajectories.
[0023] Furthermore, the matching of target detection boxes and target prediction boxes based on the two-dimensional cost matrix and the association threshold of each target detection box includes:
[0024] For each target prediction box, based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box with the highest overlap is selected as the matching object;
[0025] For unmatched target detection boxes, based on the two-dimensional cost matrix, the unmatched target prediction boxes with the highest overlap are selected for rematching.
[0026] Furthermore, for each target prediction box, based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box with the highest overlap is selected as the matching object, including:
[0027] Using the two-dimensional cost matrix, find the target detection box with the minimum associated cost with the target prediction box;
[0028] When the association cost is less than the association threshold of the target detection box, the target detection box and the target prediction box are used as matching objects.
[0029] Furthermore, the step of selecting the unmatched target prediction box with the highest overlap for rematching based on the two-dimensional cost matrix includes: finding the unmatched target prediction box with the smallest association cost with the unmatched target detection box and less than a preset value through the two-dimensional cost matrix for matching.
[0030] Furthermore, selecting the target prediction box closest to the target detection box as the matching object includes: the center pixel of the target detection box is closest to the center pixel of the target prediction box.
[0031] Furthermore, the step of setting corresponding association thresholds based on the density of each target detection box includes:
[0032] Obtain the density values of each target detection box;
[0033] Based on the density value of each target detection box, a corresponding association threshold is set.
[0034] Furthermore, the larger the density value, the smaller the set association threshold.
[0035] Furthermore, extracting the target detection box of the current frame image also includes: removing background target detection boxes and duplicate target detection boxes from the target detection box.
[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0037] 1. This invention employs an adaptive association threshold method, setting an adaptive association threshold for each object based on the density information in its vicinity. This departs from the existing techniques that typically set the same association threshold for all objects or for each object category. This invention considers that because the environmental information of an object's location differs, the applicable association threshold for each object should be different. By adaptively determining the association threshold, it effectively avoids false associations and association failures.
[0038] 2. This invention constructs a two-dimensional matrix for correlation measurement based on the intersection-union ratio of the target detection box and the target prediction box of the active trajectory prediction. By calculating the distance between each target detection box and the target prediction box, the matching degree can be evaluated more accurately, and the best solution can be found quickly among many possible matches.
[0039] 3. This invention dynamically adjusts the association threshold of the target detection box, enabling the tracking algorithm to adaptively associate targets according to the actual situation of the current scene, thus avoiding the problem that a fixed threshold has a significant impact on the tracking accuracy in different scenes.
[0040] 4. The algorithm used in trajectory association in this invention has high computational efficiency and clear optimization rules, making it suitable for real-time processing and large-scale datasets. It can also be adjusted and optimized according to different scenarios, and its accuracy and robustness can be improved by introducing prior knowledge.
[0041] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0042] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0043] Figure 1 This is a flowchart illustrating the multi-target tracking method based on adaptive correlation threshold in an embodiment of the present invention;
[0044] Figure 2 This is a schematic diagram of the trajectory association process in an embodiment of the present invention. Detailed Implementation
[0045] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0046] A specific embodiment of the present invention discloses a multi-target tracking method based on adaptive correlation threshold, such as... Figure 1 As shown, the process includes: acquiring video images containing multiple targets to be tracked, and performing the following steps S1-S4 on each frame of the video images:
[0047] Step S1: Extract the target detection bounding box of the current frame image and obtain the target prediction bounding box of the active trajectory in the current frame.
[0048] Specifically, the current frame image is input into the target detector to obtain the target detection boxes output by the target detector; wherein, the target detector is an image recognition model, for example, a YOLO series neural network model is used. Each frame image includes multiple targets, and multiple target detection boxes are also extracted.
[0049] Furthermore, extracting the target detection box of the current frame image also includes: removing background target detection boxes and duplicate target detection boxes from the target detection box.
[0050] Specifically, all object detection boxes are categorized, and background object detection boxes are removed. For the remaining object detection boxes, for example, a non-maximum suppression algorithm is used to remove redundant object detection boxes.
[0051] The non-maximum suppression algorithm is a commonly used algorithm in object detection tasks to suppress overlapping detection results, thereby obtaining more accurate object detection results. Its core idea is to search for the optimal value of the bounding box with the highest confidence within the local range of the target, and remove redundant bounding boxes in the target's neighborhood.
[0052] Furthermore, in a multi-target tracking system, "active trajectories" refer to the trajectories of targets being tracked by the system. When a predicted target bounding box matches an actual target detection box in the current frame, it means that we have successfully found the target corresponding to the previous trajectory in the new frame. Therefore, this matched target detection box becomes the representative of the active trajectory in the current frame, which can be used to update the trajectory's state, such as position and velocity, and continue tracking the target. Each trajectory has a unique identifier (ID) to distinguish different targets. The active trajectory contains the target's position and velocity information in each frame. All target detection boxes in the first frame image are used as targets to be tracked, and their active trajectories are created.
[0053] Based on the position and velocity information of the active trajectory in the previous frame, the predicted position state of the target in the current frame is obtained through the Kalman filter algorithm, which serves as the target prediction box of the active trajectory in the current frame.
[0054] Step S2: Construct a two-dimensional cost matrix based on the association cost of any target detection box and any target prediction box.
[0055] Specifically, let N be the number of target detection boxes obtained in the current frame. det The number of target prediction boxes obtained from the active trajectories is N. pre The constructed two-dimensional cost matrix is then... Where R2[i,j] are elements in the two-dimensional cost matrix.
[0056] Furthermore, the association cost between any detected object box and any predicted object box is obtained using the following formula.
[0057]
[0058] 0<i≤N det ; 0 < j ≤ N pre
[0059] Where R2[i,j] represents the association cost between the i-th object detection box and the j-th object prediction box, i.e., an element in the two-dimensional cost matrix; IoU() represents the intersection-union ratio (IoU) between the i-th object detection box and the j-th predicted object detection box; class(i) represents the class of the i-th object detection box; class(j) represents the class of the j-th object prediction box; N det N represents the number of object detection boxes; pre This indicates the number of target prediction boxes.
[0060] Specifically, the intersection-union ratio (IU) describes the degree of overlap between two bounding boxes. It describes the relationship between two sets and is equal to the number of elements in the intersection of the two sets divided by the number of elements in their union. In image recognition, it represents the ratio of the overlapping area of two bounding boxes to the area of their union. The specific calculation formula is as follows:
[0061]
[0062] Among them, A i and B j These represent the elements contained in the i-th object detection box and the j-th predicted object detection box, respectively.
[0063] It should be noted that the cost matrix can also be constructed using metrics such as Euclidean distance or Manhattan distance instead of intersection-union ratio, but the effect will be reduced.
[0064] Step S3: Based on the density of each target detection box, set the corresponding association threshold.
[0065] Specifically, by considering the density of the bounding boxes at the location of the target object, different association thresholds are set for each target object. When the bounding boxes at the object's location are relatively dense, the association threshold should be set smaller to ensure stricter association and reduce the probability of false associations. When the bounding boxes at the object's location are relatively sparse, the association threshold should be set larger to reduce the probability of association failure. By considering the density information of the bounding boxes near each object and setting an adaptive association threshold, the probability of false associations and association failures can be effectively reduced.
[0066] Furthermore, the step of setting corresponding association thresholds based on the density of each target detection box includes:
[0067] Obtain the density values of each target detection box;
[0068] Based on the density value of each target detection box, a corresponding association threshold is set.
[0069] In one embodiment of the present invention, the density of a target detection box is defined as the maximum value of its intersection-union ratio (IU) with other detection boxes, as shown in the formula:
[0070]
[0071] Where, d i b represents the density value of the i-th object detection box; i b represents the i-th object detection box; m This represents the m-th target detection box.
[0072] In another embodiment of the present invention, a density subnet is used to learn the probability density distribution of the input data, predict the density of all detection boxes in the current frame, and obtain the density value of each detection box. For example, the density subnet can be a convolutional neural network (CNN) or a recurrent neural network (RNN).
[0073] Further, during the training process of the density sub-network, the training data set can adopt network data sets such as TUM, Euroc, and KITTI in the field of image recognition, and manually annotate the density of objects in images or video frames. Load the training data set into the density sub-network, and train the density sub-network through a standard deep learning process. Among them, the standard deep learning process includes: forward propagation, that is, the input data passes through the network layer to generate a predicted output. According to the difference between the predicted output and the label, use the mean square error (MSE) or cross-entropy loss function to train the density sub-network, perform backpropagation to update the parameters by calculating the gradient of the loss with respect to the network parameters to reduce the loss, use optimization algorithms such as stochastic gradient descent, adjust the network weights to minimize the loss function, and save the parameters of the density sub-network after the training ends.
[0074] Further, the greater the density value, the smaller the set association threshold.
[0075] Specifically, three density states are defined for the target detection box, namely dense, relatively dense, and sparse. According to the density value d of each target detection box i ,示例性的,可以使用如下公式得出每个目标检测框的关联阈值:
[0076]
[0077] 其中,λ i 为第i个检测框的关联阈值;a1表示稠密;a2表示相对稠密;a3表示稀疏;d i 表示第i个目标检测框的密度值;N det 表示目标检测框的数量。
[0078] It should be noted that a1 < a2 < a3. That is, for the detection boxes in the dense area, it is hoped that the association can be carried out more strictly because in the dense area, there may be more overlaps and interferences between targets, and higher accuracy is required to avoid false associations. Therefore, for the target detection boxes in the dense area, a smaller association threshold can be set to ensure that only the most matching trajectories can be associated with them.
[0079] For the detection boxes in the sparse area, since the spatial distribution between targets is relatively loose, a certain degree of loose association can be tolerated to cope with the uncertainty of the target appearance position. Therefore, for the target detection boxes in the sparse area, a larger association threshold can be set so that more trajectories can be associated with them.
[0080] Step S4: Based on the two-dimensional cost matrix and the association thresholds of each target detection box, perform trajectory association on each target detection box and each active trajectory in the current frame image to complete target tracking.
[0081] Furthermore, such as Figure 2 As shown, the trajectory association includes steps S41-S45:
[0082] Step S41: Based on the two-dimensional cost matrix and the association threshold of each target detection box, match the target detection box and the target prediction box.
[0083] Furthermore, for each target prediction box, based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box with the highest overlap is selected as the matching object.
[0084] Furthermore, the target detection box with the minimum associated cost with the target prediction box is found using the two-dimensional cost matrix.
[0085] Specifically, for each target prediction box, the target detection box with the lowest associated cost is found through a two-dimensional cost matrix, that is, the target detection box with the highest overlap with the target prediction box is selected.
[0086] When the association cost is less than the association threshold of the target detection box, the target detection box and the target prediction box are used as matching objects.
[0087] Specifically, the association threshold of the target detection box acts as a filter; only when the association cost is below the set threshold is the match between the target detection box and the target prediction box considered reliable. This helps avoid incorrectly associating the target detection box with the wrong trajectory.
[0088] Furthermore, for unmatched target detection boxes, based on the two-dimensional cost matrix, the unmatched target prediction boxes with the highest overlap are selected for re-matching, including:
[0089] Using the two-dimensional cost matrix, the unmatched target prediction box with the minimum association cost with the unmatched target detection box, which is less than a preset value, is found and matched.
[0090] Specifically, during the initial matching process, some target detection boxes may fail to match active trajectories due to various reasons (such as occlusion, lighting changes, detection errors, etc.). Secondary matching provides an opportunity to re-detect and match these missed targets. Simultaneously, in secondary matching, targets may not be matched due to special reasons (such as occlusion). In this case, to avoid missing targets again, the association threshold of the target detection boxes is not used for matching, increasing tracking flexibility. It should be noted that during secondary matching, to prevent over-matching, a preset value still needs to be set to prevent associating target detection boxes with any target prediction boxes. This preset value can be used as a parameter to optimize tracking performance through experimentation and adjustment, finding the optimal matching strategy.
[0091] Step S42: When multiple target prediction boxes match the same target detection box, select the target prediction box that is closest to the target detection box as the matching object.
[0092] Furthermore, selecting the target prediction box closest to the target detection box as the matching object includes: the center pixel of the target detection box is closest to the center pixel of the target prediction box.
[0093] Specifically, considering the historical information and stability of active trajectories, the nearest target detection box and target prediction box are selected for matching.
[0094] In another embodiment of the present invention, when selecting a target prediction box that matches the target detection box, a cost function-based method is used to treat the matching of the target detection box and the target prediction box as a problem of minimizing a cost function, and on this basis, the optimal matching scheme is selected.
[0095] The cost function can be either the sum of the absolute values of the distances between the detected bounding box and the predicted bounding box (L1 norm) or the sum of the squares of the distances (L2 norm).
[0096] Step S43: For the successfully matched target detection box and target prediction box, update the information of the active trajectory based on the position information of the target detection box.
[0097] Specifically, based on the location information of the target detection box, including the center point coordinates, width, and height of the target detection box, the residual between the target detection box and the matched target prediction box is calculated. Based on the covariance matrix of the target prediction box and the Jacobian matrix of the observation model, the Kalman gain is calculated and the target state estimate and covariance matrix are updated to reflect the new observation data.
[0098] Step S44: For unmatched target detection boxes, initialize a new motion trajectory as the active trajectory.
[0099] Specifically, for target detection boxes that do not match the trajectory, they are treated as new targets to be tracked, a trajectory ID is assigned to them, and an active trajectory is created.
[0100] Step S45: For unmatched target prediction boxes, set their corresponding active trajectories as invalid trajectories.
[0101] Specifically, as a failure trajectory, no target prediction bounding box is generated during the processing of the next frame image.
[0102] In summary, the multi-target tracking method based on adaptive correlation threshold of this invention has the following beneficial effects:
[0103] 1. This invention uses an adaptive association threshold method to set an adaptive association threshold for each object based on the density information near the object, which effectively reduces the results of false associations and association failures.
[0104] 2. This invention constructs a two-dimensional matrix for correlation measurement based on the intersection-union ratio of the target detection box and the target prediction box of the active trajectory prediction. By calculating the distance between each target detection box and the target prediction box, the matching degree can be evaluated more accurately, and the best solution can be found quickly among many possible matches.
[0105] 3. This invention dynamically adjusts the association threshold of the target detection box, enabling the tracking algorithm to adaptively associate targets according to the actual situation of the current scene, thus avoiding the problem that a fixed threshold has a significant impact on the tracking accuracy in different scenes.
[0106] 4. The algorithm used in trajectory association in this invention has high computational efficiency and clear optimization rules, making it suitable for real-time processing and large-scale datasets. It can also be adjusted and optimized according to different scenarios, and its accuracy and robustness can be improved by introducing prior knowledge.
[0107] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-target tracking method based on adaptive correlation threshold, characterized in that, Includes the following steps: Acquire video images containing multiple targets to be tracked, and process each frame of the video images as follows: Extract the target detection bounding box of the current frame image, and obtain the target prediction bounding box of the active trajectory in the current frame; A two-dimensional cost matrix is constructed based on the association cost between any target detection box and any target prediction box; Based on the density of each target detection box, set the corresponding association threshold; Based on the two-dimensional cost matrix and the association threshold of each target detection box, trajectory association is performed on each target detection box and each active trajectory in the current frame image to complete target tracking.
2. The method according to claim 1, characterized in that, Based on any object detection box and any object prediction box, the association cost is obtained using the following formula: 0<i≤N det ;0<j≤N pre Where R2[i,j] represents the association cost between the i-th object detection box and the j-th object prediction box; IoU() represents the intersection-union ratio of the detection boxes; class(i) represents the class of the i-th object detection box; class(j) represents the class of the j-th object prediction box; N det N represents the number of object detection boxes; pre This indicates the number of target prediction boxes.
3. The method according to claim 2, characterized in that, The step of associating each target detection box and each active trajectory in the current frame image based on the two-dimensional cost matrix and the association threshold of each target detection box includes: Based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box and the target prediction box are matched; When multiple target prediction boxes match the same target detection box, the target prediction box closest to the target detection box is selected as the matching object; For successfully matched target detection boxes and target prediction boxes, the information of the active trajectory is updated based on the location information of the target detection boxes; For unmatched bounding boxes, initialize a new motion trajectory as the active trajectory; For unmatched target prediction boxes, their corresponding active trajectories are set as invalid trajectories.
4. The method according to claim 3, characterized in that, The matching of target detection boxes and target prediction boxes based on the two-dimensional cost matrix and the association threshold of each target detection box includes: For each target prediction box, based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box with the highest overlap is selected as the matching object; For unmatched target detection boxes, based on the two-dimensional cost matrix, the unmatched target prediction boxes with the highest overlap are selected for rematching.
5. The method according to claim 4, characterized in that, For each predicted target bounding box, based on the two-dimensional cost matrix and the association threshold of each target detection box, the target detection box with the highest overlap is selected as the matching object, including: Using the two-dimensional cost matrix, find the target detection box with the minimum associated cost with the target prediction box; When the association cost is less than the association threshold of the target detection box, the target detection box and the target prediction box are used as matching objects.
6. The method according to claim 5, characterized in that, The step of selecting the unmatched target prediction box with the highest overlap for rematching based on the two-dimensional cost matrix includes: finding the unmatched target prediction box with the smallest association cost with the unmatched target detection box and less than a preset value through the two-dimensional cost matrix for matching.
7. The method according to claim 3, characterized in that, Selecting the target prediction box closest to the target detection box as the matching object includes: the center pixel of the target detection box is closest to the center pixel of the target prediction box.
8. The method according to any one of claims 1-7, characterized in that, The step of setting corresponding association thresholds based on the density of each target detection box includes: Obtain the density values of each target detection box; Based on the density value of each target detection box, a corresponding association threshold is set.
9. The method according to claim 8, characterized in that, The larger the density value, the smaller the set association threshold.
10. The method according to claim 1, characterized in that, Extracting the target detection bounding box of the current frame image also includes: removing background target detection bounding boxes and duplicate target detection bounding boxes from the target detection bounding box.