Online multi-target tracking method based on confidence optimization

Abstract

The present invention discloses an online multi-target tracking method based on confidence optimization, which takes the detection confidence and matching cost corresponding to the trajectory as parameters, iteratively updates the trajectory confidence frame by frame based on the exponential moving average, and constructs a confidence weight matrix in combination with the trajectory and trajectory confidence, and performs weighted optimization on the similarity matrix. Unlike previous trajectory confidence models, the present invention combines the common intermediate quantities (detection confidence and matching cost) in MOT tasks, uses the EMA model for iterative update, has low computational complexity, and does not require additional parameter solution; it is beneficial to online tasks; unlike the common practice of TBD trackers in the past, the present invention does not set any confidence threshold for the objects involved in the association. When the association is first made, the output and activated trajectories of all detectors are allowed to participate in the association, which helps to recover interrupted trajectories from ultra-low score detections, reduces the complexity of the hyperparameter space, and facilitates actual deployment.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-target tracking, and in particular relates to an online multi-target tracking method based on confidence optimization. Background Art

[0002] Online multi-object tracking (MOT) is a highly anticipated task in computer vision. It aims to estimate the states of multiple objects in a video sequence and maintain the continuity of their identities (i.e., IDs) in the face of frequent occlusions, similar appearances, and camera motion. With the development of object detectors, online tracking methods based on tracking-by-detection (TBD) have become a focus of academic research. This method calculates various similarity distances (such as IOU distance and appearance distance) between the current detection box and historical trajectories, constructs an association cost matrix, and uses the Hungarian algorithm (HA) to determine the matching relationship between detections and trajectories. This method is generally effective, but in high-occlusion environments, the similar spatial positions and incomplete appearance information between objects can make the cost matrix unreliable, which can easily lead to association errors.

[0003] A key challenge for existing TBD-based methods is correctly associating unreliable detections with trajectories. For example, Reference 1 sets a detection confidence threshold to avoid unreliable low-confidence detections, but the loss of these detections can cause irreparable trajectory interruptions. Reference 2 proposes an online tracking method that first matches reliable high-confidence detections before processing unreliable low-confidence detections, effectively improving tracking accuracy and robustness. Reference 3 introduces similarity constraints to filter out ambiguous matching relationships in the cost matrix and handle them separately according to a specific strategy, improving the algorithm's robustness to occlusion.

[0004] In practice, the aforementioned TBD-type methods all partition the association cost matrix through hyperparameters to perform hierarchical, multi-stage data association, thereby mitigating the interference of fuzzy matching. However, the setting of hyperparameters is affected by many factors, such as detector performance, dataset, and specific task characteristics. Even when using the same detector, different datasets can lead to significant variations in the confidence distribution. These differences are caused by different dataset characteristics, such as the degree of object occlusion, video shooting angle, and lighting conditions. Therefore, fixed hyperparameters generally cannot achieve optimal results in all cases. Furthermore, while existing multi-stage trackers have tried their best to recover tracks from low-confidence detections, they still do not achieve true full detection association. This has adverse effects on practical deployments: on the one hand, it increases the complexity of the hyperparameter space, requiring repeated debugging to determine the threshold for isolating very low-scoring detections; on the other hand, directly abandoning very low-confidence detections can cause track interruptions.

[0005] References:

[0006] Document 1: [Bewley A, Ge Z, Ott L, et al.Simple online and realtime tracking[C] / / 2016IEEE international conference on image processing(ICIP).IEEE,2016:3464-3468.];

[0007] Document 2: [Zhang Y, Sun P, Jiang Y, et al. Bytetrack: Multi-object tracking by associat ing every detection box [C] / / European Conference on ComputerVision. Cham: Springer Nature Switzerland, 2022: 1-21.];

[0008] Document 3: [Stadler D, Beyerer J. Modeling ambiguous assignments for multi-person tracking in crowds[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision.2022:133-142.]. Summary of the Invention

[0009] In order to achieve data association with low hyperparameter dependence and full detection participation, and reduce the workload of hyperparameter adjustment in engineering deployment, the present invention provides an online multi-target tracking method based on confidence optimization, so as to reduce the actual deployment difficulty of the online multi-target tracking algorithm and improve the tracking performance.

[0010] The present invention provides an online multi-target tracking method based on confidence optimization. The method uses the detection confidence and matching cost corresponding to the trajectory as parameters, iteratively updates the trajectory confidence frame by frame based on the exponential moving average, and constructs a confidence weight matrix based on the trajectory and the trajectory confidence. The method performs weighted optimization on the similarity matrix. The method specifically includes the following steps:

[0011] Step 1: Input the current frame to the detector, and the detector outputs the detection bounding boxes of all tracked targets;

[0012] Step 2: Use the Kalman filter to predict the state of all trajectory bounding boxes in the previous frame at the current frame time;

[0013] Step 3: Calculate the IOU distance between the objects involved in the first association and obtain the similarity matrix U first , and construct the confidence weight matrix W first , using W first with U first Find the Vandermonde product with respect to U first Perform weighted optimization and then use the Hungarian algorithm to obtain the matching relationship, where is the confidence vector of all detections, is the confidence vector of all activated trajectories;

[0014] The objects involved in the first association include all detection bounding boxes and activated tracks. The activated tracks include normal matching tracks and lost tracks.

[0015] Step 4: Calculate the IOU distance between the objects involved in the secondary association and obtain the similarity matrix U second , and construct the confidence weight matrix W second , using W second with U second Find the Vandermonde product with respect to U second Perform weighted optimization and then use the Hungarian algorithm to obtain the matching relationship, where ι is a 1×m unit row vector;

[0016] The objects involved in the secondary association include new trajectories, normal trajectories that were not matched in the first association, and detections that were not matched in the first association;

[0017] In step 5, based on the matching relationship between the detection and trajectory obtained from the two association processes, trajectory management is performed to obtain the trajectory of the current frame, and the trajectory confidence is iteratively updated based on the exponential moving average.

[0018] Furthermore, the trajectory confidence is modeled based on the exponential moving average, that is, the detection confidence corresponding to the trajectory

[0019]

[0020] in, To test confidence, represents the IOU distance between the k-th frame detection box i and the trajectory j prediction box, is the initial frame of trajectory j.

[0021] Furthermore, the first associated problem model is

[0022]

[0023] Among them, g ij is the threshold control coefficient and ε first is the maximum cost threshold for the first association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

[0024] Furthermore, the quadratic correlation problem model is

[0025]

[0026] Among them, g ij is the threshold control coefficient and ε second is the maximum cost threshold of secondary association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

[0027] Furthermore, trajectory management includes the following steps:

[0028] ⑴ Generate new trajectories: For unmatched detections that undergo secondary association, above the confidence threshold All detections are converted to new tracks, which are below the confidence threshold Delete all the detections;

[0029] ⑵ Eliminate inactive tracks: Set a buffer period for lost tracks. Tracks within the buffer period participate in the first association to restore the ID from the lost state. If the lost track still has not restored the ID after the buffer period, the state will be changed to inactive and will no longer participate in the association;

[0030] ⑶ Update track status: For the tracks involved in the association, according to and Update the status of all trajectories. The specific operations are as follows: for continuously associated normal trajectories, continue to maintain them; for new trajectories that have been successfully matched for two consecutive frames, convert them to normal trajectories; for unmatched trajectories within the buffer period, convert them to lost trajectories; for trajectories that have recovered from the lost state, convert them to recovered trajectories.

[0031] The present invention also protects an online multi-target tracker, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the above-mentioned online multi-target tracking method based on confidence optimization is implemented, and a computer-readable storage medium having the computer program stored thereon, wherein the computer program can be executed by the processor to implement the various steps of the above-mentioned online multi-target tracking method based on confidence optimization.

[0032] The beneficial effects of the present invention are as follows: 1. Unlike previous trajectory confidence models, the present invention combines common intermediate quantities in MOT tasks (detection confidence and matching cost) and uses the EMA model for iterative update, with low computational complexity and no need for additional parameter solution, which is beneficial for online tasks; 2. Unlike the common practice of TBD trackers in the past, the present invention does not set any confidence threshold for objects participating in the association. During the first association, the outputs and activated trajectories of all detectors are allowed to participate in the association, which helps to recover interrupted trajectories from ultra-low-score detections, reduces the complexity of the hyperparameter space, and facilitates actual deployment; 3. The online tracker based on confidence optimization proposed in the present invention consists of the first association, the second association, and the trajectory management. The three parts are progressively implemented to complete data association. The modular structure is conducive to subsequent continuous improvement and compatibility with other modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the main process of the online multi-target tracking method based on confidence optimization proposed in this invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described to better illustrate the principles of the invention and its practical application, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for specific applications.

[0035] Example 1

[0036] An online multi-target tracking method based on confidence optimization is proposed. The detection confidence and matching cost corresponding to the trajectory are used as parameters. The trajectory confidence is iteratively updated frame by frame based on the exponential moving average (EMA). A confidence weight matrix is constructed by combining the trajectory and trajectory confidence, and the similarity matrix is weightedly optimized.

[0037] Trajectory confidence is a quantitative measure of a trajectory's reliability, primarily dependent on two factors: the matching cost with the corresponding detection and the degree of target occlusion. A lower matching cost indicates that the trajectory more closely matches the motion model's predictions; a lower degree of target occlusion indicates that the trajectory is less susceptible to occlusion noise. The matching cost can be expressed as various distances between the trajectory and the detection, while the degree of occlusion is negatively correlated with the detection's confidence score. Furthermore, trajectory confidence comprehensively considers all states of the trajectory within a time window, favoring recent performance.

[0038] This embodiment uses the IOU distance between the trajectory and the corresponding detection and the detection confidence as parameters, and models the trajectory confidence based on EMA, that is, the detection confidence corresponding to the trajectory.

[0039]

[0040] in, To test confidence, represents the IOU distance between the k-th frame detection box i and the trajectory j prediction box, is the initial frame of trajectory j.

[0041] The online multi-target tracking method based on confidence optimization proposed in this embodiment is divided into three main parts: first association, second association and trajectory management. Figure 1 shown.

[0042] 1. First association

[0043] 1. Construct similarity matrix

[0044] The linear Kalman filter is used as the motion model to predict the positions of all activated trajectories in the current frame, and then the IOU distance matrix U is constructed.first , which is the similarity matrix.

[0045] In addition, a confidence vector for all detections is constructed and the confidence vectors of all activated trajectories The confidence weight matrix W is obtained by calculating the outer product of s and θ first , and then compare it with U first Calculate the Vandermonde product to obtain the optimized first-association cost matrix

[0046] 2. Model and solution of association problem

[0047]

[0048] Among them, g ij is the threshold control coefficient and ε first is the maximum cost threshold for the first association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

[0049] Solve the problem model by the Hungarian algorithm and obtain the approximate optimal matching matrix of the first association And output matching tracks, unmatched tracks and unmatched detection sets.

[0050] Second, secondary correlation

[0051] This embodiment adopts an isolated matching strategy for new tracks, as these tracks are likely false positives. If matched alongside normal tracks, this could result in numerous association errors. Furthermore, secondary association provides a reconnection opportunity for normal tracks that were not matched in the initial association. Lost tracks cannot participate in secondary association and can only be restored in the initial association. In summary, secondary association involves new tracks, normal tracks not matched in the initial association, and detections not matched in the initial association.

[0052] 1. Construct similarity matrix

[0053] As with the first association, we first construct the IOU distance matrix U between the trajectory and the detection second , and then the confidence weight matrix W of the secondary association is obtained second , and finally the quadratic association cost matrix is obtained Where ι is a 1×m unit row vector whose elements are all 1. Its function is to expand s into a matrix and unify the matrix calculation dimensions.

[0054] It is worth noting that since the new trajectory is born relatively soon, its trajectory confidence cannot reliably reflect the credibility of the trajectory, so W second Only relevant for detection confidence.

[0055] 2. Model and solution of association problem

[0056]

[0057] Among them, g ij is the threshold control coefficient and ε second is the maximum cost threshold for the first association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

[0058] The difference between the secondary association and the primary association is that the trajectory confidence clues are not introduced. Because the new trajectory is born soon, the trajectory confidence model has low adaptability to historical data and cannot accurately capture the changing trend of the trajectory, making it difficult to correctly reflect its trajectory confidence.

[0059] Solve the problem model by the Hungarian algorithm and obtain the approximate optimal matching matrix of the first association And output matching tracks, unmatched tracks and unmatched detection sets.

[0060] 3. Trajectory Management

[0061] 1. Generate new trajectories: For unmatched detections that undergo secondary association, above the confidence threshold All detections are converted to new tracks, which are below the confidence threshold All detections are deleted here. 0.7 is recommended.

[0062] 2. Eliminate inactive tracks: Set a buffer period for lost tracks. Tracks within the buffer period participate in the first association to restore the ID from the lost state. If the lost track still has not restored the ID after the buffer period, the state will be changed to inactive and will no longer participate in the association;

[0063] 3. Update track status: For the tracks involved in the association, according to and Update the status of all trajectories. The specific operations are as follows: for continuously associated normal trajectories, continue to maintain them; for new trajectories that have successfully matched two consecutive frames, convert them to normal trajectories; for unmatched trajectories within the buffer period, convert them to lost trajectories; for trajectories that have recovered from the lost state, convert them to recovered trajectories;

[0064] 4. Iteratively update the trajectory confidence based on the exponential moving average.

[0065] The online multi-target tracking method proposed in this paper was tested on two popular benchmarks, the MOT17 and MOT20 test sets. The test results, shown in Table 1, show that on the MOT17 test set, the method achieved 63.0 HOTA and 77.1 IDF1, comparable tracking performance to recent similar methods, with inference speeds meeting real-time requirements. On the more complex and crowded MOT20 test set, the method achieved 62.4 HOTA and 76.1 IDF1, outperforming recent similar methods and avoiding the tedious hyperparameter tuning process, making it attractive for practical applications, especially in complex and crowded scenarios.

[0066] Table 1

[0067]

[0068] Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without making any creative work should fall within the scope of protection of the present invention.

Claims

1. An online multi-target tracking method based on confidence optimization, characterized in that: Using the detection confidence and matching cost corresponding to the trajectory as parameters, the trajectory confidence is iteratively updated frame by frame based on the exponential moving average. The confidence weight matrix is constructed by combining the trajectory and the trajectory confidence, and the similarity matrix is weighted optimized. The specific steps include: Step 1: Input the current frame to the detector, and the detector outputs the detection bounding boxes of all tracked targets; Step 2: Use the Kalman filter to predict the state of all trajectory bounding boxes in the previous frame at the current frame time; Step 3: Calculate the IOU distance between the objects involved in the first association and obtain the similarity matrix U first , and construct the confidence weight matrix W first , using W first with U first Find the Vandermonde product with respect to U first Perform weighted optimization and then use the Hungarian algorithm to obtain the matching relationship, where is the confidence vector of all detections, is the confidence vector of all activated trajectories; The objects involved in the first association include all detection bounding boxes and activated tracks. The activated tracks include normal matching tracks and lost tracks. Step 4: Calculate the IOU distance between the objects involved in the secondary association and obtain the similarity matrix U second , and construct the confidence weight matrix W second , using W second with U second Find the Vandermonde product with respect to U second Perform weighted optimization and then use the Hungarian algorithm to obtain the matching relationship, where ι is a 1×m unit row vector; The objects involved in the secondary association include new trajectories, normal trajectories that were not matched in the first association, and detections that were not matched in the first association; In step 5, based on the matching relationship between the detection and trajectory obtained from the two association processes, trajectory management is performed to obtain the trajectory of the current frame, and the trajectory confidence is iteratively updated based on the exponential moving average.

2. The online multi-target tracking method based on confidence optimization according to claim 1, characterized in that: Modeling trajectory confidence based on exponential moving average, that is, the detection confidence corresponding to the trajectory in, To test confidence, represents the IOU distance between the k-th frame detection box i and the trajectory j prediction box, is the initial frame of trajectory j.

3. The online multi-target tracking method based on confidence optimization according to claim 1, characterized in that: The problem model for the first association is Among them, g ij is the threshold control coefficient and ε first is the maximum cost threshold for the first association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

4. The online multi-target tracking method based on confidence optimization according to claim 1, characterized in that: The problem model of quadratic correlation is Among them, g ij is the threshold control coefficient and ε second is the maximum cost threshold of secondary association, a ij is a binary number, a ij =1 means detection i is associated with track j, a ij =0 means that detection i is irrelevant to trajectory j.

5. The online multi-target tracking method based on confidence optimization according to claim 1, characterized in that: Trajectory management includes the following steps: (1) Generate new trajectories: For unmatched detections that undergo secondary association, above the confidence threshold All detections are converted to new tracks, which are below the confidence threshold Delete all the detections; (2) Eliminate inactive trajectories: Set a buffer period for lost trajectories. Trajectories within the buffer period participate in the first association to recover the ID from the lost state. If the lost trajectory fails to recover the ID after the buffer period, the state becomes inactive and no longer participates in the association. (3) Update trajectory status: For the trajectory involved in the association, according to and Update the status of all trajectories. The specific operations are as follows: for continuously associated normal trajectories, continue to maintain them; for new trajectories that have been successfully matched for two consecutive frames, convert them to normal trajectories; for unmatched trajectories within the buffer period, convert them to lost trajectories; for trajectories that have recovered from the lost state, convert them to recovered trajectories.

6. An online multi-target tracker, characterized in that The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method implements the online multi-target tracking method based on confidence optimization according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement each step of the online multi-target tracking method based on confidence optimization according to any one of claims 1 to 5.

Images