Multi-target tracking method, system, device and storage medium based on deep association

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a multi-target tracking method, system, device and storage medium based on depth association. For image data in a video, the detection accuracy and appearance clarity of a detection target are fully considered, and four-layer association is performed according to the detection confidence and classification confidence of the detection target in combination with a depth value. Among them, according to the depth value of the detection target in a detection set and the depth value of the track target in a tracking set, a depth interval is divided, and targets under dense occlusion are processed to divide the target set under dense occlusion into sparse subsets with different depth levels, thereby achieving comprehensive association and matching of detection targets in different situations.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-target tracking method, system, device and storage medium based on depth association. Background Art

[0002] Multi-object tracking (MOT) is a key technology in the field of computer vision, and has a wide range of applications in security monitoring, autonomous driving, and intelligent transportation. Currently, tracking based on detection (TBD) is the mainstream MOT paradigm, and the core of MOT is to accurately associate detection results with tracked targets. This paradigm mainly includes two subtasks: target detection and data association.

[0003] Most existing TBD-based MOT methods are innovative developments in data association, focusing on designing better cost matrices or matching strategies. However, the performance of object detection directly affects the quality of tracking. As a key step before data association, object detection can provide richer location information of the target and often works in conjunction with motion cues to achieve more reliable target tracking. Although motion cues are widely used due to their efficiency and simplicity, the application of motion models is extremely challenging in scenes with frequent camera movements.

[0004] First, during the tracking process, it is easy to have inaccurate detection and recognition of the target's position in the image, affecting the accuracy of target detection and the clarity of appearance. Second, it is easy to have inaccurate or unreliable associations between targets, which affects the stability and continuity of tracking. Third, in order to improve the quality and accuracy of data association, it is usually adopted to introduce additional appearance cues or perform frame-by-frame camera motion compensation (CMC) on videos taken by a mobile camera. Although these additional measures have achieved effective results, for real-time multi-target tracking (MOT), this method also brings operational complexity such as parameter adjustment required after the introduction of additional measures, algorithm implementation, system configuration and deployment, and performance evaluation. Summary of the invention

[0005] The present invention provides a multi-target tracking method, system, device and storage medium based on depth association.

[0006] The technical solution of the present invention is as follows:

[0007] The present invention provides a multi-target tracking method based on depth association, comprising the following steps:

[0008] S1: Obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target;

[0009] S2: after the detection targets in the image data of the current frame are formed into a detection set and the tracking tracks of all frames before the current frame are formed into a tracking set, the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set are calculated respectively, and the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals;

[0010] S3: Based on the detection depth interval and the trajectory depth interval, the depth values ​​are prioritized from small to large. According to the priority, the detection targets and trajectory targets in the same depth interval are associated by the Hungarian algorithm. Specifically,

[0011] S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association;

[0012] S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association;

[0013] S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association;

[0014] S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association;

[0015] S4: Store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, perform the S1 operation until all frames in the video are processed, and obtain the multi-target tracking trajectory based on deep association.

[0016] In S2 of the present invention, the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, specifically,

[0017] Obtain the minimum depth value and the maximum depth value of the detection target in the detection set, and evenly divide the distance between the minimum depth value and the maximum depth value of the detection target in the detection set to obtain a number of detection depth intervals; wherein the minimum depth value of the detection target in the detection set is the minimum value among the depth values ​​of all the detection targets in the detection set; the maximum depth value of the detection target in the detection set is the maximum value among the depth values ​​of all the detection targets in the detection set;

[0018] The minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set are obtained, and the distance between the minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set is evenly divided to obtain a number of trajectory depth intervals; wherein the minimum depth value of the tracking trajectory target in the tracking set is the minimum value among the depth values ​​of all the tracking trajectory targets in the tracking set; the maximum depth value of the tracking trajectory target in the tracking set is the maximum value among the depth values ​​of all the tracking trajectory targets in the tracking set.

[0019] In S3 of the present invention, the detection target and the track target in the same depth interval are associated by the Hungarian algorithm, and also include, combining motion information and / or appearance information to perform association matching, specifically,

[0020] The first level association in S31 and the second level association in S32 both use the Mahalanobis distance cost matrix C 1 To perform association matching: , where is the Mahalanobis distance between the detection frame of the detection target and the tracking frame of the trajectory target in the image data of the current frame, is the detection confidence of the detection box of the detection target in the image data of the current frame;

[0021] The third layer association is performed in S33, using the cost matrix C 3 To perform association matching: , where is the similarity in appearance between the detection target and the track target in the image data of the current frame represented by the cosine distance;

[0022] The fourth level association is performed in S34, using the cost matrix C 4 To perform association matching: , where C 1 is the Mahalanobis distance cost matrix of the first-layer association or the second-layer association, C 3 is the cost matrix associated with the third layer, and β is the weight used to balance the similarity between motion and appearance.

[0023] The tracking track before the detection target in the image data of the current frame in S2 of the present invention constitutes a tracking set, and also includes performing motion noise processing on the tracking track before the detection target in the image data of the current frame, specifically,

[0024] Get the width and height of the track target detection box, determine the state vector, measurement vector, process noise covariance and measurement noise covariance, model the motion state of the track target, build a noise model according to the acceleration change of the track target, and combine the process noise covariance to process the motion noise of the tracking trajectory before the detection target in the image data of the current frame.

[0025] The method of combining appearance information for association matching in the present invention further includes:

[0026] The appearance state of the matched trajectory is updated through exponential moving average processing, and the exponential moving average processing uses the exponential moving average processing formula as follows: , where is the appearance state of the i-th track in the k-th frame, is the appearance state of the i-th track in the k-1 frame, The appearance embedding of the object detected in the kth frame, is the momentum term.

[0027] The S4 of the present invention also includes processing of unmatched trajectories and unmatched targets after the fourth layer association, specifically,

[0028] For the unmatched trajectories after the fourth level association, they are removed;

[0029] For the unmatched detection targets after the fourth layer association, if the detection confidence of the target is greater than the detection confidence threshold, it is initialized as a new trajectory and stored; if the detection confidence of the target is not greater than the detection confidence threshold, it is removed.

[0030] The Mahalanobis distance of the present invention is obtained by the following steps:

[0031] Based on the detection frame of the detection target and the tracking frame of the track target in the image data of the current frame, after residual processing, the residual value is obtained, and the residual covariance matrix is ​​calculated according to the measurement noise covariance. According to the residual value and the residual covariance matrix, normalization processing is performed to obtain the Mahalanobis distance.

[0032] The present invention also provides a multi-target tracking system based on depth association, comprising:

[0033] Preprocessing module: used to obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target;

[0034] Depth interval division module: used to form a detection set with the detection targets in the image data of the current frame, and form a tracking set with the tracking tracks of all frames before the current frame, and then respectively calculate the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, and divide the depth interval according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals;

[0035] Association module: Based on the detection depth interval and trajectory depth interval, the depth values ​​are prioritized from small to large, and the detection targets and trajectory targets in the same depth interval are associated according to the priority. Specifically,

[0036] S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association;

[0037] S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association;

[0038] S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association;

[0039] S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association;

[0040] Multi-target tracking trajectory generation module based on deep association: used to store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, it enters the preprocessing module until all frames in the video are processed, and the multi-target tracking trajectory based on deep association is obtained.

[0041] The present invention also provides a multi-target tracking device based on deep association, comprising a processor and a memory, wherein the processor implements the multi-target tracking method based on deep association when executing a computer program stored in the memory.

[0042] The present invention also provides a multi-target tracking storage medium based on depth association, which is used to store a computer program, wherein the computer program implements the multi-target tracking method based on depth association when executed by a processor.

[0043] Beneficial effects: The multi-target tracking method based on deep association provided by the present invention fully considers the detection accuracy and appearance clarity of the detection target, and performs four-layer association according to the detection confidence and classification confidence of the detection target in combination with the depth value, thereby realizing comprehensive association and matching of the detection targets in different situations;

[0044] The present invention divides depth intervals according to the depth values ​​of the detection targets in the detection set and the depth values ​​of the track targets in the tracking set, processes targets under dense occlusion, and divides the target set under dense occlusion into sparse subsets with different depth levels, thereby reducing the collision probability between tracks with similar positions but different depths during the association process;

[0045] The present invention uses Mahalanobis distance for association to more accurately match the target, thereby improving tracking efficiency and the quality and accuracy of data association;

[0046] The present invention significantly reduces the computational burden associated with camera motion adjustment by uniformly applying the same process compensation factor in the entire video frame sequence, and effectively improves the performance of the multi-target tracking method. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1This is a flowchart of the multi-target tracking method based on deep association in this application.

[0048] Figure 2 Schematic diagram of the geometric depth estimation method. DETAILED DESCRIPTION

[0049] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0050] The present invention provides a multi-target tracking method based on deep association, such as Figure 1 As shown, the following steps are included:

[0051] S1: Obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target;

[0052] S2: after the detection targets in the image data of the current frame are formed into a detection set and the tracking tracks of all frames before the current frame are formed into a tracking set, the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set are calculated respectively, and the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals;

[0053] S3: Based on the detection depth interval and the trajectory depth interval, the depth values ​​are prioritized from small to large. According to the priority, the detection targets and trajectory targets in the same depth interval are associated by the Hungarian algorithm. Specifically,

[0054] S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association;

[0055] S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association;

[0056] S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association;

[0057] S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association;

[0058] S4: Store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, perform the S1 operation until all frames in the video are processed, and obtain the multi-target tracking trajectory based on deep association.

[0059] The present invention provides a multi-target tracking method based on depth association, which fully considers the detection accuracy and appearance clarity of the detection target, performs four-layer association according to the detection confidence and classification confidence of the detection target in combination with the depth value, thereby realizing comprehensive association and matching of the detection targets in different situations.

[0060] S1: Obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target.

[0061] The processing may detect a detection frame of a detection target in the current frame image through a target detection algorithm, convert the detected detection frame of the target into a target detection result, and output the result.

[0062] The detection confidence considers the information of the entire detection frame, including the position and appearance, and is used to measure the probability that the detection frame contains the detection target.

[0063] The classification confidence is the predicted probability of the category label of the detection target, which is used to measure the appearance clarity of the detection target.

[0064] For example, the above process can be implemented using the YOLOX detector, which outputs a bounding box (x 1 ,y 1 , x 2 ,y 2 ), classification confidence and detection confidence.

[0065] S2: After the detection targets in the image data of the current frame are formed into a detection set and the tracking trajectories of all frames before the current frame are formed into a tracking set, the depth value of each detection target in the detection set and the depth value of each tracking trajectory target in the tracking set are calculated respectively, and the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking trajectory target in the tracking set to obtain a plurality of detection depth intervals and trajectory depth intervals.

[0066] In addition, in S2, the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, specifically,

[0067] Obtain the minimum depth value and the maximum depth value of the detection target in the detection set, and evenly divide the distance between the minimum depth value and the maximum depth value of the detection target in the detection set to obtain a number of detection depth intervals; wherein the minimum depth value of the detection target in the detection set is the minimum value among the depth values ​​of all the detection targets in the detection set; the maximum depth value of the detection target in the detection set is the maximum value among the depth values ​​of all the detection targets in the detection set;

[0068] The minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set are obtained, and the distance between the minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set is evenly divided to obtain a number of trajectory depth intervals; wherein the minimum depth value of the tracking trajectory target in the tracking set is the minimum value among the depth values ​​of all the tracking trajectory targets in the tracking set; the maximum depth value of the tracking trajectory target in the tracking set is the maximum value among the depth values ​​of all the tracking trajectory targets in the tracking set.

[0069] For example, first, calculate the detection set The minimum depth value and the maximum depth value . Then, and The distance intervals between them are evenly divided to form k detection depth intervals , k detection depth intervals represent k different geometric depth levels. In the same way, the tracking set is obtained The minimum depth value and the maximum depth value , and k trajectory depth intervals .

[0070] Regarding the division of depth intervals, an appropriate k value is selected so that the depth interval can fully cover the depth value range of the target and describe the distribution of the target in sufficient detail. Targets under dense occlusion are divided into different depth intervals. Even if the targets under dense occlusion are close in spatial position, they can be distinguished in depth, so that the association between the detected target and the trajectory during the association process is more accurate and reliable.

[0071] Furthermore, the influence of the geometric depth layer k value on the association performance was studied on the MOT17 validation set and the MOT20 validation set. The results are shown in Table 1. In Table 1, HOTA: is a target tracking performance metric that takes into account the accuracy of detection and association. It is calculated by weighting the correct association, missed detection, and false detection of detection. The higher the HOTA value, the better the target tracking performance.

[0072] AssA: The ratio of the number of correctly associated targets to the total number of associations in target tracking. The higher the AssA value, the higher the accuracy of target association.

[0073] IDF1: An identification F1 score that measures the accuracy of the target ID. It combines the precision and recall of the ID. The level of IDF1 reflects the accuracy of the target ID, that is, the ability to correctly identify the target ID.

[0074] IDs: Indicates the fragmentation of the target ID, that is, the discontinuity of the target ID due to incorrect associations. The lower the IDs, the better the continuity of the target ID and the fewer incorrect associations.

[0075] Table 1 Effect of geometric depth level k value on association performance

[0076]

[0077] Specifically, the geometric depth levels of the association process are set to 2, 4, 6, and 8, respectively. On the MOT17 validation set, the association performance gradually improves with the increase of the geometric depth level, which indicates that a higher number of geometric depth levels is more conducive to processing targets under dense occlusion and improving the association performance. However, when the geometric depth level value exceeds 6, the association performance does not improve significantly, which indicates that when the geometric depth level is 6, the division of the occluded target set is sufficiently dispersed and sparse to meet the association requirements. In addition, the same experiment was performed on the MOT20 validation set with the same settings, and the results were consistent with those of the MOT17 validation set. That is, the larger the geometric depth level, the better the improvement in association performance for cases such as densely occluded targets, and the maximum geometric depth level is 6.

[0078] The present invention divides depth intervals according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, and processes targets under dense occlusion, so as to divide the target set under dense occlusion into sparse subsets with different depth levels, so that the originally dense crowd distribution becomes more dispersed, and the collision probability between tracks with similar positions but different depths is reduced during the association process.

[0079] In addition, the depth value is obtained by a geometric depth estimation method, such as Figure 2 Specifically,

[0080] First, obtain the position point P where the object contacts the ground platform in 3D space 0 , the point P 0 Project it onto the image platform F of the camera and get the corresponding projection point P 1 .

[0081] Secondly, from point P 1 Draw a vertical line that is aligned with the bottom edge L of the image. b Intersect at point D. Project point P 1 The Euclidean distance between the intersection point D and the object is defined as the geometric depth estimate of the object, which can also be regarded as the distance between P 0 To D 0 The distance in three-dimensional space is projected onto the image platform F. In the projection transformation, P 0 To D 0 The Euclidean distance between the projection points P is positively correlated with the geometric depth estimate. 1 The pixel coordinates of the intersection point D are expressed as and The calculation formula for geometric depth estimation is as follows:

[0082]

[0083] In the formula, Represents the depth value. Although the geometric depth estimation value cannot accurately represent the actual depth of the target in the 3D space, it can effectively measure the relative depth relationship between different objects on the same ground, which also provides a basis for dense crowd segmentation at the depth level.

[0084] In existing studies, the target's motion model is usually considered as a constant velocity model that does not consider the noise caused by camera motion. However, camera motion is likely to introduce significant noise, which affects tracking performance.

[0085] Preferably, the tracking tracks before the detection target in the image data of the current frame in S2 constitute a tracking set, and further include performing motion noise processing on the tracking tracks before the detection target in the image data of the current frame, specifically,

[0086] Get the width and height of the track target detection box, determine the state vector, measurement vector, process noise covariance and measurement noise covariance, model the motion state of the track target, build a noise model according to the acceleration change of the track target, and combine the process noise covariance to process the motion noise of the tracking trajectory before the detection target in the image data of the current frame.

[0087] First, the motion of objects on the ground plane is modeled by using a discrete Kalman filter with a constant velocity model to better capture the basic nature of their motion patterns.

[0088] The state vector is,

[0089]

[0090] In the formula, is the state vector of the detected target at time k;

[0091] and Respectively represent the horizontal and vertical coordinates of the position of the detection target in the center of the image;

[0092] and Respectively represent the width and height of the detection target;

[0093] and Respectively represent the speed of the detection target in the x and y directions;

[0094] and Respectively represent the rate of change of the width and height of the detected target.

[0095] The measurement vector is,

[0096]

[0097] In the formula, is the measurement vector of the target detected at time k;

[0098] and Respectively represent the horizontal and vertical coordinate measurement values ​​of the detection target at the center of the image;

[0099] and Respectively represent the measured values ​​of the width and height of the detected object.

[0100] The process noise covariance is,

[0101]

[0102] In the formula, is the process noise covariance matrix, which represents the variance of the prediction error;

[0103] : is the width of the target detection box in the state vector The square of the variance of represents the uncertainty of the prediction of the target detection box width at time k-1;

[0104] : is the height of the target detection box in the state vector The square of the variance represents the uncertainty of the prediction of the target detection box height at time k-1;

[0105] : is the square of the variance of the target detection box width change rate in the state vector, indicating the uncertainty of the prediction of the target detection box width change rate at time k-1;

[0106] : is the square of the variance of the target detection box height change rate in the state vector, indicating the uncertainty of the prediction of the target detection box height change rate at time k-1;

[0107] represents the partial derivative with respect to position, represents the partial derivative with respect to velocity.

[0108] The measurement noise covariance is,

[0109]

[0110] In the formula, is the measurement noise covariance matrix, which represents the variance of the measurement error;

[0111] : The width of the target detection box in the measurement vector The square of the variance represents the difference between the measured value and the predicted value, that is, the variance of the measurement error of the target detection box width at time step k;

[0112] : To measure the height of the target detection box in the vector The square of the variance represents the difference between the measured value and the predicted value, that is, the variance of the measurement error of the target detection box height at time step k;

[0113] Represents the partial derivatives of the width and height of the target detection box in the measurement vector.

[0114] The present invention selects =0.05, =0.00625 and = 0.05 to match the frame rate of the motion model.

[0115] Secondly, we assume that the main source of tracking noise is the acceleration caused by camera motion and represent it through a motion model as follows:

[0116]

[0117] in, and They represent the position and velocity changes under the influence of noise, represents the change in acceleration caused by camera motion, Indicates the time interval between two image frames.

[0118] This formula describes the relationship between noise in position and velocity changes, and more accurately simulates the noise in the tracking process by considering the acceleration changes caused by camera motion.

[0119] Then, the above formula is expressed in matrix form as follows:

[0120]

[0121] The noise model is expressed in matrix form, where the matrix The relationship between the position and velocity changes of noise in each direction is described, which helps to more effectively model and handle the noise introduced by camera motion.

[0122] For the constant velocity model, the process noise covariance matrix used with the Kalman filter can be expressed as follows:

[0123]

[0124] In the formula, and Represents the process compensation factors along the x and y axes, respectively, used to deal with motion noise caused by camera tilt and rotation.

[0125] The present invention significantly reduces the computational burden associated with camera motion adjustment by uniformly applying the same process compensation factor in the entire video frame sequence, and effectively improves the performance of the multi-target tracking method.

[0126] "Uniformly apply the same compensation parameters" means using the same parameters for camera motion compensation throughout the entire video frame sequence. Usually, the movement of the camera will cause the position of the target in the image to change, so camera motion compensation is required to ensure the accuracy of target tracking. The traditional method is to calculate the camera motion parameters for each frame separately, and then apply these parameters to adjust the target position. However, this application uses the same compensation parameters throughout the entire video frame image sequence, rather than calculating them separately for each frame.

[0127] The present invention uses a Kalman filter to predict the state of an existing trajectory and obtain the predicted position of the target. The process noise covariance matrix of the Kalman filter is used to represent the noise introduced by the camera motion. By combining the noise model and the Kalman filter, the state of the target can be estimated more accurately and the correlation performance of the multi-target tracking method can be improved.

[0128] S3: Based on the detection depth interval and the trajectory depth interval, the depth values ​​are prioritized from small to large. According to the priority, the detection targets and trajectory targets in the same depth interval are associated through the Hungarian algorithm.

[0129] The Hungarian algorithm can be used to effectively find the global optimal solution, that is: in each layer of association, according to the detection confidence and classification confidence of the detection target, the corresponding cost matrix is ​​selected, the cost matrix is ​​used as input, and the Hungarian algorithm is used to solve the assignment problem to find the optimal match that minimizes the total cost.

[0130] Specifically,

[0131] S31: Obtain the detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and perform the first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix.

[0132] In terms of motion information, the Mahalanobis distance cost matrix C is used 1 Associative matching as a cost matrix:

[0133]

[0134] In the formula, is the Mahalanobis distance between the detection frame of the detection target and the tracking frame of the trajectory target in the image data of the current frame, It is the detection confidence of the detection box of the detected target in the image data of the current frame.

[0135] In crowded motion scenes, some target detection frames may occlude each other, which may cause the correct detection frame to not match. To solve this problem, the Mahalanobis distance cost matrix is ​​used as the cost matrix to calculate the matching cost, and the detection confidence of the detection frame of the corresponding detection target is multiplied by the Mahalanobis distance, that is, the detection confidence of the detection frame of the corresponding detection target and the distance between the detection frame of the detection target in the image data of the current frame and the tracking frame of the track target are taken into account in the matching process, which improves the matching accuracy and reduces the cost matrix corresponding to the detection with low detection confidence, making the cost matrix more accurate and reasonable.

[0136] After completing the first layer of association, the matched tracks after the first layer of association, the unmatched tracks after the first layer of association, and the unmatched detection targets after the first layer of association are obtained.

[0137] Among them, the new tracks in the unmatched tracks after the first layer of association will be removed. The "new tracks" refer to the tracks that have not been matched with any detection frame identified in the first layer of association. These tracks are the target positions predicted by the tracker based on previous observation data. When the tracker cannot match these predicted target positions with the current detection frame, these predicted target positions are marked as "new tracks". The remaining unmatched tracks after the first layer of association and the unmatched detection targets after the first layer of association will enter the second layer of association.

[0138] S32: Obtain the detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and perform second-layer association with the unmatched trajectories and unmatched detection targets after the first-layer association using the Mahalanobis distance cost matrix.

[0139] Since the detection target to be matched in the second layer association has an accurate position but an unclear appearance, the Mahalanobis distance cost matrix C is also used in the second layer association 1 As a cost matrix, association matching is performed.

[0140] After completing the second level association, the matched tracks after the second level association, the unmatched tracks after the second level association, and the unmatched detection targets after the second level association are obtained. Among them, the unmatched tracks after the second level association will enter the third level association, and the unmatched detection targets after the second level association will enter the fourth level association.

[0141] S33: Obtain the detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and perform third-layer association with the unmatched tracks after the second-layer association.

[0142] Since the detection to be matched in this stage has a clear appearance but the detection is not accurate, the appearance information cost matrix C is used in the association matching. 3 As a cost matrix:

[0143]

[0144] In the formula, It is the similarity in appearance between the detection target and the track target in the image data of the current frame represented by the cosine distance.

[0145] Furthermore, the combination of appearance information for association matching also includes:

[0146] The appearance state of the matched trajectory is updated through exponential moving average processing, and the exponential moving average processing uses the exponential moving average processing formula as follows:

[0147]

[0148] In the formula, is the appearance state of the ith track in the kth frame, indicating the appearance features of the track in the current frame, is the appearance state of the i-th track in the k-1 frame, that is, the appearance feature of the track in the previous frame, The appearance embedding of the object detected in the kth frame, is the momentum term and takes the value of 0.9.

[0149] After completing the third-layer association, the matched tracks after the third-layer association, the unmatched tracks after the third-layer association, and the unmatched detection targets after the third-layer association are obtained. Among them, the unmatched tracks after the third-layer association and the unmatched detection targets after the third-layer association will enter the fourth-layer association.

[0150] S34: Acquire the detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, the detection targets that are not matched after the second layer association, the trajectories that are not matched after the third layer association, and the detection targets that are not matched after the third layer association, and perform the fourth layer association.

[0151] Since the matching in this stage mainly involves detecting inaccurate and ambiguous objects, the cost matrix C is used 4 Perform association matching to weigh motion and appearance similarity:

[0152]

[0153] In the formula, C 1 is the Mahalanobis distance cost matrix of the first-layer association or the second-layer association, C 3is the cost matrix associated with the third layer, and β is the weight used to balance the similarity between motion and appearance. The value range of β is usually between 0 and 1. When β is close to 0, it means that more weight is assigned to appearance similarity, while the influence of motion similarity is relatively small. On the contrary, when β is close to 1, the importance of motion similarity is higher, while the influence of appearance similarity is relatively small.

[0154] After completing the fourth-layer association, the matched tracks after the fourth-layer association, the unmatched tracks after the fourth-layer association, and the unmatched detection targets after the fourth-layer association are obtained.

[0155] In image plane motion modeling, Intersection over Union (IoU) is the most commonly used distance metric for data association. However, in scenes with high-speed object motion, low frame rate capture, or the presence of a moving camera, the lack of overlap between detection boxes and trajectories makes IoU ineffective. In contrast, by adopting the normalized Mahalanobis distance in ground plane modeling, the problem of poor IoU performance can be effectively solved.

[0156] Furthermore, the Mahalanobis distance is obtained by the following steps:

[0157] Based on the detection frame of the detection target and the tracking frame of the track target in the image data of the current frame, after residual processing, the residual value is obtained, and the residual covariance matrix is ​​calculated according to the measurement noise covariance. According to the residual value and the residual covariance matrix, normalization processing is performed to obtain the Mahalanobis distance.

[0158] Specifically,

[0159] (1) Calculate the residual value :

[0160]

[0161] In the formula, represents the measurements mapped on the ground plane, is the predicted trajectory state, is the observation matrix.

[0162] (2) Calculate the residual covariance matrix :

[0163]

[0164] In the formula, is the prediction covariance matrix, is the observation matrix, is the measurement noise covariance matrix.

[0165] (3) Calculate the normalized Mahalanobis distance :

[0166]

[0167] In the formula, is the residual value, is the observation matrix, Representation Matrix The determinant of is the natural logarithm.

[0168] By using the normalized Mahalanobis distance, the logarithm of the determinant of the residual covariance matrix is ​​included. This ensures that the data association decision is not based solely on the difference between the measurement and prediction, but also fully considers the accuracy and uncertainty of the measurement. Therefore, this will produce more robust and reliable association decisions in object tracking.

[0169] S4: storing the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association,

[0170] In addition, for the unmatched trajectories after the fourth layer association, they are removed;

[0171] For the unmatched detection targets after the fourth layer association, if the detection confidence of the target is greater than the detection confidence threshold, it is initialized as a new trajectory and stored; if the detection confidence of the target is not greater than the detection confidence threshold, it is removed.

[0172] After the current frame number is increased by one, the S1 operation is executed until all frames in the video are processed, and a multi-target tracking trajectory based on depth association is obtained.

[0173] Furthermore, this application adopts a trajectory management strategy specifically for handling trajectories entering and exiting scenes. Four states are defined for trajectories, namely New, Tracked, Lost, and Removed. The New state is suitable for initializing trajectories, corresponding to detection targets with higher detection confidence. After the maximum number of matching frames After a successful match, the track in the New state will change to the Tracked state. When the track in the Tracked state has the maximum number of consecutive matching frames If the match fails, its status will change to Removed. In addition, if a frame in the new track fails to match, its status will also change to Removed.

[0174] By properly managing the trajectory, false negatives and false positives can be effectively avoided, ensuring the accuracy and timeliness of the trajectory status, thereby improving the reliability of the trajectory of objects in the scene.

[0175] The present invention also provides a multi-target tracking system based on depth association, comprising:

[0176] Preprocessing module: used to obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target;

[0177] Depth interval division module: used to form a detection set with the detection targets in the image data of the current frame, and form a tracking set with the tracking tracks of all frames before the current frame, and then respectively calculate the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, and divide the depth interval according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals;

[0178] Association module: Based on the detection depth interval and trajectory depth interval, the depth values ​​are prioritized from small to large, and the detection targets and trajectory targets in the same depth interval are associated according to the priority. Specifically,

[0179] S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association;

[0180] S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association;

[0181] S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association;

[0182] S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association;

[0183] Multi-target tracking trajectory generation module based on deep association: used to store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, it enters the preprocessing module until all frames in the video are processed, and the multi-target tracking trajectory based on deep association is obtained.

[0184] The present invention also provides a multi-target tracking device based on deep association, comprising a processor and a memory, wherein the processor implements the multi-target tracking method based on deep association when executing a computer program stored in the memory.

[0185] The present invention also provides a multi-target tracking storage medium based on depth association, which is used to store a computer program, wherein the computer program implements the multi-target tracking method based on depth association when executed by a processor.

Claims

1. A multi-target tracking method based on deep association, characterized in that: The following steps are involved: S1: Obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target; S2: after the detection targets in the image data of the current frame are formed into a detection set and the tracking tracks of all frames before the current frame are formed into a tracking set, the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set are calculated respectively, and the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals; S3: Based on the detection depth interval and the trajectory depth interval, the depth values ​​are prioritized from small to large. According to the priority, the detection targets and trajectory targets in the same depth interval are associated by the Hungarian algorithm. Specifically, S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association; S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association; S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association; S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association; S4: Store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, perform the S1 operation until all frames in the video are processed, and obtain the multi-target tracking trajectory based on deep association.

2. The multi-target tracking method based on deep association according to claim 1, characterized in that: In S2, the depth interval is divided according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, specifically, Obtain the minimum depth value and the maximum depth value of the detection target in the detection set, and evenly divide the distance between the minimum depth value and the maximum depth value of the detection target in the detection set to obtain a number of detection depth intervals; wherein the minimum depth value of the detection target in the detection set is the minimum value among the depth values ​​of all the detection targets in the detection set; the maximum depth value of the detection target in the detection set is the maximum value among the depth values ​​of all the detection targets in the detection set; The minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set are obtained, and the distance between the minimum depth value and the maximum depth value of the tracking trajectory target in the tracking set is evenly divided to obtain a number of trajectory depth intervals; wherein the minimum depth value of the tracking trajectory target in the tracking set is the minimum value among the depth values ​​of all the tracking trajectory targets in the tracking set; the maximum depth value of the tracking trajectory target in the tracking set is the maximum value among the depth values ​​of all the tracking trajectory targets in the tracking set.

3. The multi-target tracking method based on deep association according to claim 1, characterized in that: In the step S3, the detection target and the track target in the same depth interval are associated by the Hungarian algorithm, and the associated matching is performed in combination with the motion information and / or the appearance information, specifically, The first-layer association in S31 and the second-layer association in S32 both use the Mahalanobis distance cost matrix C1 for association matching: , where is the Mahalanobis distance between the detection frame of the detection target and the tracking frame of the trajectory target in the image data of the current frame, is the detection confidence of the detection box of the detection target in the image data of the current frame; The third layer association is performed in S33, and the cost matrix C3 is used for association matching: , where is the similarity in appearance between the detection target and the track target in the image data of the current frame represented by the cosine distance; The fourth level of association is performed in S34, and the cost matrix C4 is used for association matching: , where C1 is the Mahalanobis distance cost matrix of the first-layer association or the second-layer association, C3 is the cost matrix of the third-layer association, and β is the weight used to balance the similarity between motion and appearance.

4. The multi-target tracking method based on deep association according to claim 3 is characterized in that: The tracking track before the detection target in the image data of the current frame in S2 forms a tracking set, and also includes performing motion noise processing on the tracking track before the detection target in the image data of the current frame, specifically, Get the width and height of the track target detection box, determine the state vector, measurement vector, process noise covariance and measurement noise covariance, model the motion state of the track target, build a noise model according to the acceleration change of the track target, and combine the process noise covariance to process the motion noise of the tracking trajectory before the detection target in the image data of the current frame.

5. The multi-target tracking method based on deep association according to claim 3 is characterized in that: The combination of appearance information for association matching also includes: The appearance state of the matched trajectory is updated through exponential moving average processing, and the exponential moving average processing uses the exponential moving average processing formula as follows: , where is the appearance state of the i-th track in the k-th frame, is the appearance state of the i-th track in the k-1 frame, The appearance embedding of the object detected in the kth frame, is the momentum term.

6. The multi-target tracking method based on deep association according to claim 1, characterized in that: The S4 also includes processing of unmatched trajectories and unmatched targets after the fourth layer association, specifically, For the unmatched trajectories after the fourth level association, they are removed; For the unmatched detection targets after the fourth layer association, if the detection confidence of the target is greater than the detection confidence threshold, it is initialized as a new trajectory and stored; if the detection confidence of the target is not greater than the detection confidence threshold, it is removed.

7. The multi-target tracking method based on deep association according to claim 4 is characterized in that: The Mahalanobis distance is obtained by following the steps below: Based on the detection frame of the detection target and the tracking frame of the track target in the image data of the current frame, after residual processing, the residual value is obtained, and the residual covariance matrix is ​​calculated according to the measurement noise covariance. According to the residual value and the residual covariance matrix, normalization processing is performed to obtain the Mahalanobis distance.

8. A multi-target tracking system based on deep association, characterized in that: include: Preprocessing module: used to obtain the image data of the current frame in the video, process it, and obtain the detection frame, detection confidence, and classification confidence of the detection target; Depth interval division module: used to form a detection set with the detection targets in the image data of the current frame, and form a tracking set with the tracking tracks of all frames before the current frame, and then respectively calculate the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, and divide the depth interval according to the depth value of each detection target in the detection set and the depth value of each tracking track target in the tracking set, so as to obtain a plurality of detection depth intervals and track depth intervals; Association module: Based on the detection depth interval and trajectory depth interval, the depth values ​​are prioritized from small to large, and the detection targets and trajectory targets in the same depth interval are associated according to the priority. Specifically, S31: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing first-layer association with the tracking trajectories of all frames before the current frame using the Mahalanobis distance cost matrix, and obtaining trajectories matched after the first-layer association, trajectories not matched after the first-layer association, and detection targets not matched after the first-layer association; S32: acquiring detection targets whose detection confidence is greater than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and performing second-layer association with unmatched trajectories and unmatched detection targets after first-layer association using the Mahalanobis distance cost matrix to obtain matched trajectories after second-layer association, unmatched trajectories after second-layer association, and unmatched detection targets after second-layer association; S33: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is greater than the classification confidence threshold in the image data of the current frame, and performing third-layer association with unmatched trajectories after second-layer association to obtain matched trajectories after third-layer association, unmatched trajectories after third-layer association, and unmatched detection targets after third-layer association; S34: acquiring detection targets whose detection confidence is less than the detection confidence threshold and whose classification confidence is less than the classification confidence threshold in the image data of the current frame, and detection targets that are not matched after the second layer association, trajectories that are not matched after the third layer association, and detection targets that are not matched after the third layer association, and performing fourth layer association to obtain trajectories that are matched after the fourth layer association, trajectories that are not matched after the fourth layer association, and detection targets that are not matched after the fourth layer association; Multi-target tracking trajectory generation module based on deep association: used to store the trajectories matched after the first layer association, the trajectories matched after the second layer association, the trajectories matched after the third layer association, and the trajectories matched after the fourth layer association. After the current frame number is increased by one, it enters the preprocessing module until all frames in the video are processed, and the multi-target tracking trajectory based on deep association is obtained.

9. A multi-target tracking device based on deep association, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the multi-target tracking method based on deep association as described in any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the multi-target tracking method based on depth association as described in any one of claims 1 to 7 is implemented.

Images