A multi-target tracking method, device and medium

Abstract

Embodiments of the application disclose a multi-target tracking method, device and medium. A feature correlation degree between an image corresponding to a detection target and an image corresponding to a track target is obtained, and a first association cost matrix is established based on the feature correlation degree; coordinate position information of a track trajectory prediction box of a current frame is obtained, and a second association cost matrix is established based on the coordinate position information of the track trajectory prediction box and coordinate position information of a detection box; the detection target and the state-determined track target are cascadedly matched based on the first association cost matrix and the second association cost matrix, to obtain a cascaded matching set, unmatched track targets and unmatched detection targets; IOU matching is performed based on the unmatched track targets and the unmatched detection targets, to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set. Through the above method, the association accuracy between the detection target and the track target is improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a multi-target tracking method, device and medium. Background Technology

[0002] Multi-target tracking is an important research topic in the field of autonomous driving. Target tracking is based on target detection and continuously estimates the motion state of the target to solve the problem of unstable sensor detection performance. It provides continuous target detection information for the decision-making, control, and execution modules of intelligent vehicles. Through accurate and effective tracking, autonomous vehicles can know the speed of vehicles within their field of vision and make corresponding motion plans.

[0003] Current tracking algorithms rely solely on image appearance information or LiDAR's 3D position information to independently track targets, resulting in low tracking accuracy. Furthermore, because vehicles may encounter situations where the target vehicle is obscured during travel, the target ID switches frequently, leading to low accuracy in associating detected and tracked targets. Summary of the Invention

[0004] This application provides a multi-target tracking method, device, and medium to solve the following technical problem: when a vehicle is driving, it may encounter a situation where the target vehicle is obscured, which leads to frequent switching of the tracking target ID, resulting in a low accuracy of the association between the detected target and the tracked target.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] This application provides a multi-target tracking method. The method includes: inputting the image corresponding to the detected target in the current frame into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the detected target and the image corresponding to the trajectory target; establishing a first correlation cost matrix based on the feature correlation degree; obtaining the coordinate position information of the predicted tracking trajectory box in the current frame based on the coordinate position information of the tracking trajectory box in the previous frame and a Kalman filter; establishing a second correlation cost matrix based on the coordinate position information of the predicted tracking trajectory box and the coordinate position information of the detected box; performing cascaded matching between the detected target and the trajectory target with a determined state based on the first and second correlation cost matrices to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets; performing IOU matching between the unmatched trajectory targets and the unmatched detected targets to obtain an IOU matching set, thereby achieving multi-target tracking based on the cascaded matching set and the IOU matching set.

[0007] This application embodiment improves the matching accuracy between detected and tracked targets by inputting multimodal information and utilizing the appearance features of images and the 3D positional information of point clouds. For long-term target occlusion, this application embodiment uses cosine distance based on appearance features to associate the target's motion state. When encountering vehicles with similar appearances, Mahalanobis distance based on 3D motion state information is used to associate the target's motion state. Combining these two measurement methods effectively avoids false detections and missed detections when one fails, and effectively reduces the problem of frequent target ID switching caused by occlusion. Furthermore, this application embodiment uses a two-level matching algorithm to maximize the matching efficiency between the detected target sequence and the tracked target sequence, reducing the occurrence of missed matches.

[0008] In one implementation of this application, based on a first association cost matrix and a second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with a determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets. Specifically, this includes: initializing multiple detected targets as an unmatched target set and initializing multiple trajectory targets as a trajectory set; and performing cascaded matching based on the first association cost matrix and the second association cost matrix corresponding to the unmatched target set, the trajectory set, and a preset maximum number of lost frames to obtain an initial matching set, unmatched trajectory targets, and unmatched detected targets.

[0009] In one implementation of this application, cascaded matching is performed based on the first and second association cost matrices of the unmatched target set, the trajectory set, and a preset maximum number of lost frames to obtain an initial matching set, unmatched trajectory targets, and unmatched detection targets. Specifically, this includes: determining a first sub-trajectory set with 0 lost frames in the trajectory set; using the first association cost matrix and the second association cost matrix of the first sub-trajectory set and the unmatched target set as input to a Hungarian algorithm to obtain a first matching set and a first unmatched target set; determining a second sub-trajectory set with K lost frames in the trajectory set; using the first association cost matrix and the second association cost matrix of the second sub-trajectory set and the first unmatched target set as input to a Hungarian algorithm to obtain a second matching set and a second unmatched target set; gradually increasing the value of K until it reaches a preset maximum number of lost frames, and determining the unmatched trajectory targets and unmatched detection targets corresponding to the preset maximum number of lost frames; and using the matching sets corresponding to different numbers of lost frames as the initial matching set.

[0010] In one implementation of this application, before performing cascade matching between the detected target and the trajectory target with a determined state based on the first association cost matrix and the second association cost matrix, the method further includes: obtaining the number of times the trajectory target is detected; when the number of times is greater than a preset number threshold, dividing the trajectory target into determined trajectory targets for cascade matching; and obtaining unmatched trajectory targets for IOU matching.

[0011] In one implementation of this application, IOU matching is performed based on unmatched trajectory targets and unmatched detection targets to obtain an IOU matching set. Specifically, this includes: establishing an IOU correlation matrix based on unmatched trajectory targets and unmatched detection targets; and using the IOU correlation matrix as input to the Hungarian algorithm to obtain the matching relationship between unmatched trajectory targets and unmatched detection targets respectively.

[0012] This embodiment calculates the cost correlation matrix between the detected target and the tracked target using Mahalanobis distance and cosine distance. This cost correlation matrix is ​​then used as input to the Hungarian algorithm, which outputs the optimal matching relationship between the detected target and the tracked target, thus completing the target matching problem between consecutive frames. This embodiment employs a two-level matching algorithm to maximize the matching efficiency between the detected target sequence and the tracked target sequence, minimizing missed matches.

[0013] In one implementation of this application, the coordinate position information of the predicted tracking trajectory box in the current frame is obtained based on the coordinate position information of the tracking trajectory box in the previous frame and the Kalman filter. Specifically, this includes: obtaining the predicted state vector corresponding to the tracking trajectory in the current frame through the prediction model in the Kalman filter and the state vector corresponding to the tracking trajectory in the previous frame; obtaining the measurement noise corresponding to the tracking trajectory through a preset sensor; and updating the predicted state vector through the measurement model in the Kalman filter and the measurement noise to obtain the final state vector corresponding to the tracking trajectory in the current frame.

[0014] In one implementation of this application, a second association cost matrix is ​​established based on the predicted bounding box coordinate position information and the detection bounding box coordinate position information. Specifically, this includes: obtaining the detection bounding box coordinate position information corresponding to the target in the current frame based on the 3D point cloud information; and establishing a second association cost matrix based on the Mahalanobis distance between the detection bounding box coordinate position information corresponding to the target in the current frame and the predicted bounding box coordinate position information corresponding to the tracking trajectory of the image in the current frame.

[0015] In one implementation of this application, before inputting the image corresponding to the target detected in the current frame into a preset depth feature descriptor, the method further includes: establishing a corresponding tracking target manager based on each trajectory target; recording trajectory information for different trajectory targets through the tracking target manager; wherein the trajectory information includes at least one of the following: motion state information of the trajectory target, feature information of the trajectory target, state information of the trajectory target, and state parameter information of the trajectory target.

[0016] This application provides a multi-target tracking device, including: at least one processor; and,

[0017] A memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to: input the image corresponding to the detected target in the current frame into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the detected target and the image corresponding to the trajectory target, and establish a first correlation cost matrix based on the feature correlation degree; obtain the coordinate position information of the predicted tracking trajectory box in the current frame based on the coordinate position information of the tracking trajectory box in the previous frame and the Kalman filter, and establish a second correlation cost matrix based on the coordinate position information of the predicted tracking trajectory box and the coordinate position information of the detected box; perform cascade matching on the detected target and the trajectory target with a determined state based on the first correlation cost matrix and the second correlation cost matrix to obtain a cascade matching set, unmatched trajectory targets, and unmatched detected targets; perform IOU matching on the unmatched trajectory targets and the unmatched detected targets to obtain an IOU matching set, so as to realize multi-target tracking based on the cascade matching set and the IOU matching set.

[0018] This application provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured to: input the image corresponding to the detected target in the current frame into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the detected target and the image corresponding to the trajectory target; and establish a first correlation cost matrix based on the feature correlation degree; obtain the coordinate position information of the tracking trajectory prediction box in the current frame based on the coordinate position information of the tracking trajectory prediction box and the detection box coordinate position information; perform cascade matching on the detected target and the trajectory target with a determined state based on the first and second correlation cost matrices to obtain a cascade matching set, unmatched trajectory targets, and unmatched detected targets; and perform IOU matching on the unmatched trajectory targets and unmatched detected targets to obtain an IOU matching set, thereby achieving multi-target tracking based on the cascade matching set and the IOU matching set.

[0019] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: This application embodiment improves the matching accuracy between detected and tracked targets by inputting multimodal information and utilizing the appearance features of the image and the three-dimensional position information of the point cloud. For the problem of long-term target occlusion, this application embodiment uses cosine distance based on appearance features to associate the target's motion state. When encountering vehicles with similar appearances, Mahalanobis distance based on three-dimensional motion state information is used to associate the target's motion state. Combining the two measurement methods effectively avoids false detections and missed detections when one fails, and effectively reduces the problem of frequent tracking target ID switching caused by occlusion. Furthermore, this application embodiment uses a two-level matching algorithm to maximize the matching efficiency between the detected target sequence and the tracked target sequence, reducing the occurrence of missed matches. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0021] Figure 1 A flowchart of a multi-target tracking method provided in an embodiment of this application;

[0022] Figure 2 A flowchart of a multi-target tracking algorithm provided in this application embodiment;

[0023] Figure 3 A trajectory state transition diagram provided in this application embodiment;

[0024] Figure 4 A schematic diagram of a constant velocity model provided in an embodiment of this application;

[0025] Figure 5 A flowchart of a Kalman filter algorithm provided in this application embodiment;

[0026] Figure 6 This application provides a schematic diagram of target tracking at different times as described in its embodiments.

[0027] Figure 7 A schematic diagram of point cloud tracking at different times provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of the calculated trajectory and heading angle of the tracking target provided in this application embodiment;

[0029] Figure 9 A tracking effect diagram in a daytime scene provided as an embodiment of this application;

[0030] Figure 10 An example of tracking effect in a night scene provided in this application embodiment;

[0031] Figure 11 A road test map provided for an embodiment of this application;

[0032] Figure 12 This application provides a tracking trajectory and heading angle diagram for road testing in a daytime scenario.

[0033] Figure 13 This application provides a tracking trajectory and heading angle diagram for road testing in a nighttime scene.

[0034] Figure 14 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application. Detailed Implementation

[0035] This application provides a multi-target tracking method, device, and medium.

[0036] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0037] Multi-target tracking is an important research topic in the field of autonomous driving. Target tracking is based on target detection and continuously estimates the motion state of the target to solve the problem of unstable sensor detection performance. It provides continuous target detection information for the decision-making, control, and execution modules of intelligent vehicles. Through accurate and effective tracking, autonomous vehicles can know the speed of vehicles within their field of vision and make corresponding motion plans.

[0038] Current tracking algorithms rely solely on image appearance information or LiDAR's 3D position information to independently track targets, resulting in low tracking accuracy. Furthermore, because vehicles may encounter situations where the target vehicle is obscured during travel, the target ID switches frequently, leading to low accuracy in associating detected and tracked targets.

[0039] This application embodiment improves the matching accuracy between detected and tracked targets by inputting multimodal information and utilizing the appearance features of images and the 3D positional information of point clouds. For long-term target occlusion, this application embodiment uses cosine distance based on appearance features to associate the target's motion state. When encountering vehicles with similar appearances, Mahalanobis distance based on 3D motion state information is used to associate the target's motion state. Combining these two measurement methods effectively avoids false detections and missed detections when one fails, and effectively reduces the problem of frequent target ID switching caused by occlusion. Furthermore, this application embodiment uses a two-level matching algorithm to maximize the matching efficiency between the detected target sequence and the tracked target sequence, reducing the occurrence of missed matches.

[0040] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0041] Figure 1 This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this application. Figure 1 As shown, the multi-target tracking method includes the following steps:

[0042] S101. The multi-target tracking device inputs the image corresponding to the detected target in the current frame into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the detected target and the image corresponding to the trajectory target, and establishes a first correlation cost matrix based on the feature correlation degree.

[0043] In one embodiment of this application, a corresponding tracking target manager is established for each trajectory target. The tracking target manager records trajectory information for different trajectory targets. The trajectory information includes at least one of the following: trajectory target motion state information, trajectory target feature information, trajectory target state information, and trajectory target state parameter information.

[0044] Specifically, each trajectory contains a wealth of information, so it is necessary to establish relevant data structures to organize and save different trajectories in order to match them with the detection target and output the tracking results.

[0045] Furthermore, the tracking target manager mainly has the following functions:

[0046] (1) Record the motion state of the trajectory, predict and update the motion state of the trajectory through the Kalman filter algorithm, calculate the similarity with the motion state of the target detection sequence, and form a motion information association matrix.

[0047] (2) Save the features of the trajectory. Each trajectory can save up to 100 appearance information features, which are used to calculate the similarity with the features of the detected target and form an appearance information association matrix. The existence of 100 appearance features helps to solve the problem of frequent ID switching caused by occlusion.

[0048] (3) Save the status information of the trajectory. The trajectory has three existence states: Confirmed, Tentative, and Deleted. Trajectories with different states will face different processing methods.

[0049] (4) Save the parameter information that determines the trajectory status: Hits count, which records the number of times the trajectory has been matched, i.e., the number of times it has been successfully tracked. If the count is greater than 3, the trajectory is considered to be a definite existing trajectory, which can avoid false detection to a certain extent. Age count, which records how many frames have been matched since the trajectory was initialized. Regardless of whether the trajectory is matched or not, this count will be incremented by one, i.e., the length of the trajectory's existence. Time_Since_Update count, which records how many frames the trajectory has not been matched. If it exceeds 30 frames, the trajectory will be marked as deleted. The existence of this count can solve the problem of the target being occluded within 30 frames, and it will not be marked as a new trajectory after reappearing due to occlusion.

[0050] Figure 3 This is a trajectory state transition diagram provided in an embodiment of this application. Figure 3 As shown, the matching module outputs unmatched detected targets, matched pairs, and unmatched trajectories. Unmatched detected targets are initialized as new trajectories, and their status information is defined as tentative. The unmatched detected target is then added to the trajectory library. The number of hits corresponding to matched pairs is counted. If the number of hits is not less than 3, the trajectory is defined as confirmed and added to the trajectory library. The status of unmatched pairs is confirmed. If confirmed, they are added to the trajectory library. If not, the number of frames for which the trajectory has not been matched is checked. If it is greater than 30, the unmatched pair is deleted; otherwise, it is added to the trajectory library.

[0051] In one embodiment of this application, a small deep learning network, called a deep feature descriptor, is pre-trained to specifically learn the features of the vehicle. The network structure of the deep feature descriptor is shown in Table 1.

[0052] Name Patch Size / Stride Output Size Conv 1 3×3 / 1 32×128×64 Conv 2 3×3 / 1 32×128×64 MaxPool 3 3×3 / 2 32×64×32 Residual 4 3×3 / 1 32×64×32 Residual 5 3×3 / 1 32×64×32 Residual 6 3×3 / 2 64×32×16 Residual 7 3×3 / 1 64×32×16 Residual 8 3×3 / 2 128×16×8 Residual 9 3×3 / 1 128×16×8 Dense 10 128 Batch and L2 normalization 128

[0053] Table 1

[0054] As shown in Table 1, the final output of the depth feature descriptor is a 128-dimensional feature of the target. The corresponding image portion of the target in each detection sequence is input into the depth feature descriptor to obtain the 128-dimensional feature. The cosine distance is used to compare it with the features saved in the trajectory target sequence to evaluate the degree of correlation between the features.

[0055] In one embodiment of this application, the similarity between two feature vectors is evaluated using cosine distance. A first association cost matrix is ​​established based on the distance metric; that is, an association matrix is ​​established between the appearance feature vectors of the detected target sequence and the tracked target sequence.

[0056] Specifically, assume the target sequence to be detected is D = {D0, D1, D2, ..., D...} i The target sequence to be tracked is T = {T0, T1, T2, ..., T}. j The cosine distance correlation matrix is ​​established.

[0057]

[0058] For each tracked target, a maximum of 100 feature vectors are stored. Therefore, for each detected target, the maximum of 100 features stored for a certain trajectory are calculated, and the value with the smallest cosine distance is taken as the value in the correlation matrix.

[0059] S102. The multi-target tracking device obtains the coordinate position information of the predicted tracking trajectory box in the current frame based on the coordinate position information of the tracking trajectory box in the previous frame and the Kalman filter, and establishes a second correlation cost matrix based on the coordinate position information of the predicted tracking trajectory box and the coordinate position information of the detection box.

[0060] In one embodiment of this application, the predicted state vector corresponding to the tracking trajectory in the current frame is obtained by using the prediction model in the Kalman filter and the state vector corresponding to the tracking trajectory in the previous frame. Measurement noise corresponding to the tracking trajectory is acquired through a preset sensor. The predicted state vector is updated using the measurement model in the Kalman filter and the measurement noise to obtain the final state vector corresponding to the tracking trajectory in the current frame.

[0061] Specifically, this application embodiment uses the Kalman filter algorithm to estimate the motion state. This algorithm solves the problem of linear filtering of discrete data using a recursive method. The Kalman filter is described by a series of recursive formulas and is an efficient and computable state estimation method, using the minimum mean square error as the criterion for optimal estimation.

[0062] Specifically, Kalman filtering mainly uses two models: a prediction model and a measurement model. The prediction model is generally an empirical inductive formula or a physical formula, and its function is to predict the state at time k+1 using the state at time k. The measurement model describes the computational relationship when parameters directly measurable by sensors are transformed into some or all parameters in the state vector, resulting in some or local parameters in the final state vector obtained through the measurement model. The state vector obtained from the prediction model is updated by the measurement model to obtain the optimal state estimate. The standard Kalman filter algorithm can only be used when both the prediction and measurement models are linear. The standard Kalman filter consists of two steps: prediction and update.

[0063] The formula for calculating the prediction step is as follows:

[0064] x k =Fx k +Bu k

[0065] P k =FP k F T +Q

[0066] Where, x k The vector to be estimated is an n-dimensional vector. F is the prediction matrix, an n×n matrix. B is the control matrix, u k P represents the control vector. k Let be the covariance matrix, and Q be the prediction noise. In this step, the optimal state estimate from the previous time step is known, and the state value for that time step is predicted using the prediction model. Furthermore, the covariance is updated through prediction; the covariance describes the magnitude of the error in the prediction process and reflects the reliability of the predicted state values.

[0067] The formula for calculating the update steps is as follows:

[0068] G k =P k H T HP k H T +R) -1

[0069] x k =x k +G k (z k -Hx k )

[0070] P k =(1-G k H)P k

[0071] Specifically, the first step is to calculate the Kalman filter gain G. k The calculation principle is to compare whether the prediction error or the sensor measurement error is larger, that is, to compare the magnitude of the covariance. For prediction, the covariance is P. k For measurement, the sensor measurement covariance is R. H is the measurement matrix. The second step is to update the predicted value using the measured value to obtain the optimal state estimate, which is the linear weighted result of the predicted value and the sensor measurement value. The third step is to adjust the covariance matrix P. k The system is updated by calculating the error in the prediction process during this state estimation. This error is used to calculate the Kalman gain for the next state estimation. If the error is large in this round, the weight of the predicted value will be reduced accordingly in the next round, making the measured value more influential.

[0072] In one embodiment of this application, the coordinate position information of the detection box corresponding to the target in the current frame is obtained based on the three-dimensional point cloud information. A second association cost matrix is ​​established based on the Mahalanobis distance between the coordinate position information of the detection box corresponding to the target in the current frame and the coordinate position information of the predicted box corresponding to the tracking trajectory of the current frame image.

[0073] Figure 4 This is a schematic diagram of a constant velocity model provided in an embodiment of this application. Figure 4 As shown, the formulas and parameter meanings of Kalman filtering are further explained based on the constant velocity model and parameter settings.

[0074] Prediction steps:

[0075] In this embodiment, the state vector is taken as x. k =(p x p y v x v y Using a constant speed (CV) model for the vehicle, as shown in the figure above, assuming the vehicle travels at a constant speed v on the road, the calculation formula can be obtained:

[0076] s t =s t-1 +vΔt

[0077] v t =v t-1

[0078] Decomposing the formula along the X and Y directions yields the following equation:

[0079]

[0080] The matrix form is as follows:

[0081]

[0082] Corresponding formula x k+1 =Fx k Therefore, in this model, the prediction matrix F is as follows:

[0083]

[0084] However, in actual vehicle movement, the speed is not always constant and there will be certain acceleration and deceleration. In this model, the corresponding throttle tension is used as a control variable, and its expression is as follows, from which the control matrix B and control variable u can be obtained.

[0085]

[0086] The second step in forecasting is updating the covariance. P k Q is the covariance matrix of the probability distribution of the state vector. In reality, for example, vehicles may be affected by wind, resulting in fluctuations, i.e., the introduction of noise. Q is the covariance matrix of the prediction noise, which is introduced to compensate for the prediction process. In this model, Q is a 4×4 matrix, which is empirically initialized as follows:

[0087]

[0088] Update steps:

[0089] This step updates the predicted values ​​above using sensor measurements. In this model, the predicted values ​​can be obtained through measurement (p) x p y The formula between the state vector and the observations gives the measurement matrix H as:

[0090]

[0091]

[0092] R represents the measurement noise of the sensor. Here, the sensor is a lidar, and R is a 2×2 matrix. Based on experience, it is initialized as follows:

[0093]

[0094] As can be seen from the prediction model and the measurement model, both are linear models, thus meeting the requirements of standard Kalman filtering.

[0095] Figure 5 This is a flowchart of a Kalman filter algorithm provided for an embodiment of this application. Figure 5As shown, Kalman filtering mainly uses two models: a prediction model and a measurement model. First, Q, P, and R are initialized. Then, the predicted values ​​are calculated based on the prediction model, and the covariance is also calculated. Next, the Kalman gain is calculated, and the predicted values ​​and covariance are updated based on the measured values. This method enables Kalman filtering calculation.

[0096] In one embodiment of this application, Mahalanobis distance represents the covariance distance between state vectors, and is an effective method for calculating the similarity between two vectors. The principle of Mahalanobis distance is to rotate the state vectors along the direction of the feature vectors, making each variable dimension independent, and then standardize them to ensure that each variable dimension has a distribution of the same scale. The formula for calculating the Mahalanobis distance of a single data point is as follows:

[0097]

[0098] The Mahalanobis distance between data points x and y is calculated as follows, where ∑ is the covariance matrix of the multidimensional random variable and μ is the mean of the variable.

[0099]

[0100] In this paper, Mahalanobis distance is used to calculate the degree of motion correlation between 3D detection boxes and 3D tracking prediction boxes. The calculation formula is as follows:

[0101]

[0102] Where d j and y i S represents the 3D position of the j-th detection box and the 3D position of the ith tracking prediction box, respectively. i The covariance matrix between the two can be used to obtain the correlation matrix of the Mahalanobis distance metric.

[0103]

[0104] In this embodiment, the inverse χ 2 The distribution's 95% confidence interval is thresholded using the Mahalanobis distance. In this embodiment, the measurement state is set as (x, y, γ, h), having four dimensions. The corresponding χ² value is then looked up. 2 The threshold for the distribution table is t = 9.4877. If the Mahalanobis distance is greater than this threshold, the detected target motion state and the tracked target motion state are considered unrelated, and the cosine distance between them is set to 100000, which is considered to be an infinite state.

[0105] For long-term occlusion problems, the cosine distance of appearance features is very effective in this application embodiment. When encountering vehicles with similar appearances, the Mahalanobis distance based on three-dimensional motion state information has a great advantage. Combining the two measurement methods can effectively avoid the occurrence of false detection and missed detection when one of the conditions fails, and greatly improve the accuracy of matching.

[0106] S103. The multi-target tracking device performs cascade matching between the detected target and the trajectory target with a determined state based on the first association cost matrix and the second association cost matrix, to obtain a cascade matching set, unmatched trajectory targets and unmatched detected targets.

[0107] In one embodiment of this application, the number of times a trajectory target is detected is obtained. When the number of times is greater than a preset threshold, the trajectory target is divided into determined trajectory targets for cascade matching.

[0108] Specifically, the most important aspect of tracking tasks is matching consecutive frames of the same target, i.e., associating the target with the detection sequence and the tracking sequence. Target trajectories with fewer than 30 lost frames are retained. However, if the target trajectory is lost for an extended period, the uncertainty in trajectory prediction increases, i.e., the covariance increases. Since Mahalanobis distance uses the reciprocal of the covariance, this can actually decrease the Mahalanobis distance, causing the detected target to be matched with trajectories that have been lost for a longer period, resulting in mismatches. To address this issue, a cascaded matching algorithm is introduced. This algorithm prioritizes trajectories that have been matched more recently in time for subsequent matching. Typically, consecutively matched trajectories participate in the matching task first, then trajectories that have lost one frame participate, and so on, increasing the number of lost frames until trajectories that have lost 30 frames participate in the matching, at which point the cascaded matching ends.

[0109] In one embodiment of this application, multiple detection targets are initialized as an unmatched target set, and multiple trajectory targets are initialized as a trajectory set. Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and a preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detection targets.

[0110] Specifically, a first sub-track set with zero lost frames is determined from the track set. The first association cost matrix and the second association cost matrix of the first sub-track set and the unmatched target set are used as inputs to the Hungarian algorithm to obtain a first matching set and a first unmatched target set. A second sub-track set with K lost frames is determined from the track set. The first association cost matrix and the second association cost matrix of the second sub-track set and the first unmatched target set are used as inputs to the Hungarian algorithm to obtain a second matching set and a second unmatched target set. The value of K is gradually increased until it reaches a preset maximum number of lost frames. The unmatched track targets and unmatched detected targets corresponding to the preset maximum number of lost frames are then determined. The matching sets corresponding to different numbers of lost frames are used as the initial matching sets.

[0111] Specifically, the steps of cascading matching are as follows:

[0112] (1) The input to the cascaded matching algorithm is the tracking target sequence T = {1, ..., N} and the detection target sequence D = {1, ..., M}. The maximum number of frames that the algorithm can lose is A. max This article is 30.

[0113] (2) Calculate the first association cost matrix based on cosine distance between the two sequences.

[0114] (3) Calculate the second association cost matrix between the two sequences based on Mahalanobis distance.

[0115] (4) Initialize the matching set M to be empty. Initialize the target detection sequence D to be the unmatched target set u.

[0116] (5) Traverse the number of lost frames from 0 to A max A sub-track set with zero lost frames is selected from the track set, and its association cost matrix with the unmatched target is input into the Hungarian algorithm.

[0117] (6) Obtain the matching set and the unmatched targets. Select the trajectory set with the number of lost frames plus one and the unmatched targets in this round and perform the above operation until the number of lost frames reaches 30, then end the cascaded matching.

[0118] (7) Merge the matching sets obtained in each round to obtain the final matching set M and the detection targets that were not matched in the last round.

[0119] Matching the detected target and the trajectory target can also be categorized as a bipartite graph matching problem, and the Hungarian algorithm is chosen for matching. In this embodiment, the cost correlation matrix of the detection sequence and the tracking sequence is calculated using Mahalanobis distance and cosine distance. The cost correlation matrix serves as the input to the Hungarian algorithm, and the algorithm outputs the optimal matching relationship between the target sequence and the tracking sequence, thus completing the target matching problem between consecutive frames, i.e., RE-ID.

[0120] S104. The multi-target tracking device performs IOU matching based on unmatched trajectory targets and unmatched detection targets to obtain an IOU matching set, and then realizes multi-target tracking based on the cascaded matching set and the IOU matching set.

[0121] In one embodiment of this application, when the number of attempts is not greater than a preset threshold, the trajectory target is divided into trajectory targets to be determined, so as to perform IOU matching on the trajectory targets to be determined.

[0122] In one embodiment of this application, the number of times a trajectory target is detected is obtained. When the number of detections exceeds a preset threshold, the trajectory target is divided into defined trajectory targets for cascade matching. Unmatched trajectory targets are then obtained for IOU matching.

[0123] Specifically, when the trajectory is in a determined state, it serves as the input to the cascade matching algorithm. This is because when the trajectory state is uncertain, it means that the trajectory has been detected no more than 3 times at the current moment, making cascade matching meaningless.

[0124] In one embodiment of this application, an IOU (Intersection over Union) correlation matrix is ​​established based on unmatched trajectory targets and unmatched detected targets. The IOU correlation matrix is ​​then used as input to the Hungarian algorithm to obtain the matching relationship between unmatched trajectory targets and unmatched detected targets.

[0125] Specifically, IOU matching applies to detection and tracking sequences. The metric for determining whether two targets match is whether their IOU values ​​fall within a threshold. If the IOU value is less than the threshold, the two sequences are considered to have a certain correlation. An IOU correlation matrix between the tracking and detection sequences can be easily constructed.

[0126]

[0127] The IOU (Intersection over Union) matrix is ​​used as input to the Hungarian algorithm, and the output is the matching relationship between the tracking sequence and the detection sequence. IOU is an important metric that has emerged in the image processing field for determining whether two targets have a matching relationship. Its correct use and mastery are crucial for image training, prediction, and the matching of multiple targets. The tracking target sequence and detection sequence with determined states are input into the cascaded matching algorithm to obtain matched pairs, unmatched tracking targets, and unmatched detection targets. Then, the tracking target sequence with undetermined states and the unmatched tracking targets are combined to form a new set of unmatched tracking targets, which, along with the unmatched detection targets, are input into the IOU matching algorithm.

[0128] This application embodiment uses a two-level matching algorithm to maximize the matching efficiency between the detected target sequence and the tracked target sequence, thereby reducing the occurrence of missed matches.

[0129] This application embodiment improves the matching accuracy between detected and tracked targets by inputting multimodal information and utilizing the appearance features of images and the three-dimensional position information of point clouds. For long-term occlusion issues, cosine distance based on appearance features is used to associate the target's motion state. When encountering vehicles with similar appearances, Mahalanobis distance based on three-dimensional motion state information is used to associate the target's motion state. Combining these two measurement methods effectively avoids false detections and missed detections when one fails, and effectively reduces the problem of frequent target ID switching caused by occlusion.

[0130] Figure 2 This is a flowchart illustrating a multi-target tracking algorithm provided in an embodiment of this application. Figure 2 As shown, firstly, a separate tracking manager is established for each target to manage the tracking target information. The appearance and motion state information of the target are used to establish a correlation matrix between the detected and tracked targets. Then, cascaded matching and IOU matching algorithms are used to correlate the detection and tracking sequences across consecutive frames. Finally, a Kalman filter based on a constant velocity model is used to predict and update the target motion state.

[0131] Figure 6 This is a schematic diagram illustrating target tracking at different times, provided as an embodiment of this application. Figure 5As shown, a scene from the KITTI tracking dataset was randomly selected to demonstrate the tracking effect. Two vehicles were tracked in this scene, and the tracking results at three time points were captured, from top to bottom: time point 1, time point 2, and time point 3. At time point 1, target 224 had already been tracked for some time, while target 230 had just begun to be tracked. At time point 2, tracking continued for 1.3 seconds based on the first time point, and at time point 3, tracking continued for 0.8 seconds based on the second time point.

[0132] Figure 7 This is a schematic diagram of point cloud tracking at different times provided in an embodiment of this application. Figure 7 As shown, from left to right, the point cloud tracking display effect and tracking trajectory are shown for the first, second and third time points.

[0133] pass Figure 6 and Figure 7 As can be seen, the tracking algorithm in this embodiment can continuously track multiple targets without switching the tracking ID. However, trajectory 230 exhibits a two-frame missed detection issue. After the target is re-detected, it is still marked with the original tracking target ID, and no ID switching occurs. This scenario demonstrates that the tracking algorithm proposed in this patent can simultaneously track multiple targets, and even in the event of a brief missed detection, it can still be tracked using the original target ID, effectively reducing the problem of frequent ID switching.

[0134] Figure 8 This is a schematic diagram illustrating a calculated tracking target trajectory and heading angle provided in an embodiment of this application. Figure 8 As shown, Figure 8 (a) shows a comparison between the trajectory tracked by the KF algorithm and the actual trajectory GroundTruth. It can be seen intuitively from the figure that the trajectory tracked by KF and GroundTruth largely overlap, and the calculated position deviation is 1.059m. Figure 8 (b) and (c) show the target's position changes in the X and Y directions, with calculated root mean square errors of 1.016m and 0.303m, respectively. This indicates that the main error is in the X direction, i.e., the vehicle's driving direction. The tracking algorithm performs poorly in locating the target in the driving direction, primarily because the point cloud detection algorithm obtains a large error in the target position during clustering, leading to significant positional errors in the tracking. Figure 8 (d) shows the comparison between the tracked heading angle and the actual heading angle during the target tracking process. The mean square error of the heading angle deviation is calculated to be 0.033°, indicating that the tracking algorithm is relatively accurate in tracking the heading angle.

[0135] Figure 9 This image illustrates the tracking effect in a daytime scene, as provided in an embodiment of this application. Figure 9 As shown, Figure 9 The image shows a real vehicle test platform crossing an overpass, a scene involving changes in lighting. Figure 9 The image contains four pictures, from left to right and top to bottom: the scene upon entering the overpass, the scene under the overpass, the scene just before leaving the overpass, and the scene after leaving the overpass. It can be seen that throughout the process, the algorithm tracked target vehicle #4 in the lane ahead and target vehicle #18 in the adjacent lane to the left. During the crossing of the overpass, target #18 was lost during the final exit due to overexposure and occlusion, but it was quickly detected again without changing its target ID. Target #4 was consistently and stably tracked without any change in its target ID. This clearly demonstrates that the tracking algorithm can simultaneously track two vehicles and can mitigate the problem of frequent target ID switching caused by occlusion and other issues.

[0136] Figure 10 This image illustrates a tracking effect in a nighttime scene, as provided in an embodiment of this application. Figure 10 As shown, Figure 10 For tracking effects in nighttime scenes, Figure 10 During the tracking process, the number of vehicles being tracked changed from tracking one vehicle to tracking two vehicles simultaneously. It can be clearly seen that the tracking algorithm of this application embodiment can also adapt well to the nighttime environment and track multiple vehicles at the same time. Furthermore, the ID of the tracked target is relatively stable and does not switch frequently.

[0137] Through real-vehicle testing, it can be concluded that the algorithm can meet the requirements of the real-vehicle testing platform for multi-target detection and tracking tasks. The algorithm can complete the detection, fusion, and tracking of multiple targets in both daytime and nighttime environments. Furthermore, the fusion process can correct the detection results, effectively reducing the occurrence of missed detections and false detections. By inputting multimodal information, the algorithm can utilize the appearance features of the image and the 3D positional information of the point cloud to improve the matching accuracy between the detected and tracked targets, effectively reducing the problem of frequent target ID switching caused by occlusion.

[0138] Figure 11 This application provides a road test map as an embodiment. This embodiment uses a Kalman filter algorithm to predict and update the target's motion state, and selects a daytime scene and a nighttime scene to demonstrate the tracking effect on the target's position and heading angle. The selected daytime scene is as follows... Figure 11 As shown in (a), the night scene is as follows Figure 11 As shown in (b).

[0139] Figure 12 This application provides a tracking trajectory and heading angle diagram for road testing in a daytime scenario. Figure 12 (a) and (b) demonstrate that the tracking algorithm in this application embodiment can achieve relatively continuous and stable tracking of the vehicle target ahead, so as to obtain the target's trajectory information and heading information. The heading angle can be used to determine the target's driving direction and behavior. If the target's angle changes drastically in a short period of time, it indicates that the target may be changing lanes or turning.

[0140] Figure 13 This application provides a tracking trajectory and heading angle diagram for road testing in a nighttime scene. The target tracking results in the scene are as follows: Figure 13 As shown in (a), heading angle tracking is as follows Figure 13 As shown in (b), the tracking algorithm can also track the target position and heading angle in nighttime scenes.

[0141] Figure 14 This is a schematic diagram of the structure of a multi-target tracking device provided in an embodiment of this application. Figure 14 As shown, the multi-target tracking device includes at least one processor; and,

[0142] A memory communicatively connected to the at least one processor; wherein,

[0143] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0144] The image corresponding to the target detected in the current frame is input into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the target detected and the image corresponding to the trajectory target, and a first correlation cost matrix is ​​established based on the feature correlation degree.

[0145] Based on the tracking trajectory box coordinate position information of the previous frame and the Kalman filter, obtain the tracking trajectory prediction box coordinate position information of the current frame, and establish a second association cost matrix based on the tracking trajectory prediction box coordinate position information and the detection box coordinate position information;

[0146] Based on the first association cost matrix and the second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with the determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets;

[0147] Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set.

[0148] This application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0149] The image corresponding to the target detected in the current frame is input into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the target detected and the image corresponding to the trajectory target, and a first correlation cost matrix is ​​established based on the feature correlation degree.

[0150] Based on the tracking trajectory box coordinate position information of the previous frame and the Kalman filter, the tracking trajectory prediction box coordinate position information of the current frame is obtained, and a second association cost matrix is ​​established based on the tracking trajectory prediction box coordinate position and detection box coordinate position information.

[0151] Based on the first association cost matrix and the second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with the determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets;

[0152] Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set.

[0153] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0154] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] The above description is merely an embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of this application should be included within the scope of the claims of this application.

Claims

1. A multi-target tracking method, characterized in that, The method includes: The image corresponding to the target detected in the current frame is input into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the target detected and the image corresponding to the trajectory target, and a first correlation cost matrix is ​​established based on the feature correlation degree. Based on the tracking trajectory box coordinate position information of the previous frame and the Kalman filter, obtain the tracking trajectory prediction box coordinate position information of the current frame, and establish a second association cost matrix based on the tracking trajectory prediction box and the detection box coordinate position information; Based on the first association cost matrix and the second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with the determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets; Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set; The step of performing cascaded matching between the detected target and the trajectory target with a determined state, based on the first association cost matrix and the second association cost matrix, to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets, specifically includes: The plurality of detected targets are initialized as a set of unmatched targets, and the plurality of trajectory targets are initialized as a set of trajectories; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detection targets. Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, which specifically includes: Based on the unmatched trajectory targets and the unmatched detected targets, an IOU correlation matrix is ​​established; The IOU correlation matrix is ​​used as input to the Hungarian algorithm to obtain the matching relationship between the unmatched trajectory target and the unmatched detected target; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detected targets, specifically including: A first sub-track set with 0 lost frames is determined from the track set. The first sub-track set, the first association cost matrix of the first sub-track set, and the second association cost matrix of the unmatched target set are used as inputs to the Hungarian algorithm to obtain a first matching set and a first unmatched target set. A second sub-track set with K lost frames is determined from the track set. The second sub-track set, the first association cost matrix of the second unmatched target set, and the second association cost matrix of the first unmatched target set are used as inputs to the Hungarian algorithm to obtain a second matching set and a second unmatched target set. Gradually increase the value of K until it reaches the preset maximum number of lost frames, and determine the unmatched trajectory targets and unmatched detection targets corresponding to the preset maximum number of lost frames; The matching sets corresponding to different numbers of lost frames are used as the initial matching sets; Before performing cascade matching between the detected target and the trajectory target with a determined state based on the first association cost matrix and the second association cost matrix, the method further includes: The number of times the trajectory target is detected is obtained. When the number of times is greater than a preset threshold, the trajectory target is divided into determined trajectory targets, and cascade matching is performed on the determined trajectory targets. Obtain unmatched trajectory targets, and perform IOU matching on the unmatched trajectory targets; The establishment of the second association cost matrix based on the coordinates of the predicted tracking trajectory bounding box and the location information of the detection box specifically includes: Based on the 3D point cloud information, obtain the coordinate position information of the detection box corresponding to the target in the current frame; Based on the Mahalanobis distance between the detection box coordinate position information corresponding to the target detected in the current frame and the prediction box coordinate position information corresponding to the tracking trajectory of the current frame image, the second association cost matrix is ​​established. Before inputting the image corresponding to the target detected in the current frame into the preset depth feature descriptor, the method further includes: Establish a corresponding tracking target manager for each trajectory target; The tracking target manager records trajectory information for different trajectory targets; wherein the trajectory information includes at least one of the following: the trajectory target's motion state information, the trajectory target's feature information, the trajectory target's state information, and the trajectory target's state parameter information.

2. The multi-target tracking method according to claim 1, characterized in that, The step of obtaining the coordinate position information of the predicted tracking trajectory box in the current frame based on the coordinate position information of the tracking trajectory box in the previous frame and the Kalman filter specifically includes: The predicted state vector corresponding to the tracking trajectory in the current frame is obtained by using the prediction model in the Kalman filter and the state vector corresponding to the tracking trajectory in the previous frame. The measurement noise corresponding to the tracking trajectory is obtained through a preset sensor; The predicted state vector is updated by using the measurement model in the Kalman filter and the measurement noise to obtain the final state vector corresponding to the tracking trajectory of the current frame.

3. A multi-target tracking device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The image corresponding to the target detected in the current frame is input into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the target detected and the image corresponding to the trajectory target, and a first correlation cost matrix is ​​established based on the feature correlation degree. Based on the tracking trajectory box coordinate position information of the previous frame and the Kalman filter, obtain the tracking trajectory prediction box coordinate position information of the current frame, and establish a second association cost matrix based on the tracking trajectory prediction box coordinate position information and the detection box coordinate position information; Based on the first association cost matrix and the second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with the determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets; Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set; The step of performing cascaded matching between the detected target and the trajectory target with a determined state, based on the first association cost matrix and the second association cost matrix, to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets, specifically includes: The plurality of detected targets are initialized as a set of unmatched targets, and the plurality of trajectory targets are initialized as a set of trajectories; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detection targets. Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, which specifically includes: Based on the unmatched trajectory targets and the unmatched detected targets, an IOU correlation matrix is ​​established; The IOU correlation matrix is ​​used as input to the Hungarian algorithm to obtain the matching relationship between the unmatched trajectory target and the unmatched detected target; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detected targets, specifically including: A first sub-track set with 0 lost frames is determined from the track set. The first sub-track set, the first association cost matrix of the first sub-track set, and the second association cost matrix of the unmatched target set are used as inputs to the Hungarian algorithm to obtain a first matching set and a first unmatched target set. A second sub-track set with K lost frames is determined from the track set. The second sub-track set, the first association cost matrix of the second unmatched target set, and the second association cost matrix of the first unmatched target set are used as inputs to the Hungarian algorithm to obtain a second matching set and a second unmatched target set. Gradually increase the value of K until it reaches the preset maximum number of lost frames, and determine the unmatched trajectory targets and unmatched detection targets corresponding to the preset maximum number of lost frames; The matching sets corresponding to different numbers of lost frames are used as the initial matching sets; Before performing cascade matching between the detected target and the trajectory target with a determined state based on the first association cost matrix and the second association cost matrix, the method further includes: The number of times the trajectory target is detected is obtained. When the number of times is greater than a preset threshold, the trajectory target is divided into determined trajectory targets, and cascade matching is performed on the determined trajectory targets. Obtain unmatched trajectory targets, and perform IOU matching on the unmatched trajectory targets; The establishment of the second association cost matrix based on the coordinates of the predicted tracking trajectory bounding box and the location information of the detection box specifically includes: Based on the 3D point cloud information, obtain the coordinate position information of the detection box corresponding to the target in the current frame; Based on the Mahalanobis distance between the detection box coordinate position information corresponding to the target detected in the current frame and the prediction box coordinate position information corresponding to the tracking trajectory of the current frame image, the second association cost matrix is ​​established. Before inputting the image corresponding to the target detected in the current frame into the preset depth feature descriptor, the method further includes: Establish a corresponding tracking target manager for each trajectory target; The tracking target manager records trajectory information for different trajectory targets; wherein the trajectory information includes at least one of the following: the trajectory target's motion state information, the trajectory target's feature information, the trajectory target's state information, and the trajectory target's state parameter information.

4. A non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows: The image corresponding to the target detected in the current frame is input into a preset depth feature descriptor to obtain the feature correlation degree between the image corresponding to the target detected and the image corresponding to the trajectory target, and a first correlation cost matrix is ​​established based on the feature correlation degree. Based on the tracking trajectory box coordinate position information of the previous frame and the Kalman filter, obtain the tracking trajectory prediction box coordinate position information of the current frame, and establish a second association cost matrix based on the tracking trajectory prediction box coordinate position information and the detection box coordinate position information; Based on the first association cost matrix and the second association cost matrix, a cascaded matching is performed between the detected target and the trajectory target with the determined state to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets; Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, so as to realize multi-target tracking according to the cascaded matching set and the IOU matching set; The step of performing cascaded matching between the detected target and the trajectory target with a determined state, based on the first association cost matrix and the second association cost matrix, to obtain a cascaded matching set, unmatched trajectory targets, and unmatched detected targets, specifically includes: The plurality of detected targets are initialized as a set of unmatched targets, and the plurality of trajectory targets are initialized as a set of trajectories; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detection targets. Based on unmatched trajectory targets and unmatched detected targets, IOU matching is performed to obtain an IOU matching set, which specifically includes: Based on the unmatched trajectory targets and the unmatched detected targets, an IOU correlation matrix is ​​established; The IOU correlation matrix is ​​used as input to the Hungarian algorithm to obtain the matching relationship between the unmatched trajectory target and the unmatched detected target; Based on the first and second association cost matrices of the unmatched target set, the trajectory set, and the preset maximum number of lost frames, cascaded matching is performed to obtain an initial matching set, unmatched trajectory targets, and unmatched detected targets, specifically including: A first sub-track set with 0 lost frames is determined from the track set. The first sub-track set, the first association cost matrix of the first sub-track set, and the second association cost matrix of the unmatched target set are used as inputs to the Hungarian algorithm to obtain a first matching set and a first unmatched target set. A second sub-track set with K lost frames is determined from the track set. The second sub-track set, the first association cost matrix of the second unmatched target set, and the second association cost matrix of the first unmatched target set are used as inputs to the Hungarian algorithm to obtain a second matching set and a second unmatched target set. Gradually increase the value of K until it reaches the preset maximum number of lost frames, and determine the unmatched trajectory targets and unmatched detection targets corresponding to the preset maximum number of lost frames; The matching sets corresponding to different numbers of lost frames are used as the initial matching sets; Before performing cascade matching between the detected target and the trajectory target with a determined state based on the first association cost matrix and the second association cost matrix, the method further includes: The number of times the trajectory target is detected is obtained. When the number of times is greater than a preset threshold, the trajectory target is divided into determined trajectory targets, and cascade matching is performed on the determined trajectory targets. Obtain unmatched trajectory targets, and perform IOU matching on the unmatched trajectory targets; The establishment of the second association cost matrix based on the coordinates of the predicted tracking trajectory bounding box and the location information of the detection box specifically includes: Based on the 3D point cloud information, obtain the coordinate position information of the detection box corresponding to the target in the current frame; Based on the Mahalanobis distance between the detection box coordinate position information corresponding to the target detected in the current frame and the prediction box coordinate position information corresponding to the tracking trajectory of the current frame image, the second association cost matrix is ​​established. Before inputting the image corresponding to the target detected in the current frame into the preset depth feature descriptor, the method further includes: Establish a corresponding tracking target manager for each trajectory target; The tracking target manager records trajectory information for different trajectory targets; wherein the trajectory information includes at least one of the following: the trajectory target's motion state information, the trajectory target's feature information, the trajectory target's state information, and the trajectory target's state parameter information.

Images