Multi-target tracking method and system based on cross Transform data association algorithm

Abstract

The invention provides a multi-target tracking method and system based on a cross Transform data association algorithm, and belongs to the field of artificial intelligence. The problems that an existing traditional MTT algorithm based on radar data is high in storage requirement and low in calculation speed, a learning-based algorithm needs to restrain the number of targets, network design considers typical features of position data, two cross tracks cannot be distinguished, and missing detection or false detection may exist are solved. According to the method, the optimal estimation in the aspects of precision and efficiency of radar-based position measurement is obtained by using a cross Transform-based data association network and adopting a BLUE filter; furthermore, a score-based trajectory management strategy is proposed to be based on the associated historical measurements; simulation experiments on various data sets show that the correlation and tracking method provided by the invention is superior to the existing method in complex multi-target tracking scenes including track intersection, a large number of false alarms and missing detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-target tracking method and system based on a cross Transformer data association algorithm. Background Art

[0002] Multi-target tracking (MTT) or multiple object tracking (MOT) plays a vital role in various fields such as surveillance, autonomous driving, and robotics. The MTT problem refers to associating relative measurements obtained by sensors over time to multiple moving objects (targets) and estimating their trajectories and other properties (velocity, acceleration, etc.). A key issue in MTT is to resolve the unknown correspondences between targets and measurements, a task known as data association. The MTT task is highly challenging in scenarios with occlusions, sensor noise, clutter, and targets that may enter or leave the scene.

[0003] In order to address these challenges, many algorithms and methods for multi-target tracking have been proposed, which can generally be divided into maximum likelihood data association methods and Bayesian data association methods. The maximum likelihood data association method is based on the likelihood ratio of the observed data for association. Classical algorithms include joint maximum likelihood algorithm, integer programming, generalized association method, etc. The maximum likelihood data association method is not suitable for situations with intersecting targets, maneuvering targets and dense multi-target scenes, which limits its application in practice. Bayesian data association methods are based on the Bayesian criterion and mainly include the nearest neighbor (NN) algorithm, probabilistic data association (PDA) algorithm, joint probabilistic data association (JPDA) algorithm, and multiple hypothesis tracking (MHT) method. These methods, such as JPDA, need to assume that the number of targets is constant. The computational complexity of the MHT method increases rapidly with the increase in the number of targets. Another type of MTT method is based on random finite sets (RFS), which uses multi-object conjugate priors to derive closed expressions for multiple object posteriors and obtain Bayesian optimal estimates. Typical algorithms include probability hypothesis density (PHD) filters, potential probability hypothesis density (CPHD) filters, and generalized label multi-Bernoulli (GLMB) trackers. RFS-based methods are able to transfer more information between different time steps, thereby improving tracking results. However, these methods usually rely on prior information about the true posterior density. The approximation of the true posterior density may lead to inaccurate data association. In addition, due to the use of high-dimensional sampling processes, the computational and storage requirements of these methods are increased.

[0004] In recent years, there has been a significant surge in the use of deep learning-based MTT algorithms, especially for object tracking tasks in the field of computer vision. These methods involve optimizing models with a large number of parameter features by minimizing empirically on datasets tailored to solve a specific problem. Tracking bypass detection algorithms achieve state-of-the-art algorithms using high-performance feature extraction networks and efficient learning-based association methods. However, the improvement in tracking performance comes at the cost of increased computational and model complexity. At the same time, the tracking and association accuracy of these methods heavily rely on extracting features from high-dimensional image data, which limits their applicability in cases where only the target location is known. In recent years, learning-based algorithms have been increasingly applied to radar-based MTT problems. Compared with the high-dimensional information obtained from image-based data, radar data can only provide limited position measurement information, which may hinder the distinction between two close tracks. The above challenging scenarios will make the results of data association worse. Long-short-term memory (LSTM) networks are used for MTT in simple scenarios, where the LSTM network is specifically designed to learn the probability of measurement-to-track association from noisy radar measurements and pre-existing tracks. The above algorithms address the multi-target matching problem considering missing detections and the multi-measurement matching problem with data association. However, current learning-based algorithms applied to radar-based MTT problems require constraints on the number of targets, have high storage requirements, and are slow to compute. In addition, the network design does not take into account specific data features and is combined with tracking algorithms. The network design considers typical features of location data and cannot distinguish between two intersecting tracks, which may result in missed detection or false detection. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to solve the problems of high storage requirements and slow calculation speed of the existing traditional MTT algorithm based on radar data, the learning-based algorithm needs to constrain the number of targets, and the network is designed to consider the typical characteristics of position data. When faced with two intersecting trajectories, it is impossible to distinguish between them, and there may be problems of missed detection or false detection.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The present invention provides a multi-target tracking method based on a cross Transformer data association algorithm, comprising the following steps:

[0009] S100, establishing a dynamic model and a universal measurement equation for a series of maneuvering dynamic targets, considering relative distance and relative angle measurement, and converting the target tracking problem into a posteriori estimation of the target state for data association in step S200;

[0010] S200, the architecture of the data association network based on the cross Transformer, establishes the correlation between the current measurement and the trajectory;

[0011] S300, maintaining each trajectory in step S200 by using an optimal linear unbiased estimation filter and performing a state update;

[0012] S400: Using a scoring-based track management strategy to process unknown targets to be tracked and a changing number of targets to be tracked, determine initialization, update and deletion of tracks.

[0013] Furthermore, in step S100, it includes:

[0014] Consider tracking a group of moving targets, and assume that the dynamic model of the i-th target (i∈{1,2,…,N}) is,

[0015] x i (k) = f(x i (k-1),u i (k),η i (k)) (1)

[0016] Among them, x i (k) and u i (k) is the state and input vector at time k, η i (k) is the process noise vector, Assume a zero-mean Gaussian process;

[0017] The general measurement equation is,

[0018] z i (k) = h(x i (k),ν i (k)) (2)

[0019] Among them, z i (k) is the measured value, v i (k) is the observation noise, Assume a zero-mean Gaussian process;

[0020] Considering relative distance and relative angle measurements,

[0021]

[0022] Among them, (x i (k),y i (k),z i (k) represents the position of target i at time k; r i (k) is the relative distance between the sensor and target i at time k, θ i (k) and φ i(k) are azimuth and elevation respectively; v i,r (k), v i,θ (k) and v i,φ (k) are r i (k),θ i (k) and φ i (k) measurement noise, and assuming that v i,r (k), v i,θ (k) and v i,φ (k) are all zero-mean Gaussian processes, that is and

[0023] Let X(k) = {x i (k)|i=1,2,…,N(k)} represents the state of all targets at k, where the xth i (k) is the state of the ith target, N(k) is the number of targets; let Z(k) = {z j M(k)|j=1,2,…,M(k)} represents all the measured values ​​of the target at k, where z j (k) is the jth measurement value, M(k) is the number of measurements; let represents the set of all target states from the initial moment to time step K, and represents the set of all measurements, where K is the total number of time steps;

[0024] The multi-target tracking problem is formulated as obtaining a mapping as, Convert the target tracking problem into a posterior estimation of the target state.

[0025]

[0026] Where, Θ represents the parameters of the MTT algorithm;

[0027] If new measurements are obtained, past tracking results will be retained;

[0028] Let Y(k)={(y l (k),α l (k))|l=1,2,…,O(k)} represents the initial tracking estimate of the target at the kth moment obtained by association, y l (k) represents the initial tracking estimate of trajectory l at the kth moment, O(k) is the number of trajectories at the kth moment, and α l (k) is the trajectory label, which is used to mark the ownership of trajectory l;

[0029] The multi-target tracking problem can be rewritten as:

[0030]

[0031] Furthermore, in step S200, it includes:

[0032] Assume that there are p trajectory predictions and q measurements at time k; represents the association matrix with p rows and q columns provided by the cross-Transformer-based data association network algorithm at time k; let is p trajectory prediction, {z j (k)|j=1,2,…,q} is q measurement values; let a ij (k), i∈{1,2,…,p},j∈{1,2,…,q} is the j-th row and j-th column element in A(k), r i and c j are the i-th row and j-th column vectors respectively;

[0033] The following three types of dependencies are defined:

[0034] (1) Measurement without associated track:

[0035] (2) Tracks without associated measurements:

[0036] (3) Measurement-track correlation pair:

[0037] In the multi-target tracking problem, each measurement or prediction belongs to one of three relevant cases.

[0038] Furthermore, in step S300, it includes:

[0039] The optimal linear unbiased estimation filter is used, and the calculation process is:

[0040] Forecast, set and P(k-1|k-1) are the state estimates and covariances at time k-1; the state estimates and covariance propagation are,

[0041]

[0042] and,

[0043] P(k|k-1)=F(P)P(k-1|k-1)F(k) T +G(k)Q k G(k) T (7)

[0044] in, P(k|k-1) and P(k|k) represent the prediction and estimation of the state covariance matrix at time k, respectively. and They represent the prediction and estimation of the state at time k respectively; Q k Represents the prior estimation matrix of process noise;

[0045] Update, define and and

[0046] set up and

[0047] set up and

[0048] The updating process of state and covariance is,

[0049]

[0050] and,

[0051] P(k|k)=(IK(k)H(k))P(k|k-1) (9)

[0052] Where I is the identity matrix, The Kalman gain matrix K(k) is,

[0053] K(k)=μ1P(k|k)JS -1 (10)

[0054] Where J is a transformation matrix used to operate P(k|k)J to merge the columns of P(k|k) corresponding to x, y, and z into a new n×3 matrix; the matrix S = [s ij ] 3×3 The elements of are, Among them, ∑ x ,∑ y and∑ z Represent the variance of x, y and z estimates respectively; ∑ xy ,∑ yz and∑ xy denote the covariance between the estimates of x, y, and z, respectively.

[0055] Furthermore, in step S400, it includes:

[0056] S410, obtaining a trajectory estimate of the target using an optimal linear unbiased estimation filter associated with the trajectory;

[0057] set up represents a label set, where is a set of positive integers; let express two subsets of in, is the set of trajectory labels being maintained, is the label set of candidate trajectories to be confirmed;

[0058] Let S l (k) is the trajectory score of the lth trajectory at time k, k≥0; let ΔS l (k) is the increment of the trajectory score at time k;

[0059] Assign trajectory scores with truncation and decay forms,

[0060]

[0061] Among them, λ s ∈(0,1] is the attenuation factor, represents the cutoff parameter for track scoring;

[0062] set up is the initialization time of the tracking trajectory with label l; assuming that for S l (k) = 0; according to the correlation classification, design ΔS l (k) are as follows:

[0063] (1) For targets without any association By using the new label Initialize the best linear unbiased estimation filter to create new candidate trajectories, where And the label l will be added to The state of the target is initialized by measurements and priors, and the initial score of the trajectory l is given in the form of log-likelihood ratio, defined as follows:

[0064]

[0065] Among them, P B represents the probability of the target being born, P FA It indicates the probability that the clutter point cannot be distinguished from the real target; P B and P FA All are calculated in advance based on the sensor characteristics;

[0066] (2) For a trajectory l without measurement, update the trajectory estimate by using the prediction,

[0067]

[0068] and,

[0069]

[0070] The corresponding trajectory score increment is,

[0071]

[0072] Among them, P D represents the target detection probability of the sensor;

[0073] (3) For the relevant measurement value z j (k) The trajectory l, the state and covariance estimates of the trajectory are updated by using formula (8) and formula (9);

[0074] The increment of the corresponding trajectory score is calculated based on the position measurement and the estimate of trajectory l,

[0075]

[0076] Where pr(·) represents the probability density function;

[0077] Given the predicted state at time k and the covariance matrix set up The probability density function of

[0078]

[0079] Assume that the clutter is evenly distributed on the plane of region D, and the number of clutter N FA It follows a Poisson distribution with parameter λ c for,

[0080]

[0081] Substituting distribution formula (17) and formula (18) into formula (16), we obtain:

[0082]

[0083] in,

[0084] S420, Order For the specified confirmation score threshold, the score-based trajectory management strategy is,

[0085] if Then the track l is confirmed; the label of track l is changed from L c Remove and add to L m middle;

[0086] If 0≤S l (k) c , then trajectory l is still considered as a candidate trajectory;​

[0087] If S l (k)<0, then the trajectory l and the corresponding BLUE filter will be deleted;

[0088] For labels And the trajectory in maintenance state: If S l (k) ≥ 0, then the trajectory l will be retained; otherwise, if S l (k)<0, the track l will be deleted and marked as the current label.

[0089] A multi-target tracking system based on a cross-Transformer data association algorithm has a program module corresponding to the above steps, and executes the steps in the multi-target tracking method based on the cross-Transformer data association algorithm when running.

[0090] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of a multi-target tracking method based on a cross-Transformer data association algorithm when called by a processor.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] The present invention provides a multi-target tracking method and system based on a cross Transformer data association algorithm, and solves the multi-target association and trajectory tracking problem based on radar position measurement. We consider a challenging scenario in which the number of targets is unknown and changes dynamically over time. In addition, we also consider the cases of missed detection and false alarm. In order to solve these challenging tasks, a data association network of CTDA is used, and a BLUE filter is adopted to obtain the best estimate of the accuracy and efficiency of radar-based position measurement. In addition, a score-based trajectory management strategy is proposed to manage the target according to the relevant historical measurements. Simulation experiments on various data sets show that the association and tracking method proposed in the present invention is superior to existing methods in complex multi-target tracking scenarios (including trajectory intersections, a large number of false alarms and missed detections). Compared with the RFS-based method with better effect in the existing tracking algorithm, the computational workload and storage requirements are greatly reduced, and real-time tracking is realized; compared with the learning-based tracking algorithm, the problem of its limitation on the number of targets is solved, and a special design is carried out based on radar data, which greatly improves the processing effect of false alarms and missed detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flow chart of a multi-target tracking method based on a cross Transformer data association algorithm in an embodiment of the present invention;

[0094] Figure 2 Flow chart of a data association method based on a cross Transformer network architecture in an embodiment of the present invention;

[0095] Figure 3 Flow chart of data filling and blocking in the data preprocessing process in an embodiment of the present invention;

[0096] Figure 4 : is a diagram of a Transformer-based encoder network structure in an embodiment of the present invention;

[0097] Figure 5 : is a decoder network structure diagram based on Transformer in an embodiment of the present invention;

[0098] Figure 6 This is a data association network diagram based on a cross Transformer in an embodiment of the present invention;

[0099] Figure 7 A diagram of a processing method of the Kuhn-Munkres (KM) algorithm in an embodiment of the present invention;

[0100] Figure 8 This is a sample tracking result diagram of a test data set in a simulation experiment in an embodiment of the present invention;

[0101] Fig. 9 This is a tracking result diagram of the xy plane obtained by using the algorithm in the simulation experiment in the embodiment of the present invention;

[0102] Fig.10 This is a tracking result diagram of the x-axis changing with time obtained by using this algorithm in a simulation experiment in an embodiment of the present invention;

[0103] Fig.11 This is a tracking result diagram of the y-axis changing with time obtained by using this algorithm in a simulation experiment in an embodiment of the present invention;

[0104] Fig.12 It is a comparison curve diagram of the OSPA distance using the algorithm and PHD, CPHD and GLMB in the simulation experiment in the embodiment of the present invention;

[0105] Fig.13 It is a comparison curve diagram of the position between the present algorithm and PHD, CPHD and GLMB in the simulation experiment in the embodiment of the present invention;

[0106] Fig.14 This is a comparison curve chart of the cardinality results of the algorithm used in the simulation experiment in the embodiment of the present invention with those of PHD, CPHD and GLMB. DETAILED DESCRIPTION

[0107] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0108] Specific implementation plan 1: Combine Figures 1 to 7 As shown, the present invention provides a multi-target tracking method based on a cross Transformer data association algorithm, comprising the following steps:

[0109] Step S100: Consider the problem of tracking a group of moving targets. The number of these targets is preset to be unknown in advance. Therefore, the dynamic model of the i-th target (i∈{1,2,…,N}) is given by the following formula:

[0110]

[0111] Among them, x i (k) and u i (k) is the state and input vector at time k, η i (k) is the process noise vector, Assume a zero-mean Gaussian process;

[0112] The general measurement equation is given by,

[0113]

[0114] Among them, z i (k) is the measured value, v i (k) is the observation noise, Assume a zero-mean Gaussian process;

[0115] Specifically, the present invention considers relative distance and relative angle measurements, i.e.,

[0116]

[0117] Among them, (x i (k),y i (k),z i (k) represents the position of target i at time k; r i (k) is the relative distance between the sensor and target i at time k, θ i (k) and φ i (k) are azimuth and elevation respectively; v i,r (k), v i,θ (k) and v i,φ (k) are r i (k),θ i (k) and φ i (k) measurement noise, and assuming that v i,r (k), v i,θ (k) and vi,φ (k) are all zero-mean Gaussian processes, that is and

[0118] Let X(k) = {x i (k)|i=1,2,…,N(k)} represents the state of all targets at k, where the xth i (k) is the state of the ith target, N(k) is the number of targets; let Z(k) = {z j M(k)|j=1,2,…,M(k)} represents all the measured values ​​of the target at k, where z j (k) is the jth measurement value, M(k) is the number of measurements; let represents the set of all target states from the initial moment to time step K, and represents the set of all measurements, where K is the total number of time steps;

[0119] The multi-target tracking problem can be formulated as obtaining a mapping: Among them, for the data set For each element in, there is a dataset There is at most one element corresponding to it in , and vice versa; In actual scenarios, the true state of the target cannot be obtained, so the target tracking problem is transformed into the posterior estimation of the target state, that is, solving the following optimization problem:

[0120]

[0121] Where, Θ represents the parameters of the MTT algorithm;

[0122] Since the MTT method operates in real time, past tracking results are retained rather than optimized once new measurements are obtained;

[0123] Let Y(k)={(y l (k),α l (k))|l=1,2,…,O(k)} represents the initial tracking estimate of the target at the kth moment obtained by association, y l (k) represents the initial tracking estimate of trajectory l at the kth moment, O(k) is the number of trajectories at the kth moment, and α l (k) is the trajectory label, which is used to mark the ownership of trajectory l;

[0124] According to the definition of tracking estimation, the MTT problem can be rewritten as:

[0125]

[0126] By re-expressing formula (4) as formula (5), the multi-target tracking process can be divided into two independent processes: data association process and filtering process;

[0127] The goal of the present invention is to obtain a correlation matrix of two sets of position data, taking into account false positives and missed detections, and accurately tracking multiple target trajectories;

[0128] S200, the architecture of the Cross-Transformer Data Association (CTDA) network, establishes the correlation between the current measurement and the trajectory,

[0129] Assume that there are p trajectory predictions and q measurements at time k; Denotes the incidence matrix with p rows and q columns provided by the CTDA algorithm at time k; let For p trajectory predictions, {z j (k)|j=1,2,…,q} are q measurements; it is worth noting that the association matrix A(k) is usually not full rank because there are missed detections and false alarms in the measurements; let a ij (k), i∈{1,2,…,p},j∈{1,2,…,q} is the j-th row and j-th column element in A(k), r i and c j are the i-th row and j-th column vectors respectively; then the following three types of correlation can be defined:

[0130] (1) Measurement without associated track:

[0131] (2) Tracks without associated measurements:

[0132] (3) Measurement-track correlation pair:

[0133] In the multi-target tracking problem, each measurement or prediction must belong to one of three relevant cases;

[0134] S300, each trajectory in step S200 is maintained by using the best linear unbiased estimation filter and performing a state update,

[0135] Assume that the motion and measurement equations of each target are described in formula (1) and formula (2). Since formula (2) is nonlinear, any nonlinear estimation method, such as the extended Kalman filter (EKF), unscented Kalman filter (UKF) and particle filter, is allowed to perform state estimation. In the multi-target tracking problem considered, the measurement equation is constrained to be the relative distance r, azimuth angle θ and elevation angle φ relative to the target. Let (x, y, z) represent the position of the target and use the measurement equation defined in formula (3). In order to provide the best estimate in terms of accuracy and efficiency under this measurement condition, a variant of the Kalman filter, the best linear unbiased estimation filter (BLUE), is used, which is calculated as follows:

[0136] Forecast: and P(k-1|k-1) are the state estimates and covariances at time k-1; then the state estimates and covariance propagation are,

[0137]

[0138] and,

[0139] P(k|k-1)=F(k)P(k-1|k-1)F(k) T +G(k)Q k G(k) T (7)

[0140] in, P(k|k-1) and P(k|k) represent the prediction and estimation of the state covariance matrix at time k, respectively. k represents the prior estimation matrix of process noise, and They represent the prediction and estimation of the state at time k respectively;

[0141] Update: For brevity, we define the following notation and and

[0142] In addition,

[0143] and Then we have and

[0144] The updating process of state and covariance is,

[0145]

[0146] and,

[0147] P(k|k)=(IK(k)H(k))P(k|k-1) (9)

[0148] Where I is the identity matrix, The Kalman gain matrix K(k) is given by,

[0149] K(k)=μ1P(k|k)JS -1 (10)

[0150] Where J is a transformation matrix that combines the columns of P(k|k) corresponding to x, y, and z into a new n×3 matrix by operating P(k|k)J; the matrix S = [s ij ] 3×3 The elements of are: Among them, Σ x ,Σ y and∑ z Represent the variance of x, y and z estimates respectively; ∑ xy ,Σ yz and Σ xy denote the covariance between the estimates of x, y, and z, respectively;

[0151] S400, using a scoring-based track management strategy to handle unknown targets to be tracked and a changing number of targets to be tracked, determining initialization, updating and deletion of tracks, including,

[0152] Since each trajectory is considered as an independent entity and is effectively maintained by using the best linear unbiased estimation (BLUE) filter of step S300; the scoring strategy aims to assign a score to each trajectory, which is updated according to the trajectory's association history; each trajectory is initialized by a single measurement and is deleted when its score is below a certain threshold;

[0153] S410, in order to track multiple targets simultaneously, a series of BLUE filters associated with the trajectories are used to obtain trajectory estimates of the targets;

[0154] set up represents a label set, where is a set of positive integers; let express two subsets of in, is the set of trajectory labels being maintained, is the label set of candidate trajectories to be confirmed; it should be noted that candidate trajectories are not “real” trajectories, which means that the states corresponding to them are not considered as tracking results before being confirmed as maintenance trajectories;

[0155] Let S l (k) is the trajectory score of the lth trajectory at time k, k≥0; let ΔS l (k) is the increment of the trajectory score at time k, which can also be considered as the evaluation score of the relationship between the measurement value at time k and the trajectory with label l; in order to consider the association information of the historical trajectory, we assign a trajectory score with truncation and decay as follows,

[0156]

[0157] Among them, λ s ∈(0,1] is the attenuation factor, represents the cutoff parameter for track scoring;

[0158] set up is the initialization time of the tracking trajectory with label l; we assume that for S l (k) = 0; According to the correlation classification, we design ΔS l (k) are as follows:

[0159] (4) For targets without any association By using the new label Initialize the BLUE filter to create new candidate tracks, where And the label l will be added to The state of the target is initialized by measurement and priors, and the initial score of the trajectory l is given in the form of log-likelihood ratio (LLR), which is defined as follows:

[0160]

[0161] Among them, P B represents the probability of the target being born, P FA It indicates the probability that the clutter point cannot be distinguished from the real target; P B and P FA All can be calculated in advance based on sensor characteristics;

[0162] (5) For a trajectory l without measurement, the trajectory estimate is updated by using the prediction, i.e.,

[0163]

[0164] and,

[0165]

[0166] At this time, the increment of the corresponding trajectory score is,

[0167]

[0168] Among them, P D represents the target detection probability of the sensor;

[0169] (6) For the relevant measurement value z j (k) of the trajectory l, the state and covariance estimates of the trajectory are updated using equations (8) and (9); in this case, the increment of the corresponding trajectory score is calculated based on the position measurement and the estimate of the trajectory l,

[0170]

[0171] Where pr(·) represents the probability density function;

[0172] Given the predicted state at time k and the covariance matrix And assume The probability density function of Right now,

[0173]

[0174] Assume that the clutter is evenly distributed on the plane of region D, and the number of clutter N FA It follows a Poisson distribution with parameter λ c for,

[0175]

[0176] Substituting distribution formula (17) and formula (18) into formula (16), we obtain:

[0177]

[0178] in,

[0179] S420, Order For the specified confirmation score threshold, we have the following score-based trajectory management strategy,

[0180] For labels with The candidate trajectory of

[0181] if Then the track l is confirmed; the label of track l is changed from L c Remove and add to L m middle;

[0182] If 0≤S l (k) c , then trajectory l is still considered as a candidate trajectory;

[0183] If S l (k)<0, then the trajectory l and the corresponding BLUE filter will be deleted;

[0184] For labels And the trajectory in maintenance state: If S l (k) ≥ 0, then the trajectory l will be retained; otherwise, if S l (k)<0, the track l will be deleted and marked as the current label.

[0185] In practice, it is not always necessary that all parameters needed to compute the score are accurate. It is often sufficient to provide parameters similar to those in the real scene. s and λ t , you can fine-tune the performance of the algorithm to better meet specific requirements.

[0186] Specific embodiment 2: Combination Figures 2 to 7 As shown, in step S200, including:

[0187] The data association process aims to solve the problem of determining whether a set of data (which can be measurement data or estimated data) originates from the same target. In the MTT process, new target detection requires "measurement-measurement" data association across multiple sampling periods. In order to update the track and maintain the tracking estimate, "measurement-track" association is required to identify new measurement data for track correction. The correlation between these associations, including "measurement-measurement" and "measurement-track" data associations, is described by the association matrix. Given two sets of data: and To be associated. Then the association matrix of the two sets of data is expressed as,

[0188]

[0189] in,

[0190]

[0191] It should be noted that each column and each row of the incidence matrix is ​​allowed to have at most one non-zero element, i.e. 1, while all other elements are 0. The purpose of this algorithm is to estimate the state of different numbers of targets by using position measurements based on noisy radar;

[0192] ​S210, define a relative distance feature (RelativeDistanceFeature, RDF) matrix to ensure the generalization of normalized features of different data, including:

[0193] set up and D1 and D2 represent two sets of position data, which can be measurements, estimates, and predictions from real targets or false alarms;

[0194] In order to characterize the association relationship between data sets D1 and D2, an m×n matrix of relative distances in the x, y, and z directions is defined respectively, which is called the Relative Distance Feature (RDF) matrix.

[0195] Among them, for the x-axis direction,

[0196] set up The RDF matrix associated with the x direction is as follows,

[0197]

[0198] in, and are the x-direction data from dataset D1 and dataset D2 respectively; bias is a small quantity of appropriate size to avoid data errors caused by division by zero; x mid is the geometric median of the absolute values ​​of all relative distances between two data sets,

[0199]

[0200] The RDF matrices for the y-axis and z-axis directions are expressed as and Can be constructed in the same way;

[0201] In the RDF matrix, a value close to 1 indicates that the corresponding data points in datasets D1 and D2 are similar and are likely to be related to each other; conversely, a value close to 0 indicates that the data points are significantly different and do not belong to the same target;

[0202] S220, data preprocessing, introducing a block method and a filling method, expanding the data in the to-be-associated data set D1 and the data set D2 with additional data of appropriate dimensions and performing block processing, including:

[0203] In many tasks, the number and dimensionality of measurements taken vary significantly; furthermore, even within a single task, the number of measurements taken at different times may vary;

[0204] When tracking a limited number of targets, an appropriate maximum padding value can be pre-allocated; however, when faced with a large number of targets and clutter, an excessively large padding value becomes computationally infeasible; in order to make the data association algorithm proposed in the present invention applicable to data sets of different sizes, it is necessary to introduce a better data partitioning technique.

[0205] Let N pad is a fixed value, and N pad Less than or equal to m or n; let and are integers for the rows and columns of the scaled padding matrix, is the round-up operation;

[0206] make is the fill matrix associated with x after partitioning,

[0207]

[0208] In order to adapt to the input dimension of the CTDA network, it is further divided as follows,

[0209]

[0210] in, and are the number of partitions for rows and columns respectively; each partition submatrix is an N pad ×N pad matrix;

[0211] Using the partitioning defined in Equation (25), the scaled padding matrix is ​​reshaped into the batch matrix as follows,

[0212]

[0213] The padding matrices associated with directions y and z can be defined in the same way as direction x; subsequently, the augmented matrix containing the relative distance features is used as the input of the CTDA network;

[0214] Compared with the prior art, the data filling technology of the present invention has a clearer physical meaning, because it is equivalent to generating multiple false alarms at a place infinitely far away from the original data; therefore, in the entire filling and partitioning process, no additional information is introduced into the RDF matrix; it is ensured that the basic characteristics of the matrix after the filling process are not affected;

[0215] S230, build a data association network based on CrossTransformer, including,

[0216] S231, build the Transformer encoder, including three integrated modules, namely Multi-Head Attention (MHA), Feed Forward Network (FFN) and Normalization Network, such as Figure 4 As shown, the most important one is the attention module, which is the cornerstone of the MHA backbone;

[0217] The main goal of the self-attention neural network is to use Query to retrieve the attention values ​​corresponding to Keys and assign them to the corresponding Values; then redistribute the Value according to the attention weight; let Attention represent the attention processing, which is as follows,

[0218]

[0219] in, and Represent Query and Key matrices respectively; represents the y matrix value, d is the dimension of Q, K and V in the self-attention neural network, Q, K and V represent the same feature encoding;

[0220] Furthermore, by using the matrix W Q , W K and W V Combined with the multi-head attention network, different types of relationships can be extracted from the data.

[0221] MultiHead(Q,K,V)=Concat(head1,…,head n )W O (28)

[0222] in, is each attention network head, and W O It is a trainable parameter, which is equivalent to the Q, K and V tensors having the same shape;

[0223] S232. Construct a Transformer decoder, including two multi-head attention networks (MHAs), set a mask layer in the first multi-head attention network, and the mask layer can be configured as a matrix with all elements equal to 1, which is the same as the standard multi-head attention network; use Q, K and V in the second multi-head attention network;

[0224] Specifically, the decoder input is used as the query feature, while the encoder input is used as the key and value; this arrangement enables the decoder to retrieve information from the encoder output using the encoded features obtained through multi-head attention as a query; subsequently, the resulting weighted input encoding is combined with the decoder input and normalized and fed-forwarded to produce the decoder output;

[0225] S233, combining the Transformer encoder of step S231 and the Transformer decoder of step 232 to construct a cross-Transformer-based data association network (CTDA), including:

[0226] Construct a row-column crossover Transformer network, which is applied to the three channels of the input matrix; each channel is processed separately by a row Transformer encoder and a column Transformer encoder, which treat the rows and columns of the input as feature vectors respectively; it is worth noting that the input of each channel is processed independently; the output of the encoder is then input to the row and column Transformer decoders in a crossover mode;

[0227] Since the Transformer decoder includes two MHA modules, in the context of the row-column cross Transformer network, the outputs of the row and column encoders serve as the Key and Value inputs of the second MHA module in the row and column Transformer decoders, respectively; at the same time, the output of each encoder is cross-fed into another Transformer decoder as the input of the first MHA module layer; the outputs of the row and column decoders (each channel consists of two matrices) are combined using a processing module; in order to meet different requirements, an element-by-element method is introduced to improve the efficiency of this algorithm,

[0228]

[0229] Among them, h e is the output of the mixing matrix, Reshape(·) means converting the dimension of the matrix to a specified number, [·] means the concatenation operation, W and b are trainable weights and biases, and h1 and h2 are decoders;

[0230] A 3-channel cross Transformer network is constructed. For the 3-channel encoded data obtained by bidirectional processing, a cross Transformer network structure is implemented; the output of the element-by-element mixing processing module consisting of the 3-channel data is initially directed to three Transformer encoders to obtain the encoded features; then, the output of each encoder corresponding to each channel is simultaneously fed to the first MHA module of the decoder of other channels; in addition, each output is simultaneously used as the key and value input of the second MHA module in the corresponding decoder of the channel; then the output matrix is ​​obtained by element-by-element mixing,

[0231]

[0232] Among them, O1, O2 and O3 are the outputs of the decoders corresponding to the three channels;

[0233] The cross transformer used in the 3-channel cross transformer network is different from the corresponding cross transformer in the row-column cross transformer; first, the 3-channel cross transformer network contains a reduced number of layers compared to the row-column transformer module. This adjustment is made by observing that the information fusion between different channels is often relatively less complex than the operations required by the row-column cross transformer module. Using a deeper network in this particular case may lead to overfitting; second, the number of transformers in the cross structure applied between each pair of channels varies with the dimension of the data, rather than being fixed; in addition, the number of encoders and decoders can be different;

[0234] S234. Since the data association matrix exhibits sparsity and is composed of binary values ​​(0-1 matrix), the binary cross entropy loss is used as the cost to calculate the relative association probability of the network output.

[0235]

[0236] in, and The incidence matrix O pr and the true incidence matrix O gt The predicted element in row i and column j;

[0237] S240, each element of the matrix obtained from the cross-Transformer-based data association network represents the association probability of a given input data pair; in order to further convert the network output into an association matrix, the Kuhn-Munkres (KM) algorithm, also known as the Hungarian matching algorithm, is used to find the best assignment. The specific steps are combined Figure 7 As shown;

[0238] It is worth noting that the KM algorithm is specifically designed to solve complex situations and is very useful when dealing with more challenging scenarios; in most cases, it is sufficient to directly threshold the network output to generate the association matrix.

[0239] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0240] Specific implementation scheme three: The present invention provides a multi-target tracking system based on a cross-Transformer data association algorithm. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned multi-target tracking method based on the cross-Transformer data association algorithm during operation.

[0241] The other combinations and connection relationships of this embodiment are the same as those of the specific embodiment one or two.

[0242] Specific implementation scheme four: The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a multi-target tracking method based on a cross-Transformer data association algorithm when called by a processor.

[0243] The other combinations and connection relationships of this embodiment are the same as those of the specific embodiment one or two.

[0244] Simulation experiment

[0245] A customized 3D dataset is generated and used to evaluate the association accuracy of the proposed algorithm, and a challenging 2D multi-target tracking scenario is used to compare the proposed multi-target tracking algorithm with several existing popular multi-target tracking algorithms, including PHD, CPHD and GLMB. Monte Carlo simulation is used to evaluate the tracking effectiveness in this scenario.

[0246] In order to evaluate the results of different multi-target tracking algorithms, the Optimal Sub-Pattern Assignment (OSPA) distance is widely used; OSPA distance is a metric that captures the difference between two sets by considering the difference in the number of points and the difference in point values. It provides association accuracy and estimation accuracy. Define the target true position set as P = {p1, p2, ... p m}, the estimated state set of the target is where m and n are the number of ground truth and estimated targets respectively; p∈P, are the real target state vector and the estimated target state vector at a certain moment. Then the calculation of OSPA distance is,

[0247]

[0248] Among them, n Represents all combinations of m elements in set X, the number of combinations is Indicates that the minimum distance is found using the following expression; r is the distance weight parameter, 1≤r≤inf, is the cutoff distance between two positions, where c>0 is the cutoff parameter;

[0249] The estimates of localization and cardinality errors are given by,

[0250]

[0251] and,

[0252]

[0253] It can be seen and The value of This improves the accuracy of positioning and target quantity estimation.

[0254] On the other hand, when evaluating the results of multi-target association, the root mean square error (RMSE) is usually used for single-target evaluation. RMSE only focuses on the estimation accuracy of a single target.

[0255] The proposed CTDA method is able to associate two-dimensional or three-dimensional trajectory data. The advantage is that no prior quantitative information, such as false alarm probability and missed detection probability, is required. In order to train the network in CTDA, a large amount of data is required to enhance the generalization ability of the network and prevent overfitting. However, obtaining real trajectory data can be challenging, and the available data is often limited. To overcome these limitations, we developed a comprehensive multi-objective simulation platform to generate a large amount of simulated data of trajectory measurements for network training.

[0256] First, build a 3D dynamic model and a measurement model to generate training data. Select the state vector as x = [x, y, z, v, ζ, ψ] T and the measurement z = [r,θ,φ], we have,

[0257]

[0258] Where x, y and z represent the position of the target; v, ζ and ψ represent the velocity, pitch angle and yaw angle respectively; θ and φ represent the azimuth and elevation angle respectively; u D 、u L and u Gare the control of velocity, pitch angle and yaw angle respectively; η is the process noise and measurement noise vector. The measurement equation is described in formula (3). The parameters of the customized multi-target dataset are shown in Table 1.

[0259] Table 1

[0260]

[0261]

[0262] By using these parameters, a total of 2000 trajectories were generated. Among them, 1800 trajectories were used for training and the remaining 200 trajectories were reserved for testing the network performance.

[0263] The network was trained for 8 epochs on the training dataset with a batch size of 256. The Adam optimizer was used with a learning rate of 10 -4 Since the training process is relatively short, no learning rate decay is used.

[0264] In the CTDA network of this algorithm, the data dimension is selected as 40, which corresponds to the size of the input and output matrices. The multi-head attention (MHA) module in CTDA uses 4 heads. A dropout rate of 0.3 is selected for regularization. During testing and filtering, a threshold of 0.1 (λ a =0.1).

[0265] The sample tracking results of the test dataset are as follows Figure 8 The gray image depicts all measurements, while the tracks of multiple targets are distinguished by different colors. The results presented illustrate the accurate estimation of tracks from noisy measurements, as well as the successful identification of track birth and disappearance events.

[0266] Next, several simulation experiments are conducted to verify and evaluate the proposed association and multi-target tracking algorithms. In addition, we compare the proposed algorithm with RFS-based tracking algorithms including PHD, CPHD, and GLMB to demonstrate the advantages of the proposed MTT algorithm.

[0267] To make a fair comparison, the parameter settings used in the experiments are the same as in . The experiments study a variable number of objects (up to 10) over time. These objects undergo births, deaths, and transits. The kinematics of each individual object is calculated using the state vector to describe, where [x,y] is the plane position, is the speed, and ω is the turning rate. The sampling period is set to 1 second. In discrete time, the state equation of each target is,

[0268] x k+1 =Fω x k +v ω (twenty three)

[0269] set up,

[0270]

[0271] in, is the process noise, σ ω =15m / s 2 and σ u =(π / 180)rad / s is the standard deviation of process noise.

[0272] The measurement results are noisy 2D position and distance detection as,

[0273]

[0274] The range is limited to a disk with a radius of 2000m, and the standard deviations of the noise are σ r =5m and σ θ =(π / 360)rad. According to the experimental settings in, the target is set to be born near four positions. The training parameters are shown in Table 2 below.

[0275] Table 2

[0276]

[0277] Tracking results such as Fig. 9 , Fig.10 , Fig.11 As shown in the figure. The grey plot represents all the measurements and the tracks corresponding to multiple targets are marked with different colors. The tracking results demonstrate the effectiveness of the proposed algorithm, which successfully tracks multiple targets in real time and accurately assigns unique labels to each target. This is evident from the different colors assigned to the targets. Most importantly, the proposed algorithm is able to handle track crossing situations with false positives and missed detections while also being able to resume tracking after encountering adverse situations. In particular, the algorithm ensures that the track labels do not switch during the entire tracking process, which is very beneficial for practical applications.

[0278] To further demonstrate the tracking performance of our method, we compare it with PHD, CPHD, and GLMB. We perform 100 Monte Carlo simulations on the test dataset, and the average OSPA distance of each method over time is shown in the figure. Fig.12 (Choose c=100 and r=1). Correspondingly, the individual position and cardinality results are as follows: Fig.13 and Fig.14Table 3 summarizes the OSPA distance, position, and cardinality results obtained from our method and the comparison methods.

[0279] Table 3

[0280]

[0281] As shown in Table 3, our method outperforms all other methods in terms of OSPA distance. Our method ranks second in terms of OSPA position, while achieving similar results to other methods in terms of OSPA cardinality. Fig.13 and Fig.14 It can be seen that our method produces tracking accuracy similar to that of PHD, surpasses that of CPHD and GLMB, and performs better than PHD in terms of OSPA cardinality. Fig.14 This demonstrates the rapid response of our method when the target is born or disappears. It can be seen that at the main nodes when the target is born or disappears, including 10s, 60s, 80s and other time nodes, this method can drop to the normal tracking state more quickly.

[0282] It is worth mentioning that the proposed tracking method has a relatively low complexity due to the sample-free algorithm and the number of filters under the scale is linear with the number of targets. In addition, the running time of the complete tracking process on our platform (Inteli7-8700K) is 0.3 seconds, which is suitable for real-time tracking.

[0283] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A multi-target tracking method based on a cross-Transformer data association algorithm, characterized in that: The following steps are involved: S100, establishing a dynamic model and a universal measurement equation for a series of maneuvering dynamic targets, considering relative distance and relative angle measurement, and converting the target tracking problem into a posteriori estimation of the target state for data association in step S200; S200, the architecture of the data association network based on the cross Transformer, establishes the correlation between the current measurement and the trajectory; S300, maintaining each trajectory in step S200 by using an optimal linear unbiased estimation filter and performing a state update; S400: Using a scoring-based track management strategy to process unknown targets to be tracked and a changing number of targets to be tracked, determine initialization, update and deletion of tracks.

2. According to claim 1, a multi-target tracking method based on a cross-Transformer data association algorithm is characterized in that: In step S100, it includes: Consider tracking a group of moving targets, and assume that the dynamic model of the i-th target (i∈{1,2,…,N}) is, x i (k)=f(x i (k-1),u i (k),η i (k)) (1) Among them, x i (k) and u i (k) is the state and input vector at time k, η i (k) is the process noise vector, Assume a zero-mean Gaussian process; The general measurement equation is, z i (k)=h(x i (k),v i (k)) (2) Among them, z i (k) is the measured value, v i (k) is the observation noise, Assume a zero-mean Gaussian process; Considering relative distance and relative angle measurements, Among them, (x i (k),y i (k),z i (k) represents the position of target i at time k; r i (k) is the relative distance between the sensor and target i at time k, θ i (k) and φ i (k) are azimuth and elevation respectively; v i,r (k), v i,θ (k) and ν i,φ (k) are r i (k),θ i (k) and φ i (k) measurement noise, and assuming that v i,r (k), v i,θ (k) and v i,φ (k) are all zero-mean Gaussian processes, that is and Let X(k) = {x i (k)|i=1,2,…,N(k)} represents the state of all targets at k, where the xth i (k) is the state of the ith target, N(k) is the number of targets; let Z(k) = {z j M(k)|j=1,2,…,M(k)} represents all the measured values ​​of the target at k, where z j (k) is the jth measurement value, M(k) is the number of measurements; let represents the set of all target states from the initial moment to time step K, and represents the set of all measurements, where K is the total number of time steps; The multi-target tracking problem is formulated as obtaining a mapping as, Convert the target tracking problem into a posterior estimation of the target state. Where, Θ represents the parameters of the MTT algorithm; If new measurements are obtained, past tracking results will be retained; Let Y(k)={(y l (k),α l (k))|l=1,2,…,O(k)} represents the initial tracking estimate of the target at the kth moment obtained by association, y l (k) represents the initial tracking estimate of trajectory l at the kth moment, O(k) is the number of trajectories at the kth moment, and α l (k) is the trajectory label, which is used to mark the ownership of trajectory l; The multi-target tracking problem can be rewritten as:

3. The multi-target tracking method based on the cross Transformer data association algorithm according to claim 2 is characterized in that: In step S200, it includes: Assume that there are p trajectory predictions and q measurements at time k; represents the association matrix with p rows and q columns provided by the cross-Transformer-based data association network algorithm at time k; let is p trajectory prediction, {z j (k)|j=1,2,…,q} is q measurement values; let a ij (k), i∈{1,2,…,p},j∈{1,2,…,q} is the j-th row and j-th column element in A(k), r i and c j are the i-th row and j-th column vectors respectively; The following three types of dependencies are defined: (1) Measurement without associated track: (2) Tracks without associated measurements: (3) Measurement-track correlation pair: In the multi-target tracking problem, each measurement or prediction belongs to one of three relevant cases.

4. The multi-target tracking method based on the cross-Transformer data association algorithm according to claim 3 is characterized in that: In step S300, it includes: The optimal linear unbiased estimation filter is used, and the calculation process is: Forecast, set and P(k-1|k-1) are the state estimates and covariances at time k-1; the state estimates and covariance propagation are, and, P(k|k-1)=F(k)P(k-1|k-1)F(k) T +G(k)Q k G(k) T (7) in, P(k|k-1) and P(k|k) represent the prediction and estimation of the state covariance matrix at time k, respectively. and They represent the prediction and estimation of the state at time k respectively; Q k Represents the prior estimation matrix of process noise; Update, define and and set up and set up and The updating process of state and covariance is, and, P(k|k)=(iK(k)H(k))P(k|k-1) (9) Where I is the identity matrix, The Kalman gain matrix K(k) is, K(k)=μ1P(k|k)JS -1 (10) Where J is a transformation matrix used to operate P(k|k)J to merge the columns of P(k|k) corresponding to x, y, and z into a new n×3 matrix; the matrix S = [s ij ] 3×3 The elements of are, Among them, Σ x ,Σ y and Σ z Represents the variance of x, y and z estimates respectively; Σ xy ,Σ yz and∑ xy denote the covariance between the estimates of x, y, and z, respectively.

5. The multi-target tracking method based on the cross Transformer data association algorithm according to claim 4 is characterized in that: In step S400, it includes: S410, obtaining a trajectory estimate of the target using an optimal linear unbiased estimation filter associated with the trajectory; set up represents a label set, where is a set of positive integers; let express two subsets of in, is the set of trajectory labels being maintained, is the label set of candidate trajectories to be confirmed; Let S l (k) is the trajectory score of the lth trajectory at time k, Let ΔS l (k) is the increment of the trajectory score at time k; Assign trajectory scores with truncation and decay forms, Among them, λ s ∈(0,1] is the attenuation factor, represents the cutoff parameter for track scoring; set up is the initialization time of the tracking trajectory with label l; assuming that for S l (k) = 0; according to the correlation classification, design ΔS l (k) are as follows: (1) For targets without any association The measurement is done by using the new label Initialize the best linear unbiased estimation filter to create new candidate trajectories, where And the label l will be added to The state of the target is initialized by measurements and priors, and the initial score of the trajectory l is given in the form of log-likelihood ratio, defined as follows: Among them, P B represents the probability of the target being born, P FA It indicates the probability that the clutter point cannot be distinguished from the real target; P B and P FA All are calculated in advance based on the sensor characteristics; (2) For a trajectory l without measurement, update the trajectory estimate by using the prediction, and, The corresponding trajectory score increment is, Among them, P D represents the target detection probability of the sensor; (3) For the relevant measurement value z j (k) The trajectory l, the state and covariance estimates of the trajectory are updated by using formula (8) and formula (9); The increment of the corresponding trajectory score is calculated based on the position measurement and the estimate of trajectory l, Where pr(·) represents the probability density function; Given the predicted state at time k and the covariance matrix set up The probability density function of Assume that the clutter is evenly distributed on the plane of region D, and the number of clutter N FA It follows a Poisson distribution with parameter λ c for, Substituting distribution formula (17) and formula (18) into formula (16), we obtain: in, S420, Order For the specified confirmation score threshold, the score-based trajectory management strategy is, if Then the track l is confirmed; the label of track l is changed from L c Remove and add to L m middle; If 0≤S l (k) c , then trajectory l is still considered as a candidate trajectory;​ If S l (k)<0, then the trajectory l and the corresponding BLUE filter will be deleted; For labels And the trajectory in maintenance state: If S l (k) ≥ 0, then the trajectory l will be retained; otherwise, if S l (k)<0, the track l will be deleted and marked as the current label.

6. A multi-target tracking system based on a cross-Transformer data association algorithm, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 5 above, and executes the steps in the above-mentioned multi-target tracking method based on the cross-Transformer data association algorithm when running.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the multi-target tracking method based on the cross-Transformer data association algorithm described in any one of claims 1 to 5 when called by a processor.

Images