Multi-target tracking method and system based on spatial relationship and long and short term trajectory prediction

Abstract

The invention discloses a multi-target tracking method based on spatial relation and long and short term trajectory prediction. The method comprises the following steps: acquiring all detection targets and survival trajectories in a current frame image; obtaining tracks of all matched detection targets in the previous frame of image as motion tracks; updating the binding target of each activity track based on the binding scores of each activity track and other activity tracks; obtaining a predicted position of each survival track in the current frame based on the historical positions of the survival tracks; obtaining a matching cost matrix based on the detection target and the prediction position, and matching the survival track with the detection target to obtain a matched survival track, an unmatched survival track and an unmatched detection target; obtaining a retrieved trajectory based on the unmatched survival trajectory and the bound target thereof; performing low-score target frame position correction based on the matched survival trajectory to obtain a corrected trajectory; and based on the matching result, the retrieved track and the corrected track, position state updating is correspondingly carried out. And multi-target accurate tracking is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and more specifically to a multi-target tracking method and system based on spatial relationship and long- and short-term trajectory prediction. Background Art

[0002] Multi-target tracking is an important research problem in the field of computer vision, which is widely used in video surveillance and security, autonomous driving, intelligent transportation, virtual reality and other fields. The task aims to simultaneously identify and track multiple targets through video sequences. Unlike single target tracking, multi-target tracking not only requires identifying each target, but also needs to handle the appearance, disappearance, occlusion, and intersection of targets in a complex dynamic environment.

[0003] At present, detection-based multi-target tracking methods often use Kalman filters for linear modeling or only consider their historical trajectories and use deep learning methods for modeling when predicting trajectories, while ignoring the influence between trajectories, especially in dense scenes. At the same time, when the trajectory is lost for a long time due to occlusion, the existing methods simply treat the predicted trajectory position as its actual position, which will cause the prediction error to become larger and larger. Even if the subsequent lost trajectory is successfully detected by the detector, it will not be successfully matched and associated due to the large gap between the predicted trajectory position and the detected target position, resulting in tracking failure. For the feature representation of the trajectory, the existing methods often use a single feature storage method. However, for trajectories that have been lost for a long time, their features cannot be updated for a long time and there is only a single feature representation, which will limit the retrieval of the lost trajectory to a certain extent, making the feature representation not stable enough for occlusion. At the same time, if the trajectory features at each moment are stored separately, it will bring great overhead.

[0004] Therefore, how to overcome the occlusion and loss of targets during multi-target tracking, and thus better achieve nonlinear motion prediction of trajectories and recovery after loss, is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a multi-target tracking method and system based on spatial relationship and long-term and short-term trajectory prediction, so as to better deal with the occlusion and loss of targets in the multi-target tracking process, and further better realize the nonlinear motion prediction of trajectories and recovery after loss. The robustness of multi-target tracking to occlusion loss and complex motion can be enhanced, and more accurate tracking of multiple targets can be achieved.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] Multi-target tracking methods based on spatial relationships and long- and short-term trajectory prediction include:

[0008] Get all detected targets and survival tracks in the current frame image;

[0009] Obtaining all trajectories matching the detection target in the previous frame image as active trajectories;

[0010] updating the binding target of each activity track based on the binding score between each activity track and other activity tracks;

[0011] Obtaining a predicted position of each of the survival trajectories in a current frame based on the historical positions of the survival trajectories;

[0012] Obtaining a matching cost matrix based on the detected target and the predicted position;

[0013] Matching the survival trajectory and the detection target based on the matching cost matrix to obtain a matched survival trajectory, an unmatched survival trajectory and an unmatched detection target;

[0014] Obtaining a retrieval trajectory based on the unmatched survival trajectory and its binding target;

[0015] Correcting the position of the low-score target frame based on the matched survival trajectory to obtain a corrected trajectory;

[0016] The position state is updated accordingly based on the matched survival trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

[0017] Preferably, updating the binding target of each activity track specifically includes:

[0018] Sequentially obtain the binding score between each of the activity tracks and other activity tracks;

[0019] Select the activity track corresponding to the highest binding score greater than the binding threshold to update the binding target of the corresponding activity track, and record the corresponding binding score;

[0020] The binding score acquisition method is:

[0021] Obtaining historical bounding box information of all said activity tracks;

[0022] Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a center point distance set and a bottom vertical distance set are obtained respectively;

[0023] The binding score between the two activity tracks is obtained based on the center point distance set and the bottom vertical distance set.

[0024] Preferably, obtaining the binding score between the two activity tracks based on the center point distance set and the bottom vertical distance set specifically includes:

[0025] Taking average values ​​based on the center point distance set and the bottom vertical distance set, respectively, to obtain a center mean and a bottom mean;

[0026] Obtaining the degree of central point dispersion based on the central mean value;

[0027] Obtaining center point correlation based on the center point dispersion degree and the center mean value;

[0028] Obtaining a bottom correlation of a bounding box based on the central point discreteness and the bottom mean value;

[0029] The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

[0030] Preferably, the predicted position acquisition method is:

[0031] Obtaining long-term motion information of the survival trajectory to be predicted based on the historical position of the survival trajectory;

[0032] Extracting the influence of each of the surviving trajectories on the surviving trajectory to be predicted based on the spatial position motion state of all the surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, and obtaining the influence of each of the surviving trajectories on the surviving trajectory to be predicted as comprehensive influence information;

[0033] Obtaining a predicted position of the to-be-predicted survival trajectory in the current frame based on the fusion extraction of the long-term motion information and the comprehensive impact information;

[0034] The above process is repeated to perform position prediction on all the surviving trajectories, and the predicted position of each surviving trajectory in the current frame is obtained.

[0035] Preferably, the matching cost matrix acquisition method is:

[0036] Calculate the IoU distance based on the detected target and the predicted position to obtain a position distance cost matrix;

[0037] Obtaining a feature cost matrix based on a cosine distance between a feature representation of the detection target and a feature representation of the survival trajectory;

[0038] The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

[0039] Preferably, matching the survival trajectory and the detection target based on the matching cost matrix specifically includes:

[0040] Based on the matching cost matrix, the Hungarian algorithm is used to match the survival trajectory and the detection target;

[0041] The detection targets are divided into high-score detection targets and low-score detection targets based on confidence scores;

[0042] The matching process adopts a two-stage matching association method:

[0043] In the first stage, all the survival trajectories are associated and matched with the high-score detection targets to obtain high-score matching trajectories;

[0044] In the second stage, the surviving trajectory that was not successfully matched and associated in the first stage is matched and associated with the low-score detection target to obtain a low-score matching trajectory;

[0045] The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

[0046] Preferably, obtaining the retrieval track based on the unmatched survival track and its binding target specifically includes:

[0047] Based on the unmatched survival trajectory and its binding target, determining whether the binding trajectory of the unmatched survival trajectory successfully matches the detection target;

[0048] If so, based on the binding relationship between the unmatched survival trajectory and its binding trajectory and the spatial position relationship in the previous frame, the current estimated position of the unmatched survival trajectory is obtained, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence, to obtain the recovered trajectory;

[0049] Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

[0050] Preferably, obtaining the corrected trajectory specifically includes:

[0051] Based on the binding relationship between the low-score matching trajectory and its binding trajectory and the spatial position relationship in the previous frame, a current inferred position of the low-score matching trajectory is obtained, recorded as a second inferred position, and a confidence of the second inferred position is obtained as a second inferred confidence;

[0052] The bounding box of the low-score detection target is corrected based on the second estimated position to obtain a corrected bounding box and a corrected confidence, and then the corrected trajectory is obtained.

[0053] Preferably, the location status update specifically includes:

[0054] Based on the predicted position and the associated position, the high-score matching trajectory is updated;

[0055] updating the position of the retrieved trajectory based on the first estimated position and the first estimated confidence;

[0056] updating the position of the corrected trajectory based on the corrected bounding box and the corrected confidence;

[0057] Based on the predicted position corresponding to the lost track as the new position of the lost track;

[0058] updating a feature representation of the matching survival trajectory based on a confidence score of the detection target matched by the matching survival trajectory;

[0059] Initialize the unmatched high-score detection target as a new survival trajectory;

[0060] Based on the new survival track and the matching survival track states are both set to "active";

[0061] The status based on the retrieved track is set to "retrieved";

[0062] The status based on the lost track is set to "lost".

[0063] A multi-target tracking system based on spatial relationship and long-term and short-term trajectory prediction, including: trajectory acquisition module, binding update module, position prediction module, matching output module, trajectory retrieval module, trajectory correction module and position status update module;

[0064] The trajectory acquisition module is used to acquire all detected targets and survival trajectories in the current frame image; and acquire all trajectories matching the detected targets in the previous frame image as active trajectories;

[0065] The binding update module is used to update the binding target of each activity track based on the binding score between each activity track and other activity tracks;

[0066] The position prediction module is used to obtain the predicted position of each survival trajectory in the current frame based on the historical position of the survival trajectory;

[0067] The matching output module is used to obtain a matching cost matrix based on the detection target and the predicted position; match the survival trajectory and the detection target based on the matching cost matrix to obtain a matched survival trajectory, an unmatched survival trajectory and an unmatched detection target;

[0068] The track retrieval module is used to obtain a retrieval track based on the unmatched survival track and its binding target;

[0069] The trajectory correction module is used to correct the position of the low-score target frame based on the matched survival trajectory to obtain a corrected trajectory;

[0070] The position status updating module is used to update the position status accordingly based on the matched survival trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

[0071] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multi-target tracking method and system based on spatial relationship and long-term and short-term trajectory prediction, which has the following beneficial effects:

[0072] 1. The present invention can better model the nonlinear motion of the tracking target and the influence between trajectories in dense scenes by combining long-term temporal information with short-term spatial interaction to make predictions, thereby improving the prediction accuracy of the survival trajectory.

[0073] 2. The present invention uses the Hungarian algorithm to perform two-stage matching on high-score detection and low-score detection respectively, wherein the matching cost matrix consists of two parts: the IoU cost matrix representing the motion and the feature cost matrix. In this way, the feature distance calculation method of multiple confidence intervals can better improve the stability of the feature representation to occlusion, thereby improving the performance of the tracker in re-identifying lost trajectories.

[0074] 3. For the unsuccessfully matched trajectories, the present invention queries its bound trajectories. If its bound trajectories are not lost, the positions to which it should be associated are inferred through its bound trajectories. These targets with strong binding correlations can be well retrieved when they are lost due to long-term occlusion, thereby alleviating the loss of surviving trajectories caused by high occlusion.

[0075] 4. The present invention matches the trajectory of the low-score detection target by querying its bound trajectory. If its bound trajectory is not lost, the low-score detection frame is adaptively corrected by the position inferred from the bound trajectory. The low-score detection frame is accompanied by occlusion, which will lead to inaccurate positioning of the detection frame. This adaptive correction method can better fit the real trajectory of the target.

[0076] 5. The present invention adopts exponential linear averaging to update the features of the corresponding confidence interval for feature representation update, which can better deal with the interference of occlusion on feature representation while avoiding storing a large amount of historical feature representation.

[0077] 6. Through the above technical solutions, this method can better deal with the occlusion loss problem and complex motion patterns in multi-target tracking in dense scenes, improve the accuracy of the matching process association, and improve the tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0079] Figure 1 Flowchart of the multi-target tracking method based on spatial relationship and long- and short-term trajectory prediction provided by the present invention.

[0080] Figure 2 This is a flow chart of the binding score acquisition method provided by the present invention.

[0081] Figure 3 This is a schematic diagram of the structure of the long-term motion information extraction branch and the short-term motion impact extraction branch provided by the present invention.

[0082] Figure 4 This is a schematic diagram of the Mamba block structure provided by the present invention.

[0083] Figure 5 A schematic diagram of the structure of a multi-target tracking system based on spatial relationships and long- and short-term trajectory prediction provided by the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Example 1

[0086] like Figure 1 As shown, the embodiment of the present invention discloses a multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction, including:

[0087] Get all detected targets and survival tracks in the current frame image;

[0088] Get the trajectories of all matching detection targets in the previous frame image as active trajectories;

[0089] Update the binding target of each activity track based on the binding score of each activity track with other activity tracks;

[0090] Get the predicted position of each surviving trajectory in the current frame based on the historical position of the surviving trajectory;

[0091] Obtain a matching cost matrix based on the detected targets and predicted positions;

[0092] Match the surviving trajectories and the detection targets based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories and unmatched detection targets;

[0093] Get the retrieved trajectory based on the unmatched survival trajectory and its bound target;

[0094] Correct the position of the low-score target frame based on the matched surviving trajectory to obtain the corrected trajectory;

[0095] The position status is updated accordingly based on the matched surviving trajectories, unmatched detected targets, retrieved trajectories, and corrected trajectories.

[0096] Example 2

[0097] The embodiment of the present invention discloses a multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction, including:

[0098] Get all detected targets and survival tracks in the current frame image.

[0099] Preferably, all detected targets in the current frame image and their corresponding feature representation vectors are obtained based on the target detector and the feature extraction network; this embodiment uses YOLOX as the target detector and ReID as the feature extraction network.

[0100] Preferably, the jth detection target is represented by D j =(b j ,s j ,e j ), where b j represents the bounding box of the detected target j, s j represents the confidence score of the detected target j, e j The feature representation vector representing the detection target j. At the same time, according to the confidence score, the detection targets belonging to the interval [0.1, 0.6] are regarded as low-score detection targets, and the detection targets belonging to the interval (0.6, 1] are regarded as high-score detection targets.

[0101] Preferably, the surviving trajectory refers to the trajectory left after the updating step at time t-1 (ie, when processing the t-1th frame); during the updating step, the trajectory that has lost more than 30 frames will be deleted.

[0102] Get the trajectories of all matching detection targets in the previous frame image as active trajectories.

[0103] Preferably, the active trajectory refers to the trajectory successfully matched and associated with the detection target in the previous frame image, including the trajectory newly initialized in the previous frame.

[0104] The binding target of each active track is updated based on the binding scores of each active track with other active tracks.

[0105] Preferably, updating the binding target of each activity track specifically includes:

[0106] Obtain the binding scores between each activity track and other activity tracks in turn;

[0107] Select the activity track corresponding to the highest binding score greater than the binding threshold to update the binding target of the corresponding activity track, and record the corresponding binding score at the same time;

[0108] like Figure 2 As shown, the binding score acquisition method is:

[0109] Get the historical bounding box information of all active tracks;

[0110] Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a center point distance set and a bottom vertical distance set are obtained respectively;

[0111] The binding score between two activity trajectories is obtained based on the set of distances between the center points and the set of vertical distances from the bottom.

[0112] Preferably, the binding score between two activity tracks is obtained based on the center point distance set and the bottom vertical distance set, specifically including:

[0113] Based on the distance set between the center points and the vertical distance set at the bottom, the average values ​​are taken to obtain the center mean and the bottom mean respectively;

[0114] Based on the central mean, the degree of central point dispersion is obtained;

[0115] The correlation of the center points is obtained based on the dispersion of the center points and the center mean;

[0116] The bottom correlation of the bounding box is obtained based on the central point dispersion and the bottom mean value;

[0117] The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

[0118] Preferably, only active trajectories are considered when calculating the binding score to avoid interference caused by missing trajectories. B The active trajectory of the frame is not considered.

[0119] Preferably, historical bounding box information of the trajectory: during the tracking process, for each trajectory starting from initialization, the bounding box position (ie, coordinates) in each frame is stored and recorded, and the stored historical bounding box position is the historical bounding box information of each trajectory.

[0120] Preferably, each track is a tracked target.

[0121] Preferably, consider activity trajectory i and activity trajectory j continuously τ B The historical bounding box information of the frame and the historical bounding box information of the activity track i are The historical bounding box information of activity trajectory j is in represents the bounding box of activity track i in frame r.

[0122] Preferably, the distance between the center points dist is obtained based on the historical bounding box information of the activity trajectory i and the historical bounding box information of the activity trajectory j. center (i,j) and the bottom vertical distance dist bottom (i,j):

[0123]

[0124] Among them, u i and v i They represent the horizontal and vertical coordinates of the center of the bounding box of the activity track i, u j and v j Respectively represent the horizontal and vertical coordinates of the center of the bounding box of the activity track j, y i 2 and y j 2 represents the ordinate of the lower right corner of the bounding box of activity track i and activity track j respectively.

[0125] Based on the distance between the center points dist center (i,j) and the bottom vertical distance dist bottom (i,j) corresponds to the distance set between the center points: And the vertical distance collection from the bottom: Based on the average of the distance set between the center points and the vertical distance set at the bottom, the corresponding center mean is obtained. and bottom mean

[0126] Preferably, based on the central mean Get the degree of dispersion of the center point

[0127]

[0128] Wherein, t indicates that the currently processed frame is the tth frame; r indicates the rth frame, and r=t-1,…,t-τ B ;

[0129] Based on the degree of dispersion of the center point and the central mean Get the center point correlation

[0130] Based on the degree of dispersion of the center point and bottom mean Get bounding box bottom correlation

[0131]

[0132]

[0133] in, represents the average height of the bounding box of activity track i; Represents the average height of the bounding box of activity track j.

[0134] Based on center point correlation Correlation with the bottom of the bounding box Get the binding score σ ij :

[0135]

[0136] Preferably, according to the above calculation steps, the activity trajectory i and the other ones that meet the requirements (i.e., the trajectory length ≥ τ B The binding score between the activity trajectories of frames) and the binding score greater than the binding threshold λ is selected B The largest trajectory in is used as the new binding target to update the binding target of active trajectory i, and record the binding score. B , then unbind the original binding target.

[0137] The predicted position of each surviving trajectory in the current frame is obtained based on the historical position of the surviving trajectory.

[0138] Preferably, the predicted position acquisition method is:

[0139] Based on the historical positions of the survival trajectories, the long-term motion information of the survival trajectories to be predicted is obtained;

[0140] Based on the spatial position motion state of all surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, the influence of each surviving trajectory on the surviving trajectory to be predicted is extracted as comprehensive influence information;

[0141] The predicted position of the survival trajectory to be predicted in the current frame is obtained based on the fusion extraction of long-term motion information and comprehensive impact information;

[0142] Repeat the above process to predict the positions of all surviving trajectories and obtain the predicted position of each surviving trajectory in the current frame.

[0143] Preferably, taking the prediction of survival trajectory j as an example, firstly, the historical position X∈R of survival trajectory j is L×4 Input the long-term motion information extraction branch to extract its long-term motion information, where L represents the input of the historical position of the most recent L consecutive frames (i.e., tL frames to t-1 frames), and each line of X is the historical bounding box position at the corresponding moment (for trajectories whose historical position length is less than L, the bounding box position of the previous frame is used to fill its historical position to a length of L).

[0144] Preferably, Figure 3 As shown in the figure, the long-term motion information extraction branch embeds the input X into a high-dimensional space through a linear transformation, then extracts the motion features of the time series through stacked M Mamba blocks, and takes the Mamba block output of the last time step as the long-term motion information:

[0145]

[0146] in, represents the high-dimensional embedding of X, represents the first weight matrix, D1 represents the dimension after embedding, represents the output of the last time step of the Mamba block, i.e. The last dimension of the output is the long-term motion information.

[0147] Preferably, Figure 4 As shown, the calculation process of the Mamba block is as follows:

[0148]

[0149] Where SSM(·) represents the selective state space model, x t represents the input of the Mamba block at the tth time step, y t represents the input and output of the Mamba block at the tth time step, represents the input of the selective state space model, Conv1D(·) represents one-dimensional convolution, Linear(·) represents linear mapping, σ represents SiLU nonlinear activation function, here The i-th row of corresponds to the input of the Mamba block at the i-th time step.

[0150] Preferably, the short-term motion impact extraction branch is input based on the spatial position motion state of all surviving trajectories in the previous frame to extract the impact of each trajectory on the target trajectory in the short term. The spatial position motion state of the surviving trajectory i is expressed as in and They represent the horizontal and vertical coordinates of the center of the bounding box of the survival trajectory i at the t-1th frame, and They represent the bounding box width and height of the survival trajectory i at the t-1th frame respectively; the last 4 elements represent its motion state at the t-1th frame; That is, the difference in the corresponding coordinates of the bounding box of the survival trajectory i between the last two frames, The calculation of is similar. The spatial position motion states of all surviving trajectories are spliced ​​together to obtain I∈R N×8 , where N is the current number of trajectories, and the i-th row of I corresponds to the spatial position motion state O of the surviving trajectory i i ;

[0151] I and the spatial position motion state O of the survival trajectory j to be predicted j ∈R 1×8 Input the short-term motion influence extraction branch to obtain the influence of each surviving trajectory on the predicted surviving trajectory j as comprehensive influence information; the calculation process of the short-term motion influence extraction branch is as follows:

[0152] Q=O j W2

[0153] K=IW2

[0154]

[0155] in, and Respectively represent O j and a high-dimensional embedding of I with dimension D2; It means to expand K by Q row by row; and Denote the first weight matrix and the second weight matrix respectively; where Q, K, and V correspond to the query, key, and value used in the generated attention mechanism respectively;

[0156] Q, K, and V are input into the multi-head attention layer to extract the various interactive effects between the survival trajectories. The output of the multi-head attention layer is passed through a linear layer to obtain the influence of each survival trajectory on the predicted survival trajectory j as the comprehensive influence information y2:

[0157]

[0158] Among them, Multi_Head_Attention(·) represents the multi-head attention mechanism, Represents the output of the multi-head attention layer; represents the fourth weight matrix.

[0159] Preferably, based on the long-term motion information y of the survival trajectory j to be predicted L After being fused with the comprehensive influence information y2, it is input into the multi-layer perceptron (MLP) to regress and obtain the predicted position of the survival trajectory j in the current frame.

[0160]

[0161] The predicted position obtained will be used to calculate the IoU cost matrix with the position of the detected target for matching association.

[0162] A matching cost matrix is ​​obtained based on the detected targets and predicted positions.

[0163] Preferably, the matching cost matrix is ​​obtained by:

[0164] Calculate the IoU distance based on the detected target and predicted position to obtain the position distance cost matrix;

[0165] Based on the cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory, a feature cost matrix is ​​obtained;

[0166] The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

[0167] Preferably, the IoU distance is calculated based on the detected target and the predicted position to obtain the position distance cost matrix A1:

[0168]

[0169] Among them, represents the bounding box position of the jth detected target in the tth frame; Represents the intersection area of ​​two bounding boxes; Represents the area of ​​the union of two bounding boxes; Represents the value of the element in row i and column j of A1; based on all Get the location distance cost matrix A1.

[0170] Preferably, based on the cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory, a feature cost matrix A2 is obtained:

[0171]

[0172] Among them, D j Represents the detection target j; Represents the feature representation of the detection target j; represents the rth feature representation of survival trajectory i, T i represents the survival trajectory i; represents the element in row i and column j of A2. Based on all Get the location distance cost matrix A2.

[0173] Preferably, when calculating the cosine distance between the feature representation of the detected target and the surviving trajectory, the feature representation of the surviving trajectory takes the confidence score s of the detected target j The corresponding confidence interval is a characteristic representation.

[0174] Preferably, the matching cost matrix A is obtained based on the weighted sum of the location distance cost matrix and the feature cost matrix:

[0175] A=α1A1+(1-α1)A2

[0176] Among them, α1 represents a hyperparameter, which is used to weigh the proportion of location distance cost and feature cost.

[0177] The surviving trajectories and the detected targets are matched based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories and unmatched detected targets.

[0178] Preferably, matching the survival trajectory and the detection target based on the matching cost matrix specifically includes:

[0179] Based on the matching cost matrix, the Hungarian algorithm is used to match the surviving trajectories and the detected targets by minimizing the cost;

[0180] The detection targets are divided into high-score detection targets and low-score detection targets based on the confidence scores;

[0181] The matching process adopts a two-stage matching association method:

[0182] In the first stage, all surviving trajectories are associated and matched with high-score detection targets to obtain high-score matching trajectories;

[0183] In the second stage, the surviving trajectories that were not successfully matched and associated in the first stage are matched and associated with the low-score detection targets to obtain low-score matching trajectories;

[0184] The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

[0185] Preferably, different matching thresholds λ are set for the first high-score matching association process and the second low-score matching association process. high and λ low , match associations are performed for rejections that exceed the threshold.

[0186] The retrieved trajectory is obtained based on the unmatched survival trajectory and its bound target.

[0187] Preferably, obtaining the retrieval track based on the unmatched survival track and its binding target specifically includes:

[0188] Based on the unmatched survival trajectory and its binding target, it is determined whether the binding trajectory of the unmatched survival trajectory successfully matches the detection target;

[0189] If yes, then based on the binding relationship between the unmatched survival trajectory and its binding trajectory and the spatial position relationship in the previous frame, the current estimated position of the unmatched survival trajectory is obtained, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence, to obtain the recovered trajectory;

[0190] Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

[0191] Preferably, for example: unmatched survival trajectory T i , whose binding trajectory is T j , the binding score is σ ij , T j The detection target of the matching association is in and They are survival trajectories T i and its binding trajectory T j Detect the target at the bounding box position at the previous moment The bounding box of It represents the bounding box position of the jth detection target in the tth frame, and the unmatched survival trajectory T is inferred. i At the current position, that is, the first guess position

[0192]

[0193] in, Indicates unmatched survival trajectory T i The first guessed position of the above four equations is multiplied by a scaling factor ( or The purpose of this method is to avoid the effect of scale caused by changes in distance from the camera over time.

[0194] Preferably, at this time, since there is no matching survival trajectory T i No matches found, first guess location The confidence level of The binding score between the two is calculated, and the first guess confidence The calculation formula is as follows:

[0195]

[0196] The position of the low-score target frame is corrected based on the matched surviving trajectory to obtain the corrected trajectory.

[0197] Preferably, obtaining the corrected trajectory specifically includes:

[0198] Based on the binding relationship between the low-score matching trajectory and its binding trajectory and the spatial position relationship in the previous frame, the current inferred position of the low-score matching trajectory is obtained, recorded as the second inferred position, and the confidence of the second inferred position is obtained as the second inferred confidence;

[0199] The bounding box of the low-score detection target is corrected based on the second inferred position to obtain a corrected bounding box and a corrected confidence, and then a corrected trajectory is obtained.

[0200] Preferably, since low-score detection targets are often inaccurately detected due to occlusion, it is considered to use the binding relationship to correct the bounding box of the low-score detection target.

[0201] Preferably, the low score matching trajectory T i With low score detection target Match, and infer the current second guess position of the low-score matching track based on the binding relationship and the second guess confidence ( and The calculation process is the same as above and ) The correction process is as follows:

[0202]

[0203] in, Represents the bounding box of the low-score detection target The corrected bounding box after correction; Is the confidence level for low-score detection targets Corrected confidence after correction.

[0204] The position status is updated accordingly based on the matched surviving trajectories, unmatched detected targets, retrieved trajectories, and corrected trajectories.

[0205] Preferably, the location status update specifically includes:

[0206] Update the position of the high-score matching trajectory based on the predicted position and the associated position;

[0207] updating the position of the retrieved trajectory based on the first inferred position and the first inferred confidence;

[0208] Update the position of the corrected trajectory based on the corrected bounding box and the corrected confidence;

[0209] The predicted position corresponding to the lost track is used as the new position of the lost track;

[0210] The feature representation of the matching surviving trajectory is updated based on the confidence score of the detection target matched by the matching surviving trajectory;

[0211] Initialize the unmatched high-score detection targets as new survival tracks;

[0212] The status of both the new survival track and the matching survival track is set to "active";

[0213] The status based on the recovered track is set to "recovered";

[0214] The status based on the lost track is set to "lost".

[0215] Preferably, the position of the high-scoring matching trajectory is updated based on the predicted position and the associated position:

[0216]

[0217] in, Indicates the new position after the high-score matching trajectory is updated. is the position of the association, is the confidence of the associated position, is the predicted position.

[0218] Preferably, based on the first estimated position and first guess confidence Update the location of the retrieved track:

[0219]

[0220] in, Indicates the new position after retrieving the track update.

[0221] Preferably, based on the modified bounding box and corrected confidence Update the position of the corrected trajectory:

[0222]

[0223] in, Indicates the new position after the corrected trajectory is updated.

[0224] Preferably, based on the predicted position corresponding to the lost trajectory as the new location of the lost track.

[0225] Preferably, the feature representation of the matching surviving trajectory is updated based on the confidence score of the detection target matched by the matching surviving trajectory:

[0226] Matching survival trajectory T i The feature representation is stored as in The corresponding feature representation is within the confidence interval [0.1, 0.5); The corresponding feature representation is within the confidence interval [0.5, 0.9); The corresponding feature representation is within the confidence interval [0.9, 1];

[0227] When updating, the corresponding trajectory feature representation is updated according to the confidence score of the detection target associated with the surviving trajectory. The exponential linear average method is used during the update, such as using the feature representation of detection target j For survival trajectory i The feature representation is updated and the calculation process is:

[0228]

[0229] Where α2 represents a hyperparameter.

[0230] Preferably, the unmatched high-score detection target is initialized as a new survival track (ie, a new tracking target), and the position of the new survival track is initialized to the bounding box position of the detection target; the three feature representations of the new survival track are all initialized to the feature representation of the detection target.

[0231] Preferably, the state of the lost track is set to "lost", and the number of consecutive lost frames is recorded. If the consecutive loss exceeds 30 frames, the track is deleted.

[0232] Preferably, the above steps are repeated for the next frame of image until the tracking is completed.

[0233] Example 3

[0234] like Figure 5 As shown, the multi-target tracking system based on spatial relationship and long-term and short-term trajectory prediction includes: a trajectory acquisition module, a binding update module, a position prediction module, a matching output module, a trajectory retrieval module, a trajectory correction module and a position state update module;

[0235] The trajectory acquisition module is used to obtain all detected targets and survival trajectories in the current frame image; and obtain the trajectories of all matching detected targets in the previous frame image as active trajectories;

[0236] A binding update module, used for updating the binding target of each activity track based on the binding score of each activity track with other activity tracks;

[0237] A position prediction module is used to obtain the predicted position of each survival trajectory in the current frame based on the historical position of the survival trajectory;

[0238] A matching output module is used to obtain a matching cost matrix based on the detected target and the predicted position; based on the matching cost matrix, the surviving trajectory and the detected target are matched to obtain a matched surviving trajectory, an unmatched surviving trajectory and an unmatched detected target;

[0239] The trajectory retrieval module is used to obtain the retrieval trajectory based on the unmatched survival trajectory and its bound target;

[0240] The trajectory correction module is used to correct the position of the low-score target frame based on the matching survival trajectory to obtain the corrected trajectory;

[0241] The position status update module is used to update the position status based on the matching survival trajectory, unmatched detection targets, retrieved trajectory and corrected trajectory.

[0242] Preferably, the implementation process of each functional module in this embodiment corresponds to the above method one by one, and will not be repeated here.

[0243] Example 4

[0244] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0245] Memory, used to store computer programs;

[0246] The processor, when used to execute the program stored in the memory, can implement the multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction as in embodiment 1 or 2.

[0247] The electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction in embodiment 1 or 2.

[0248] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0249] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0250] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target tracking method based on spatial relationships and long- and short-term trajectory prediction, characterized in that: include: Get all detected targets and survival tracks in the current frame image; Obtaining all trajectories matching the detection target in the previous frame image as active trajectories; updating the binding target of each activity track based on the binding score between each activity track and other activity tracks; Obtaining a predicted position of each of the survival trajectories in a current frame based on the historical positions of the survival trajectories; Obtaining a matching cost matrix based on the detected target and the predicted position; Matching the survival trajectory and the detection target based on the matching cost matrix to obtain a matched survival trajectory, an unmatched survival trajectory and an unmatched detection target; Obtaining a retrieval trajectory based on the unmatched survival trajectory and its binding target; Correcting the position of the low-score target frame based on the matched survival trajectory to obtain a corrected trajectory; The position state is updated accordingly based on the matched survival trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

2. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1 is characterized in that: Update the binding target of each active track, including: Sequentially obtain the binding score between each of the activity tracks and other activity tracks; Select the activity track corresponding to the highest binding score greater than the binding threshold to update the binding target of the corresponding activity track, and record the corresponding binding score; The binding score acquisition method is: Obtaining historical bounding box information of all said activity tracks; Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a center point distance set and a bottom vertical distance set are obtained respectively; The binding score between the two activity tracks is obtained based on the center point distance set and the bottom vertical distance set.

3. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 2 is characterized in that: Obtaining the binding score between the two activity tracks based on the center point distance set and the bottom vertical distance set specifically includes: Taking average values ​​based on the center point distance set and the bottom vertical distance set, respectively, to obtain a center mean and a bottom mean; Obtaining the degree of central point dispersion based on the central mean value; Obtaining center point correlation based on the center point dispersion degree and the center mean value; Obtaining a bottom correlation of a bounding box based on the central point discreteness and the bottom mean value; The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

4. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1 is characterized in that: The predicted position acquisition method is: Obtaining long-term motion information of the survival trajectory to be predicted based on the historical position of the survival trajectory; Extracting the influence of each of the surviving trajectories on the surviving trajectory to be predicted based on the spatial position motion state of all the surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, and obtaining the influence of each of the surviving trajectories on the surviving trajectory to be predicted as comprehensive influence information; Obtaining a predicted position of the to-be-predicted survival trajectory in the current frame based on the fusion extraction of the long-term motion information and the comprehensive impact information; The above process is repeated to perform position prediction on all the surviving trajectories, and the predicted position of each surviving trajectory in the current frame is obtained.

5. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1 is characterized in that: The matching cost matrix acquisition method is: Calculate the IoU distance based on the detected target and the predicted position to obtain a position distance cost matrix; Obtaining a feature cost matrix based on a cosine distance between a feature representation of the detection target and a feature representation of the survival trajectory; The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

6. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 5 is characterized in that: Matching the survival trajectory and the detection target based on the matching cost matrix specifically includes: Based on the matching cost matrix, the Hungarian algorithm is used to match the survival trajectory and the detection target; The detection targets are divided into high-score detection targets and low-score detection targets based on confidence scores; The matching process adopts a two-stage matching association method: In the first stage, all the survival trajectories are associated and matched with the high-score detection targets to obtain high-score matching trajectories; In the second stage, the surviving trajectory that was not successfully matched and associated in the first stage is matched and associated with the low-score detection target to obtain a low-score matching trajectory; The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

7. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 6 is characterized in that: The retrieved track is obtained based on the unmatched survival track and its binding target, specifically including: Based on the unmatched survival trajectory and its binding target, determining whether the binding trajectory of the unmatched survival trajectory successfully matches the detection target; If so, based on the binding relationship between the unmatched survival trajectory and its binding trajectory and the spatial position relationship in the previous frame, the current estimated position of the unmatched survival trajectory is obtained, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence, to obtain the recovered trajectory; Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

8. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 7, characterized in that: The corrected trajectory specifically includes: Based on the binding relationship between the low-score matching trajectory and its binding trajectory and the spatial position relationship in the previous frame, a current inferred position of the low-score matching trajectory is obtained, recorded as a second inferred position, and a confidence of the second inferred position is obtained as a second inferred confidence; The bounding box of the low-score detection target is corrected based on the second estimated position to obtain a corrected bounding box and a corrected confidence, and then the corrected trajectory is obtained.

9. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 8, characterized in that: The location status update specifically includes: Based on the predicted position and the associated position, the high-score matching trajectory is updated; updating the position of the retrieved trajectory based on the first estimated position and the first estimated confidence; updating the position of the corrected trajectory based on the corrected bounding box and the corrected confidence; Based on the predicted position corresponding to the lost track as the new position of the lost track; updating a feature representation of the matching survival trajectory based on a confidence score of the detection target matched by the matching survival trajectory; Initialize the unmatched high-score detection target as a new survival trajectory; Based on the fact that the status of the new survival track and the matching survival track are both set to "active"; The status based on the retrieved track is set to "retrieved"; The status based on the lost track is set to "lost".

10. A multi-target tracking system based on spatial relationship and long-term and short-term trajectory prediction, applied to a multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction as claimed in any one of claims 1 to 9, characterized in that: It includes: trajectory acquisition module, binding update module, location prediction module, matching output module, trajectory retrieval module, trajectory correction module and location status update module; The trajectory acquisition module is used to acquire all detected targets and survival trajectories in the current frame image; Obtaining all trajectories matching the detection target in the previous frame image as active trajectories; The binding update module is used to update the binding target of each activity track based on the binding score between each activity track and other activity tracks; The position prediction module is used to obtain the predicted position of each survival trajectory in the current frame based on the historical position of the survival trajectory; The matching output module is used to obtain a matching cost matrix based on the detected target and the predicted position; Matching the survival trajectory and the detection target based on the matching cost matrix to obtain a matched survival trajectory, an unmatched survival trajectory and an unmatched detection target; The track retrieval module is used to obtain a retrieval track based on the unmatched survival track and its binding target; The trajectory correction module is used to correct the position of the low-score target frame based on the matched survival trajectory to obtain a corrected trajectory; The position status updating module is used to update the position status accordingly based on the matched survival trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

Images