Multi-target tracking method and system based on spatial relationship and long-term and short-term trajectory prediction

Abstract

The present invention discloses a multi-target tracking method based on spatial relationships and long- and short-term trajectory prediction, including the following steps: obtaining all detected targets and survival tracks in the current frame image; obtaining the tracks of all matching detected targets in the previous frame image as active tracks; updating the bound targets of each active track based on the binding scores of each active track with other active tracks; obtaining the predicted position of each survival track in the current frame based on the historical positions of the survival tracks; obtaining a matching cost matrix based on the detected targets and the predicted positions and matching the survival tracks with the detected targets to obtain matched survival tracks, unmatched survival tracks, and unmatched detected targets; obtaining retrieved tracks based on the unmatched survival tracks and their bound targets; correcting the positions of low-scoring target frames based on the matched survival tracks to obtain corrected tracks; and updating the position states based on the matching results, the retrieved tracks, and the corrected tracks. This method achieves precise multi-target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and more particularly to a multi-target tracking method and system based on spatial relationships and long- and short-term trajectory prediction. Background Art

[0002] Multi-target tracking is a key research problem in computer vision, with widespread applications in video surveillance and security, autonomous driving, intelligent transportation, and virtual reality. The task is to simultaneously identify and track multiple targets within a video sequence. Unlike single-target tracking, multi-target tracking requires not only identifying each target but also handling the appearance, disappearance, occlusion, and intersection of targets in complex, dynamic environments.

[0003] Current detection-based multi-target tracking methods often use linear modeling with Kalman filters or use deep learning methods to model trajectory prediction, ignoring the influence between trajectories, particularly in dense scenes. Furthermore, when a trajectory is lost for an extended period due to occlusion, existing methods simply treat the predicted trajectory position as the actual position, leading to increasing prediction errors. Consequently, even if the lost trajectory is subsequently successfully detected by the detector, the large discrepancy between the predicted trajectory position and the detected target position prevents successful matching and tracking, resulting in tracking failure. For trajectory feature representation, existing methods often use a single feature storage approach. However, for long-lost trajectories, their features remain unupdated for an extended period, and only a single feature representation is used. This limits the ability to recover the lost trajectory and makes the feature representation less robust to occlusion. Furthermore, storing trajectory features at each moment separately incurs significant overhead.

[0004] Therefore, how to overcome the occlusion and loss of targets during multi-target tracking, and thus better achieve nonlinear motion prediction of trajectories and recovery after loss, is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In light of this, the present invention provides a multi-target tracking method and system based on spatial relationships and long- and short-term trajectory prediction. This method can better address the issues of target occlusion and loss during multi-target tracking, thereby better enabling nonlinear motion prediction and retrieval after loss. This method enhances the robustness of multi-target tracking to occlusion, loss, and complex motion, enabling more accurate tracking of multiple targets.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Multi-target tracking methods based on spatial relationships and long- and short-term trajectory prediction include:

[0008] Get all detected targets and surviving tracks in the current frame image;

[0009] Obtain all trajectories matching the detection target in the previous frame image as active trajectories;

[0010] updating a binding target of each activity track based on a binding score between each activity track and other activity tracks;

[0011] Obtaining a predicted position of each surviving trajectory in a current frame based on the historical position of the surviving trajectory;

[0012] Obtaining a matching cost matrix based on the detected target and the predicted position;

[0013] Matching the surviving trajectories and the detection targets based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories, and unmatched detection targets;

[0014] Obtaining a retrieved trajectory based on the unmatched survival trajectory and its bound target;

[0015] Correcting the position of the low-scoring target frame based on the matched surviving trajectory to obtain a corrected trajectory;

[0016] The position state is updated accordingly based on the matched surviving trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

[0017] Preferably, updating the binding target of each activity track specifically includes:

[0018] Sequentially obtain the binding score between each of the activity tracks and other activity tracks;

[0019] Select the activity track corresponding to the highest binding score greater than the binding threshold, update the binding target of the corresponding activity track, and record the corresponding binding score;

[0020] The binding score acquisition method is:

[0021] Obtaining historical bounding box information for all said activity trajectories;

[0022] Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a set of center point distances and a set of bottom vertical distances are obtained respectively;

[0023] The binding score between the two activity tracks is obtained based on the center point distance set and the bottom vertical distance set.

[0024] Preferably, obtaining the binding score between the two activity tracks based on the center point distance set and the bottom vertical distance set specifically includes:

[0025] Taking average values ​​based on the center point distance set and the bottom vertical distance set, respectively, to obtain a center mean and a bottom mean;

[0026] Obtaining a degree of central point dispersion based on the central mean value;

[0027] Obtaining central point correlation based on the central point dispersion degree and the central mean value;

[0028] Obtaining a bottom correlation of a bounding box based on the central point dispersion degree and the bottom mean value;

[0029] The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

[0030] Preferably, the predicted position acquisition method is:

[0031] Obtaining long-term motion information of a to-be-predicted survival trajectory based on the historical position of the survival trajectory;

[0032] Extracting the influence of each surviving trajectory on the surviving trajectory to be predicted based on the spatial position motion state of all the surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, as comprehensive influence information;

[0033] Obtaining a predicted position of the to-be-predicted survival trajectory in the current frame based on the fusion extraction of the long-term motion information and the comprehensive impact information;

[0034] Repeat the above process to perform position prediction on all the surviving trajectories to obtain the predicted position of each surviving trajectory in the current frame.

[0035] Preferably, the matching cost matrix is ​​obtained by:

[0036] Calculate the IoU distance based on the detected target and the predicted position to obtain a position distance cost matrix;

[0037] Obtaining a feature cost matrix based on a cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory;

[0038] The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

[0039] Preferably, matching the survival trajectory and the detection target based on the matching cost matrix specifically includes:

[0040] Matching the survival trajectory and the detection target using the Hungarian algorithm based on the matching cost matrix;

[0041] The detection targets are divided into high-score detection targets and low-score detection targets based on the confidence scores;

[0042] The matching process adopts a two-stage matching association method:

[0043] In the first stage, all the surviving trajectories are associated and matched with the high-score detection targets to obtain high-score matching trajectories;

[0044] In the second stage, the surviving tracks that were not successfully matched and associated in the first stage are matched and associated with the low-score detection targets to obtain low-score matching tracks;

[0045] The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

[0046] Preferably, obtaining the retrieval track based on the unmatched survival track and its binding target specifically includes:

[0047] Determining whether the bound trajectory of the unmatched survival trajectory successfully matches the detection target based on the unmatched survival trajectory and its bound target;

[0048] If so, the current estimated position of the unmatched survival trajectory is obtained based on the binding relationship between the unmatched survival trajectory and its bound trajectory and the spatial position relationship in the previous frame, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence to obtain the recovered trajectory;

[0049] Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

[0050] Preferably, obtaining the corrected trajectory specifically includes:

[0051] Obtaining a current estimated position of the low-scoring matching track based on the binding relationship between the low-scoring matching track and its bound track and the spatial position relationship in the previous frame, recorded as a second estimated position, and obtaining a confidence of the second estimated position as a second estimated confidence;

[0052] The bounding box of the low-scoring detection target is corrected based on the second estimated position to obtain a corrected bounding box and a corrected confidence, thereby obtaining the corrected trajectory.

[0053] Preferably, the location status update specifically includes:

[0054] Performing a position update on the high-scoring matching trajectory based on the predicted position and the associated position;

[0055] updating the position of the retrieved trajectory based on the first estimated position and the first estimated confidence;

[0056] performing a position update on the corrected trajectory based on the corrected bounding box and the corrected confidence;

[0057] Based on the predicted position corresponding to the lost track as the new position of the lost track;

[0058] updating a feature representation of the matching surviving trajectory based on a confidence score of the detection target matched by the matching surviving trajectory;

[0059] Initialize the unmatched high-score detection target as a new survival trajectory;

[0060] The states of the new survival track and the matching survival track are both set to "active";

[0061] The status based on the retrieved track is set to "retrieved";

[0062] The status based on the lost track is set to "lost".

[0063] A multi-target tracking system based on spatial relationships and long- and short-term trajectory prediction, including: trajectory acquisition module, binding update module, position prediction module, matching output module, trajectory retrieval module, trajectory correction module and position status update module;

[0064] The trajectory acquisition module is used to acquire all detected targets and survival trajectories in the current frame image; and acquire all trajectories matching the detected targets in the previous frame image as active trajectories;

[0065] The binding update module is configured to update the binding target of each activity track based on the binding score between each activity track and other activity tracks;

[0066] The position prediction module is configured to obtain a predicted position of each surviving trajectory in a current frame based on the historical position of the surviving trajectory;

[0067] The matching output module is configured to obtain a matching cost matrix based on the detected target and the predicted position; match the surviving trajectories and the detected targets based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories, and unmatched detected targets;

[0068] The track retrieval module is used to obtain a retrieval track based on the unmatched survival track and its bound target;

[0069] The trajectory correction module is used to correct the position of the low-scoring target frame based on the matched surviving trajectory to obtain a corrected trajectory;

[0070] The position status updating module is configured to update the position status accordingly based on the matched surviving trajectory, the unmatched detected target, the retrieved trajectory, and the corrected trajectory.

[0071] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a multi-target tracking method and system based on spatial relationships and long- and short-term trajectory prediction, which has the following beneficial effects:

[0072] 1. The present invention uses a combination of long-term temporal information and short-term spatial interaction to make predictions, which can better model the nonlinear motion of the tracking target and the influence between trajectories in dense scenes, thereby improving the prediction accuracy of survival trajectories.

[0073] 2. The present invention uses the Hungarian algorithm to perform two-stage matching on high-score detections and low-score detections respectively. The matching cost matrix consists of two parts: the IoU cost matrix representing the motion and the feature cost matrix. This feature distance calculation method with multiple confidence intervals can better improve the stability of feature representation to occlusion, thereby improving the tracker's performance in re-identifying lost tracks.

[0074] 3. For unmatched trajectories, the present invention queries its bound trajectories. If its bound trajectories are not lost, the associated positions are inferred from its bound trajectories. This can effectively retrieve these targets with strong binding correlations when they are lost due to long-term occlusion, thereby alleviating the loss of surviving trajectories caused by high occlusion.

[0075] 4. For the trajectory of a low-scoring detection target, the present invention queries its bound trajectory. If the bound trajectory is not lost, the low-scoring detection frame is adaptively corrected using the position inferred from the bound trajectory. Low-scoring detection frames accompanied by occlusions can lead to inaccurate detection frame positioning. This adaptive correction method can better fit the target's true trajectory.

[0076] 5. The present invention adopts exponential linear averaging to update the features of the corresponding confidence interval for feature representation update, which can avoid storing a large amount of historical feature representations while better coping with the interference caused by occlusion to the feature representation.

[0077] 6. Through the above technical solutions, this method can better deal with the occlusion loss problem and complex motion patterns in multi-target tracking in dense scenes, improve the accuracy of the matching process association, and improve the tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0079] Figure 1 Flowchart of the multi-target tracking method based on spatial relationships and long- and short-term trajectory prediction provided by the present invention.

[0080] Figure 2 This is a flow chart of the binding score acquisition method provided by the present invention.

[0081] Figure 3 This is a schematic diagram of the structures of the long-term motion information extraction branch and the short-term motion impact extraction branch provided by the present invention.

[0082] Figure 4 This is a schematic diagram of the Mamba block structure provided by the present invention.

[0083] Figure 5 This is a schematic diagram of the structure of the multi-target tracking system based on spatial relationships and long- and short-term trajectory prediction provided by the present invention. DETAILED DESCRIPTION

[0084] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0085] Example 1

[0086] like Figure 1 As shown, the embodiment of the present invention discloses a multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction, including:

[0087] Get all detected targets and surviving tracks in the current frame image;

[0088] Get the trajectories of all matching detection targets in the previous frame image as active trajectories;

[0089] Update the binding target of each activity track based on the binding score of each activity track with other activity tracks;

[0090] Get the predicted position of each surviving trajectory in the current frame based on its historical position;

[0091] Obtain a matching cost matrix based on the detected target and predicted position;

[0092] Match the surviving trajectories and the detection targets based on the matching cost matrix to obtain the matched surviving trajectories, unmatched surviving trajectories and unmatched detection targets;

[0093] Get the retrieved trajectory based on the unmatched survival trajectory and its bound target;

[0094] Correct the position of the low-scoring target frame based on the matched surviving trajectory to obtain the corrected trajectory;

[0095] The position status is updated accordingly based on the matching survival trajectory, unmatched detection targets, retrieved trajectory and corrected trajectory.

[0096] Example 2

[0097] The embodiment of the present invention discloses a multi-target tracking method based on spatial relationships and long-term and short-term trajectory prediction, including:

[0098] Get all detected targets and surviving tracks in the current frame image.

[0099] Preferably, all detected targets in the current frame image and their corresponding feature representation vectors are obtained based on the target detector and feature extraction network; this embodiment uses YOLOX as the target detector and ReID as the feature extraction network.

[0100] Preferably, the jth detection target is represented by D j =(b j ,s j ,e j ), where b j represents the bounding box of the detected target j, s j represents the confidence score of the detected target j, e j The feature representation vector representing the detection target j. At the same time, according to the confidence score, the detection targets belonging to the interval [0.1, 0.6] are regarded as low-scoring detection targets, and the detection targets belonging to the interval (0.6, 1] are regarded as high-scoring detection targets.

[0101] Preferably, the surviving trajectory refers to the trajectory left after the update step at time t-1 (i.e., when processing the t-1 frame); during the update step, the trajectory that has lost more than 30 frames will be deleted.

[0102] Get the trajectories of all matching detection targets in the previous frame image as active trajectories.

[0103] Preferably, the active trajectory refers to the trajectory successfully matched and associated with the detection target in the previous frame image, including the trajectory newly initialized in the previous frame.

[0104] The binding target of each active track is updated based on the binding score of each active track with other active tracks.

[0105] Preferably, updating the binding target of each activity track specifically includes:

[0106] Obtain the binding scores between each activity track and other activity tracks in turn;

[0107] Select the activity track corresponding to the highest binding score greater than the binding threshold, update the binding target of the corresponding activity track, and record the corresponding binding score;

[0108] like Figure 2 As shown, the binding score acquisition method is:

[0109] Get the historical bounding box information of all active tracks;

[0110] Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a set of center point distances and a set of bottom vertical distances are obtained respectively;

[0111] The binding score between two activity trajectories is obtained based on the set of distances between the center points and the set of vertical distances between the bottom points.

[0112] Preferably, the binding score between the two activity tracks is obtained based on the center point distance set and the bottom vertical distance set, specifically including:

[0113] Based on the average of the distance set between the center points and the vertical distance set at the bottom, the center mean and bottom mean are obtained respectively;

[0114] The degree of dispersion of the central point is obtained based on the central mean;

[0115] The correlation of the central points is obtained based on the degree of dispersion of the central points and the central mean;

[0116] The bottom correlation of the bounding box is obtained based on the center point dispersion and the bottom mean value;

[0117] The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

[0118] Preferably, only active trajectories are considered when calculating the binding score to avoid interference caused by missing trajectories. B The active trajectory of the frame is not considered.

[0119] Preferably, the historical bounding box information of the trajectory: during the tracking process, for each trajectory, starting from initialization, the bounding box position (ie, coordinates) in each frame is stored and recorded, and the stored historical bounding box position is the historical bounding box information of each trajectory.

[0120] Preferably, each track is a tracked target.

[0121] Preferably, consider the continuous τ of activity trajectory i and activity trajectory j B The historical bounding box information of the frame, the historical bounding box information of the activity track i is The historical bounding box information of activity trajectory j is in represents the bounding box of activity track i in frame r.

[0122] Preferably, the distance between the center points dist is obtained based on the historical bounding box information of the activity track i and the historical bounding box information of the activity track j. center (i,j) and the bottom vertical distance dist bottom (i,j):

[0123]

[0124] Among them, u i and v i Respectively represent the horizontal and vertical coordinates of the center of the bounding box of the activity track i, u j and v j Respectively represent the horizontal and vertical coordinates of the center of the bounding box of the activity trajectory j, y i 2 and y j 2 represents the ordinate of the lower right corner of the bounding box of activity track i and activity track j respectively.

[0125] Based on the distance between the center points dist center (i,j) and the bottom vertical distance dist bottom (i,j) corresponds to the distance set between the center points: And the vertical distance set from the bottom: Based on the average of the distance set between the center points and the vertical distance set at the bottom, the corresponding central mean is obtained. and bottom mean

[0126] Preferably, based on the central mean Get the degree of dispersion of the center point

[0127]

[0128] Where t indicates that the currently processed frame is the tth frame; r indicates the rth frame, and r = t-1,…,t-τ B ;

[0129] Based on the dispersion of the center point and the central mean Get the center point correlation

[0130] Based on the dispersion of the center point and bottom mean Get bounding box bottom correlation

[0131]

[0132] in, represents the average height of the bounding box of activity track i; Represents the average height of the bounding box of activity track j.

[0133] Based on center point correlation Correlation with the bottom of the bounding box Get the binding score σ ij :

[0134]

[0135] Preferably, according to the above calculation steps, the activity trajectory i and the other ones that meet the requirements (i.e., the trajectory length ≥ τ B The binding score between the activity trajectories of the frame) is selected, and the binding score is greater than the binding threshold λ B The largest trajectory in the set is used as the new binding target to update the binding target of active trajectory i and record the binding score. B , then unbind the original binding target.

[0136] The predicted position of each surviving trajectory in the current frame is obtained based on the historical position of the surviving trajectory.

[0137] Preferably, the method for obtaining the predicted position is:

[0138] Obtain the long-term motion information of the survival trajectory to be predicted based on the historical position of the survival trajectory;

[0139] Based on the spatial position motion state of all surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, the influence of each surviving trajectory on the predicted surviving trajectory is extracted as comprehensive influence information;

[0140] The predicted position of the survival trajectory to be predicted in the current frame is obtained based on the fusion extraction of long-term motion information and comprehensive impact information;

[0141] Repeat the above process to predict the positions of all surviving trajectories and obtain the predicted position of each surviving trajectory in the current frame.

[0142] Preferably, taking the prediction of survival trajectory j as an example, first the historical position X∈R of survival trajectory j isL×4 Input the long-term motion information extraction branch to extract its long-term motion information, where L represents the input of the historical position of the most recent L consecutive frames (i.e., tL frames to t-1 frames), and each line of X is the historical bounding box position at the corresponding moment (for trajectories whose historical position length is less than L, the bounding box position of the previous frame is used to fill its historical position to a length of L).

[0143] Preferably, Figure 3 As shown in Figure 2, the long-term motion information extraction branch embeds the input X into a high-dimensional space through a linear transformation, then extracts the temporal motion features through stacked M Mamba blocks, and takes the Mamba block output of the last time step as the long-term motion information:

[0144]

[0145] in, represents the high-dimensional embedding of X, represents the first weight matrix, D1 represents the dimension after embedding, represents the output of the last time step of the Mamba block, i.e. The output of the last dimension is the long-term motion information.

[0146] Preferably, Figure 4 As shown, the calculation process of the Mamba block is as follows:

[0147]

[0148] Where SSM(·) represents the selective state space model, x t represents the input of the Mamba block at the tth time step, y t represents the input and output of the Mamba block at the tth time step, Represents the input of the selective state space model, Conv1D(·) represents one-dimensional convolution, Linear(·) represents linear mapping, σ represents SiLU nonlinear activation function, here The i-th row of corresponds to the input of the Mamba block at the i-th time step.

[0149] Preferably, the short-term motion impact extraction branch is input based on the spatial position motion state of all surviving trajectories in the previous frame to extract the impact of each trajectory on the target trajectory in the short term. The spatial position motion state of the surviving trajectory i is expressed as in and They represent the horizontal and vertical coordinates of the center of the bounding box of the survival trajectory i at the t-1th frame, and They represent the bounding box width and height of the survival trajectory i at the t-1th frame respectively; the last four elements represent its motion state at the t-1th frame; That is, the difference in the corresponding coordinates of the bounding box of the survival trajectory i between the last two frames, The calculation of is the same. The spatial position motion states of all surviving trajectories are spliced ​​together to obtain I∈R N×8 , where N is the current number of trajectories, and the i-th row of I corresponds to the spatial position motion state O of the surviving trajectory i i ;

[0150] Combine I with the spatial position motion state O of the survival trajectory j to be predicted j ∈R 1×8 Input the short-term motion influence extraction branch to obtain the influence of each surviving trajectory on the predicted surviving trajectory j as comprehensive influence information; the calculation process of the short-term motion influence extraction branch is as follows:

[0151] Q=O j W2

[0152] K=IW2

[0153]

[0154] in, and Respectively represent O j and a high-dimensional embedding of I with dimension D2; Indicates that K is expanded row by row using Q; and Denote the first weight matrix and the second weight matrix respectively; where Q, K, and V correspond to the query, key, and value used in the generated attention mechanism respectively;

[0155] Q, K, and V are input into the multi-head attention layer to extract the various interactions between the survival trajectories. The output of the multi-head attention layer is passed through a linear layer to obtain the impact of each survival trajectory on the predicted survival trajectory j, which is used as the comprehensive impact information y2:

[0156]

[0157] Among them, Multi_Head_Attention(·) represents the multi-head attention mechanism, Represents the output of the multi-head attention layer; represents the fourth weight matrix.

[0158] Preferably, based on the long-term motion information y of the survival trajectory j to be predicted L After being fused with the comprehensive influence information y2, it is input into the multi-layer perceptron (MLP) to regress and obtain the predicted position of the survival trajectory j in the current frame.

[0159]

[0160] The predicted position will be used to calculate the IoU cost matrix with the position of the detected target for matching association.

[0161] A matching cost matrix is ​​obtained based on the detected targets and predicted positions.

[0162] Preferably, the matching cost matrix is ​​obtained by:

[0163] Calculate the IoU distance based on the detected target and the predicted position to obtain the position distance cost matrix;

[0164] Based on the cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory, a feature cost matrix is ​​obtained;

[0165] The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

[0166] Preferably, the IoU distance is calculated based on the detected target and the predicted position to obtain the position distance cost matrix A1:

[0167]

[0168] Among them, represents the bounding box position of the jth detection target in the tth frame; Represents the intersection area of ​​two bounding boxes; Represents the area of ​​the union of two bounding boxes; Represents the value of the element in row i and column j of A1; based on all Get the location distance cost matrix A1.

[0169] Preferably, based on the cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory, the feature cost matrix A2 is obtained:

[0170]

[0171] Among them, D j Represents the detection target j; Represents the feature representation of the detection target j; represents the rth feature representation of survival trajectory i, T i represents the survival trajectory i; Represents the element in row i and column j of A2. Based on all Get the location distance cost matrix A2.

[0172] Preferably, when calculating the cosine distance between the feature representation of the detection target and the surviving trajectory, the feature representation of the surviving trajectory takes the confidence score s of the detection target j The corresponding confidence interval feature representation.

[0173] Preferably, the matching cost matrix A is obtained based on the weighted sum of the location distance cost matrix and the feature cost matrix:

[0174] A=α1A1+(1-α1)A2

[0175] Among them, α1 represents a hyperparameter, which is used to weigh the proportion of location distance cost and feature cost.

[0176] The surviving trajectories and the detected targets are matched based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories and unmatched detected targets.

[0177] Preferably, matching the surviving trajectories and the detection targets is performed based on the matching cost matrix, specifically including:

[0178] Based on the matching cost matrix, the Hungarian algorithm is used to match the surviving trajectories and the detected targets by minimizing the cost;

[0179] Detection targets are divided into high-score detection targets and low-score detection targets based on confidence scores;

[0180] The matching process adopts a two-stage matching association method:

[0181] In the first stage, all surviving trajectories are associated and matched with high-scoring detection targets to obtain high-scoring matching trajectories;

[0182] In the second stage, the surviving trajectories that were not successfully matched and associated in the first stage are matched and associated with the low-score detection targets to obtain low-score matching trajectories;

[0183] The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

[0184] Preferably, different matching thresholds λ are set for the first high-score matching association process and the second low-score matching association process. high and λ low , match association is performed for rejections exceeding the threshold.

[0185] The retrieved trajectory is obtained based on the unmatched survival trajectory and its bound target.

[0186] Preferably, obtaining the retrieved track based on the unmatched surviving track and its bound target specifically includes:

[0187] Based on the unmatched survival trajectory and its bound target, determine whether the bound trajectory of the unmatched survival trajectory successfully matches the detection target;

[0188] If so, the current estimated position of the unmatched surviving trajectory is obtained based on the binding relationship between the unmatched surviving trajectory and its bound trajectory and the spatial position relationship in the previous frame, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence to obtain the recovered trajectory;

[0189] Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

[0190] Preferably, for example: unmatched survival trajectory T i , whose binding locus is T j , the binding score is σ ij , T j The detection target of the matching association is in and The survival trajectory T i and its binding trajectory T j Detect the target at the bounding box position at the previous moment The bounding box of It represents the bounding box position of the jth detection target in the tth frame, and the unmatched survival trajectory T is inferred. i At the current position, that is, the first guess position

[0191]

[0192] in, Indicates the unmatched survival trajectory T i The first estimated position of the above four equations are multiplied by a scaling factor ( or The purpose of this is to avoid the scale effect caused by the change of distance from the camera over time.

[0193] Preferably, at this time, since there is no matching survival trajectory T i No matches found, first guess location The confidence level of The binding score between the two is calculated, and the first inference confidence is The calculation formula is as follows:

[0194]

[0195] The low-scoring target frame position is corrected based on the matched surviving trajectory to obtain the corrected trajectory.

[0196] Preferably, obtaining the corrected trajectory specifically includes:

[0197] Based on the binding relationship between the low-scoring matching track and its bound track and the spatial position relationship in the previous frame, the current estimated position of the low-scoring matching track is obtained, recorded as the second estimated position, and the confidence of the second estimated position is obtained as the second estimated confidence;

[0198] Based on the second inferred position, the bounding box of the low-scoring detection target is corrected to obtain a corrected bounding box and a corrected confidence, thereby obtaining a corrected trajectory.

[0199] Preferably, since low-score detection targets are often inaccurately detected due to occlusion, it is considered to use the binding relationship to correct the bounding box of the low-score detection target.

[0200] Preferably, the low score matching trajectory T i With low-score detection targets Match, and infer the current second estimated position of the low-scoring matching track based on the binding relationship and the second guess confidence ( and The calculation process is the same as above and ) The correction process is as follows:

[0201]

[0202] in, Represents the bounding box of the low-score detection target The corrected bounding box after correction; Is the confidence level for low-score detection targets Corrected confidence after correction.

[0203] The position status is updated accordingly based on the matching survival trajectory, unmatched detection targets, retrieved trajectory and corrected trajectory.

[0204] Preferably, the location status update specifically includes:

[0205] Update the position of the high-scoring matching trajectory based on the predicted position and the associated position;

[0206] updating the position of the retrieved trajectory based on the first estimated position and the first estimated confidence;

[0207] Update the position of the corrected trajectory based on the corrected bounding box and the corrected confidence;

[0208] The predicted position corresponding to the lost track is used as the new position of the lost track;

[0209] Update the feature representation of the matching surviving trajectory based on the confidence score of the detection target matched by the matching surviving trajectory;

[0210] Initialize the unmatched high-score detection target as a new survival trajectory;

[0211] The status of both the new survival track and the matching survival track is set to "active";

[0212] The status based on the retrieved track is set to "retrieved";

[0213] The status based on the lost track is set to "lost".

[0214] Preferably, the position of the high-scoring matching trajectory is updated based on the predicted position and the associated position:

[0215]

[0216] in, Indicates the new position after the high-score matching trajectory is updated, is the position of the association, is the confidence of the associated position, is the predicted position.

[0217] Preferably, based on the first estimated position and first guess confidence Update the position of the retrieved track:

[0218]

[0219] in, Indicates the new position after retrieving the updated trajectory.

[0220] Preferably, based on the corrected bounding box and corrected confidence Update the position of the corrected trajectory:

[0221]

[0222] in, Indicates the new position after the corrected trajectory is updated.

[0223] Preferably, based on the predicted position corresponding to the lost trajectory as the new location of the lost track.

[0224] Preferably, the feature representation of the matching surviving trajectory is updated based on the confidence score of the detection target matched by the matching surviving trajectory:

[0225] Matching survival trajectory T i The feature representation is stored as in The corresponding feature representation is within the confidence interval [0.1, 0.5); The corresponding feature representation is within the confidence interval [0.5, 0.9); The corresponding feature representation is within the confidence interval [0.9, 1];

[0226] When updating, the corresponding trajectory feature representation is updated according to the confidence score of the detection target associated with the surviving trajectory. The exponential linear average method is used during the update, such as using the feature representation of the detection target j For survival trajectory i The feature representation is updated and the calculation process is:

[0227]

[0228] Where α2 represents a hyperparameter.

[0229] Preferably, the unmatched high-scoring detection target is initialized as a new surviving track (ie, a new tracking target), and the position of the new surviving track is initialized to the bounding box position of the detection target; the three feature representations of the new surviving track are all initialized to the feature representation of the detection target.

[0230] Preferably, the state of the lost track is set to "lost", and the number of consecutive lost frames is recorded. If the consecutive loss exceeds 30 frames, the track is deleted.

[0231] Preferably, the above steps are repeated for the next frame of image until the tracking is completed.

[0232] Example 3

[0233] like Figure 5 As shown, the multi-target tracking system based on spatial relationship and long-term and short-term trajectory prediction includes: trajectory acquisition module, binding update module, position prediction module, matching output module, trajectory retrieval module, trajectory correction module and position state update module;

[0234] The trajectory acquisition module is used to obtain all detected targets and survival trajectories in the current frame image; and obtain the trajectories of all matching detected targets in the previous frame image as active trajectories;

[0235] A binding update module, configured to update a binding target of each activity track based on a binding score of each activity track with other activity tracks;

[0236] The position prediction module is used to obtain the predicted position of each surviving trajectory in the current frame based on the historical position of the surviving trajectory;

[0237] The matching output module is used to obtain a matching cost matrix based on the detected target and the predicted position; the surviving trajectories and the detected targets are matched based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories, and unmatched detected targets;

[0238] The trajectory retrieval module is used to obtain the retrieval trajectory based on the unmatched survival trajectory and its bound target;

[0239] The trajectory correction module is used to correct the position of the low-scoring target frame based on the matched surviving trajectory to obtain the corrected trajectory;

[0240] The position status update module is used to update the position status based on the matching survival trajectory, unmatched detection targets, retrieved trajectory and corrected trajectory.

[0241] Preferably, the implementation process of each functional module in this embodiment corresponds to the above method one by one, and will not be described in detail here.

[0242] Example 4

[0243] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0244] Memory for storing computer programs;

[0245] The processor, when used to execute the program stored in the memory, can implement the multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction as in embodiment 1 or 2.

[0246] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute the multi-target tracking method based on spatial relationships and long-term and short-term trajectory prediction in Example 1 or 2.

[0247] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0248] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0249] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target tracking method based on spatial relationships and long- and short-term trajectory prediction, characterized by: include: Get all detected targets and surviving tracks in the current frame image; Obtain all trajectories matching the detection target in the previous frame image as active trajectories; updating a binding target of each activity track based on a binding score between each activity track and other activity tracks; The binding score acquisition method is: Obtaining historical bounding box information for all said activity trajectories; Based on the historical bounding box information of each activity track and the historical bounding box information of other activity tracks, a set of center point distances and a set of bottom vertical distances are obtained respectively; Obtaining the binding score between the two activity tracks based on the center point distance set and the bottom vertical distance set; Obtaining a predicted position of each surviving trajectory in a current frame based on the historical position of the surviving trajectory; Obtaining a matching cost matrix based on the detected target and the predicted position; Matching the surviving trajectories and the detection targets based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories, and unmatched detection targets; Obtaining a retrieved trajectory based on the unmatched survival trajectory and its bound target; Correcting the position of the low-scoring target frame based on the matched surviving trajectory to obtain a corrected trajectory; Detection targets are divided into high-score detection targets and low-score detection targets based on confidence scores; The position state is updated accordingly based on the matched surviving trajectory, the unmatched detected target, the retrieved trajectory and the corrected trajectory.

2. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1 is characterized in that: Update the binding target of each active track, including: Sequentially obtain the binding score between each of the activity tracks and other activity tracks; The activity track corresponding to the highest binding score greater than the binding threshold is selected to update the binding target of the corresponding activity track, and the corresponding binding score is recorded at the same time.

3. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 2 is characterized in that: Obtaining the binding score between the two activity tracks based on the center point distance set and the bottom vertical distance set specifically includes: Taking average values ​​based on the center point distance set and the bottom vertical distance set, respectively, to obtain a center mean and a bottom mean; Obtaining a degree of central point dispersion based on the central mean value; Obtaining central point correlation based on the central point dispersion degree and the central mean value; Obtaining a bottom correlation of a bounding box based on the central point dispersion degree and the bottom mean value; The binding score is obtained based on the center point correlation and the bounding box bottom correlation.

4. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1 is characterized in that: The method for obtaining the predicted position is: Obtaining long-term motion information of a to-be-predicted survival trajectory based on the historical position of the survival trajectory; Extracting the influence of each surviving trajectory on the surviving trajectory to be predicted based on the spatial position motion state of all the surviving trajectories in the previous frame and the spatial position motion state of the activity trajectory to be predicted, as comprehensive influence information; Obtaining a predicted position of the to-be-predicted survival trajectory in the current frame based on the fusion extraction of the long-term motion information and the comprehensive impact information; Repeat the above process to perform position prediction on all the surviving trajectories to obtain the predicted position of each surviving trajectory in the current frame.

5. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 1, characterized in that: The matching cost matrix acquisition method is: Calculate the IoU distance based on the detected target and the predicted position to obtain a position distance cost matrix; Obtaining a feature cost matrix based on a cosine distance between the feature representation of the detected target and the feature representation of the surviving trajectory; The matching cost matrix is ​​obtained based on the location distance cost matrix and the feature cost matrix.

6. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 5, characterized in that: Matching the survival trajectory and the detection target based on the matching cost matrix specifically includes: Matching the survival trajectory and the detection target using the Hungarian algorithm based on the matching cost matrix; The matching process adopts a two-stage matching association method: In the first stage, all the surviving trajectories are associated and matched with the high-score detection targets to obtain high-score matching trajectories; In the second stage, the surviving tracks that were not successfully matched and associated in the first stage are matched and associated with the low-score detection targets to obtain low-score matching tracks; The high-score matching trajectory and the low-score matching trajectory together constitute the matching survival trajectory.

7. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 6, characterized in that: The retrieved trajectory is obtained based on the unmatched survival trajectory and its bound target, specifically including: Determining whether the bound trajectory of the unmatched survival trajectory successfully matches the detection target based on the unmatched survival trajectory and its bound target; If so, the current estimated position of the unmatched survival trajectory is obtained based on the binding relationship between the unmatched survival trajectory and its bound trajectory and the spatial position relationship in the previous frame, recorded as the first estimated position, and the confidence of the first estimated position is obtained as the first estimated confidence to obtain the recovered trajectory; Otherwise, the unmatched survival trajectory is marked as a lost trajectory.

8. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 7, characterized in that: The corrected trajectory specifically includes: Obtaining a current estimated position of the low-scoring matching track based on the binding relationship between the low-scoring matching track and its bound track and the spatial position relationship in the previous frame, recorded as a second estimated position, and obtaining a confidence of the second estimated position as a second estimated confidence; The bounding box of the low-score detection target is corrected based on the second estimated position to obtain a corrected bounding box and a corrected confidence, thereby obtaining the corrected trajectory.

9. The multi-target tracking method based on spatial relationship and long-term and short-term trajectory prediction according to claim 8, characterized in that: The location status update specifically includes: Performing a position update on the high-scoring matching trajectory based on the predicted position and the associated position; updating the position of the retrieved trajectory based on the first estimated position and the first estimated confidence; performing a position update on the corrected trajectory based on the corrected bounding box and the corrected confidence; Based on the predicted position corresponding to the lost track as the new position of the lost track; updating a feature representation of the matching surviving trajectory based on a confidence score of the detection target matched by the matching surviving trajectory; Initialize the unmatched high-score detection target as a new survival trajectory; The status of the new survival track and the matching survival track are both set to "active"; The status based on the retrieved track is set to "retrieved"; The status based on the lost track is set to "lost".

10. A multi-target tracking system based on spatial relationships and long-term and short-term trajectory prediction, applied to the multi-target tracking method based on spatial relationships and long-term and short-term trajectory prediction according to any one of claims 1 to 9, characterized in that: Includes: trajectory acquisition module, binding update module, position prediction module, matching output module, trajectory retrieval module, trajectory correction module and position status update module; The trajectory acquisition module is used to obtain all detected targets and survival trajectories in the current frame image; Obtain all trajectories matching the detection target in the previous frame image as active trajectories; The binding update module is configured to update the binding target of each activity track based on the binding score between each activity track and other activity tracks; The position prediction module is configured to obtain a predicted position of each surviving trajectory in a current frame based on the historical position of the surviving trajectory; The matching output module is used to obtain a matching cost matrix based on the detected target and the predicted position; Matching the surviving trajectories and the detection targets based on the matching cost matrix to obtain matched surviving trajectories, unmatched surviving trajectories, and unmatched detection targets; The track retrieval module is used to obtain a retrieval track based on the unmatched survival track and its bound target; The trajectory correction module is used to correct the position of the low-scoring target frame based on the matched surviving trajectory to obtain a corrected trajectory; The position status updating module is configured to update the position status accordingly based on the matched surviving trajectory, the unmatched detected target, the retrieved trajectory, and the corrected trajectory.

Images