Video multi-target tracking method and system based on intra-layer and inter-layer fusion

Abstract

The invention discloses a video multi-target tracking method and system based on intra-layer and inter-layer fusion, and relates to the technical field of video processing and computer vision. The method comprises the following steps: acquiring a video data set containing a multi-target bounding box label, and performing preprocessing to construct a track sequence data set; constructing a Kalman-transformer hybrid frame model, and training by using the trajectory sequence data set to obtain a trained Kalman-transformer hybrid frame model; collecting an input video sequence, and performing frame-by-frame target detection on the input video sequence to obtain a candidate detection set; and inputting the candidate detection set into the trained Kalman-transformer hybrid framework model, and carrying out data association and state estimation to obtain a multi-target trajectory set. According to the method, reasoning is carried out only based on the geometric trajectory on the premise of not extracting image appearance features, real-time performance and precision are both considered, and the problem that an existing Kalman filtering method is difficult to deal with nonlinear motion and long-time shielding is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing and computer vision, and more particularly to a video multi-target tracking method and system based on intra-layer-inter-layer fusion. BACKGROUND

[0002] In the current popular video multi-target tracking method, the tracking and detection paradigm mostly needs to use Kalman filter for motion prediction. However, when actually used in complex scene tracking, due to the nonlinear characteristics of target motion trajectory, it is difficult for most people to obtain accurate motion prediction information in most cases, and the target motion information is lost, resulting in unsatisfactory tracking effect.

[0003] Due to the difficulty of Kalman filter linear motion assumption in describing complex nonlinear trajectories such as sudden stop, sudden turn, curve travel, group interaction, etc., the prediction deviation accumulates. Therefore, during long occlusion or detection loss, only relying on Kalman filter extrapolation, the error gradually enlarges, and identity switching and trajectory rupture are easy to occur. Kalman filter update mainly depends on the latest observation, ignoring long-time historical patterns. Although the introduction of appearance re-identification can alleviate the association problem, the calculation and deployment cost is high, and it is sensitive to light and scale changes. While the transformer has global modeling capability in sequence modeling, but direct use of image-level feature modeling in multi-target tracking often has large overhead and poor real-time performance.

[0004] Therefore, how to improve the tracking robustness in nonlinear motion and occlusion scenes is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a video multi-target tracking method and system based on intra-layer-inter-layer fusion, which can accurately determine the position and motion state of the target when facing complex scene multi-target tracking tasks, and give the confidence of target tracking.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] A video multi-target tracking method based on intra-layer-inter-layer fusion, comprising the following steps:

[0008] Step 1: Obtain a public or self-built video dataset containing multi-target bounding box annotations, and pre-process to build a trajectory sequence dataset;

[0009] Step 2: Construct a Kalman-Transformer hybrid framework model and train the Kalman-Transformer hybrid framework model using a trajectory sequence dataset to obtain a trained Kalman-Transformer hybrid framework model; the Kalman-Transformer hybrid framework model comprises a Kalman filter prediction module, an inner trajectory motion corrector, a cross-trajectory attention module and a pseudo-observation generator connected in sequence, and uses the prediction-update mechanism of the Kalman filter prediction module as a global outer loop; the Kalman filter prediction module is used for linear motion prediction; the inner trajectory motion corrector is used for nonlinear motion residual learning; the cross-trajectory attention module is used for cross-target interaction modeling; and the pseudo-observation generator is used for generating pseudo-observations when detection is missing.

[0010] Step 3: Collect an input video sequence and perform frame-by-frame target detection on the input video sequence to obtain a candidate detection set; the candidate detection set comprises candidate detection boxes and corresponding confidence levels of each frame of image.

[0011] Step 4: input the candidate detection set into the trained Kalman-Transformer hybrid framework model to perform data association and state estimation, and obtain a multi-target trajectory set.

[0012] The technical effect of the above technical solution is that the Kalman-Transformer hybrid framework model uses only geometric trajectory data for inference without extracting image appearance features, and the single-frame complexity is approximately linear with respect to the number of targets, where K is the number of neighbors retained by the cross-trajectory attention module. The method is suitable for automatic driving, video monitoring, motion analysis and crowd behavior understanding scenes, and has advantages in nonlinear motion, frequent occlusion and dense target situations.

[0013] Preferably, the preprocessing in step 1 includes data cleaning, alignment, normalization processing, slicing according to a fixed time window and missing frame masking; a target detector is used to detect targets for each frame of image in the video dataset to generate multi-target detection boxes.

[0014] Preferably, the Kalman filter prediction module performs linear motion prediction based on historical observation states, a state transition matrix and a control input matrix to generate a prior state x KF The adaptive covariance and the prior state are updated during the prediction process; during the model training process, the observation states of each target are derived from the multi-target bounding boxes in the trajectory sequence dataset, and the trajectories are generated based on the observation states; during the inference process, the observation states of each target are derived from the candidate detection boxes, and the trajectories are initialized and generated based on the observation states.

[0015] Preferably, the inner trajectory motion corrector comprises a trajectory embedding layer, a Transformer encoding layer, a residual regression head and a residual fusion layer, and the process of absolute state residual learning comprises:

[0016] Step 221: Derive the observation state of each target according to the multi-target bounding box in the trajectory sequence data set, and construct an absolute state vector with the prior state generated by the Kalman filter prediction module. The trajectory embedding layer linearly maps each step of the absolute state vector of each target within the time window to d dimensions and adds position encoding to obtain a state sequence E of length T t-T：t-1 , t represents the time;

[0017] The absolute state vector includes the center coordinates (c x , c y ), width and height (w, h), and velocity (v x , v y ) of the bounding box;

[0018] Step 222: The transformer encoding layer inputs the state sequence E of each target t-T：t-1 into an L-layer transformer encoder to extract the long-term dependent representation Z of each target Txd ;

[0019] Step 223: The residual regression head performs aggregation and regression to extract the end token or aggregated representation Z of the long-term dependent representation Z of each target last , and regresses it into a nonlinear corrected state x NL through a multilayer perceptron;

[0020] Step 224: The residual fusion layer performs residual fusion on the prior state of each target at time t and the nonlinear corrected state at time t to generate a hybrid prediction F represents the residual fusion operator, which is usually element-wise addition, or can also use gated / weighted fusion.

[0021] Preferably, the cross-trajectory attention module includes a feature embedding layer, a projection layer, an attention calculation layer, a sparse selection layer, a weight aggregation layer, and a state mapping layer. The feature embedding layer performs query / key / value projection on the features of multiple targets in the same frame, uses the top K sparse multi-head attention to retain the most relevant neighbors and aggregate interaction information, and then maps it back to the absolute state space to obtain an interaction-enhanced state for cross-target interaction modeling. The specific process of interaction enhancement is as follows:

[0022] Step 231: The feature embedding layer embeds the hybrid prediction x hybrid of each target into a feature representation in the same frame;

[0023] Step 232: The projection layer projects the feature representation of each target into a query vector, a key vector, and a value vector;

[0024] Step 233: The attention calculation layer calculates the attention score between any two trajectories in the same frame according to the query vector, the key vector, and the value vector;

[0025] Step 234: The sparse selection layer sorts the attention scores from high to low, selects the top K trajectories in the same frame as neighbor trajectories for each trajectory, and forms a neighbor set, and the remaining trajectories in the same frame are set to negative infinity to achieve sparseness;

[0026] Step 235: The weight aggregation layer performs normalization and weighted summation on the value vector of the neighbor set to obtain the interaction aggregation representation H of the trajectory t ;

[0027] Step 236: The state mapping layer maps the interaction aggregation representation H back to the absolute state space through the shared multi-layer perception to output the interaction enhanced state t ; The sparsity of attention improves the stability and efficiency of interaction modeling in crowded scenes; the inner trajectory motion corrector and the cross-trajectory attention module enhance the cross-target attention of multiple targets in the same frame, constrain the interaction consistency, and stabilize the interaction modeling by the consistency between the interaction consistency loss and the supervision signal (such as the true value state or the fusion output) during training.

[0028] Preferably, the pseudo-observation generator is activated to generate a pseudo-observation according to a pseudo-observation generation strategy; the pseudo-observation generation strategy is: when the bounding box is missing, occluded, or the confidence is lower than the association threshold, the interaction enhanced state is taken as the pseudo-observation The pseudo-observation is input into the Kalman filter prediction module for prediction update to reduce drift and error accumulation under long-time occlusion, and the adaptive covariance is inflated to reflect the increase of uncertainty when the pseudo-observation is used continuously. When the bounding box is recovered and the confidence is equal to or higher than the association threshold, the pseudo-observation generator is stopped to generate the pseudo-observation, and the real observation state is used for Kalman update in the Kalman filter prediction module; the specific process includes:

[0029] Step 241: the interaction enhanced state of the current frame is taken as the pseudo-observation and input into the Kalman filter prediction module;

[0030] Step 242: the adaptive covariance P of the Kalman filter prediction module is inflated during continuous use of the pseudo-observation; wherein P t is inflated to P t ; t t (γ>1) or P t ←P t +βI, γ is the inflation coefficient, β is a positive scalar parameter, and I is the identity matrix. In the inference process, the detection recovery criterion is that there is an effective match through the gate and the confidence of the candidate bounding box exceeds the association threshold, and the continuous successful matching count can be combined to avoid jitter switching.​

[0031] Preferably, the observation state, the prior state and the pseudo-observation are represented by an absolute bounding box state, including the center coordinates, the area, the aspect ratio and the velocity components, respectively. According to the implementation, the optional subset is selected.

[0032] Preferably, the Kalman filter prediction module adopts constant velocity state transition and linear observation, and the process noise and the observation noise are Q and R, respectively.

[0033] Preferably, the process of training the Kalman-Transformer hybrid framework model in step 2 is as follows:

[0034] Step 21: Pre-training the inner trajectory motion corrector, inputting the absolute state vector into the inner trajectory motion corrector, learning the absolute state residual, obtaining the hybrid prediction, and learning to capture complex nonlinear motion patterns from the history by minimizing the inner trajectory loss between the hybrid prediction and the state offset; the inner trajectory loss function L inner is expressed as x t represents the observation state at time t; x hybrid,t represents the hybrid prediction at time t; represents the 2-norm operation;

[0035] Step 22: Joint fine-tuning combined with the cross-track attention module, refining the individual motion prediction and group interaction modeling capability by jointly optimizing the inner trajectory loss and the interaction consistency loss of the cross-track attention module; the interaction consistency loss function L inter is expressed as CrossAttn(x inner ) represents the output state of the cross-track attention module, x inner represents the absolute state vector;

[0036] Step 23: According to the pseudo-observation generation strategy, using the interaction enhanced state output by the cross-track attention module as the pseudo-observation to update the Kalman filter prediction module; calculating the total loss by weighted sum of the inner trajectory loss, the interaction consistency loss and the fusion output loss calculated according to the final output multi-target trajectory set, realizing end-to-end optimization by using the total loss, coordinating motion prediction, interaction modeling and noise processing, and obtaining the trained Kalman-Transformer hybrid framework model; the fusion output loss function L hybrid is expressed as x final represents the target trajectory output by the Kalman-Transformer hybrid framework model.

[0037] Preferably, the process of data association and state estimation of the Kalman-Transformer hybrid framework model in step 4 is as follows:

[0038] Step 41: Perform linear motion prediction on all initialized trajectories or all trajectories updated by Kalman filter prediction module in previous time to obtain prior state; the trajectory is an on-track trajectory, i.e., a trajectory that is still tracked and not terminated;

[0039] Step 42: Obtain a non-linear correction state by the inner trajectory motion corrector according to a known state sequence in the last T frames, and perform residual fusion on the prior state to obtain a hybrid prediction; the known state sequence is obtained by the Kalman filter prediction module after updating in each frame;

[0040] Step 43: Calculate the attention score between trajectories according to the hybrid prediction of all trajectories by the cross-trajectory attention module, and output an interaction-enhanced state;

[0041] Step 44: If there is a missing detection box in the current frame image of the input video sequence or the confidence of the candidate detection box is lower than the correlation threshold, generate a pseudo observation by the pseudo observation generator according to the interaction-enhanced state as the input of the Kalman filter prediction module to obtain the prior state as the updated trajectory, and return to step 41 until all frames of the input video sequence are traversed;

[0042] Step 45: Construct a combined cost matrix of trajectories and candidate detection boxes and perform gating screening, and perform target association and assignment according to the screened combined cost matrix using the Hungarian algorithm, update the matched trajectory, and manage the unmatched trajectory and detection;

[0043] Step 46: Output the updated and managed trajectory to obtain a multi-target trajectory set.

[0044] Preferably, in step 45, the target association uses a combined cost matrix of Mahalanobis distance and 1-IoU, and combines elliptical gating and confidence gating to screen out unreasonable pairs, and the target assignment uses the Hungarian algorithm to complete global matching for minimizing the total cost.

[0045] Preferably, the specific process of step 45 is as follows:

[0046] Step 451: Calculate the motion consistency cost and the geometric consistency cost for each combination of trajectories and candidate detection boxes, and construct a combined cost matrix;

[0047] Obtain the prior covariance P from the Kalman filter prediction module t|t-1 , the observation matrix H, the observation noise R, and the innovation covariance Combine the prior state x of the trajectory t|t-1 and the candidate detection box z to calculate the motion consistency cost d according to the Mahalanobis distance used to measure the distance between the predicted observation and the true observation; project the state space into the observation space according to the innovation covariance, and construct the observation matrix according to the prior state and the observation state;

[0048] The IoU is calculated according to the prior state output by the Kalman filter prediction module and the area of the candidate detection box, and the geometric cost is 1-IoU, where IoU is the intersection over union of the predicted box and the detection box, and the smaller the geometric cost is, the higher the position matching degree is;

[0049] The combined cost C = w m d M +w g (1-IoU) is calculated, where w m is the motion consistency weight, w g is the geometric consistency weight, and the matching degree of the trajectory and the detection is comprehensively evaluated;

[0050] Step 452: The unreasonable combinations are screened out by using the elliptical gating and confidence gating; if where d M is the motion consistency cost, i.e., the Mahalanobis distance, and τ is the elliptical gating threshold, or the confidence of the candidate detection box is lower than the correlation threshold, then the corresponding matrix element in the combined cost matrix is set to be invalid or a maximum value to be excluded from the matching;

[0051] Step 453: The Hungarian algorithm is applied to the gated combined cost matrix to obtain the matching relationship of the global minimum total cost, and the associated trajectory and candidate detection box combination is determined;

[0052] Step 454: For the trajectory successfully associated, the candidate detection box z is input as the observed state to the Kalman filter prediction module to perform Kalman update, correct the prior state and update the adaptive covariance P t , and reset the continuous loss count of the trajectory to 0; for the trajectory that is not successfully associated, update the continuous loss count miss←miss+1, when miss exceeds the termination threshold, it is judged that the target corresponding to the trajectory has left the video area or is continuously occluded, and the trajectory is terminated, when the termination threshold is not exceeded, the pseudo-observation generator is activated according to the pseudo-observation generation strategy to interact with the enhanced state as the pseudo-observation maintain the trajectory update; for the candidate detection box that is not successfully associated and whose confidence is higher than the threshold of new generation, it is considered that a new target appears, and a new trajectory is initialized according to the center coordinates of the candidate detection box and the prior state and adaptive covariance are initialized.

[0053] Preferably, the total loss function L used for training the Kalman-transformer hybrid framework model is represented as:

[0054] L = λ1·L inner + λ2·L inter + λ3·L hybrid

[0055] where L inner represents the inner trajectory loss function of the inner trajectory motion corrector, x hybrid,t denotes the mixed prediction at time t, x t denotes the observed state at time t; L inter denotes the interaction consistency loss function of the cross-track attention module, CrossAttn(x inner ) denotes the output state of the cross-track attention module; L hybrid denotes the fusion output loss, x final denotes the target trajectory output by the Kalman-Transformer hybrid framework model; λ1, λ2, λ3 represent the loss weights respectively.

[0056] Preferably, the training strategy of the Kalman-Transformer hybrid framework model includes two stages:

[0057] Pre-training stage: freeze the cross-track attention module, and only train the intra-track motion corrector to minimize the intra-track loss;

[0058] Joint fine-tuning stage: introduce the cross-track attention module and jointly optimize the interaction consistency loss and the fusion output loss, and the total loss is the weighted sum of each loss term.

[0059] A video multi-target tracking system based on intra-layer-inter-layer fusion, comprising a video acquisition module, a target detection module, a model prediction module, a data association and track management module;

[0060] The video acquisition module acquires video data sets to construct track sequence data sets and acquires input video sequences;

[0061] The target detection module performs frame-by-frame target detection on the input video sequence to obtain a candidate detection set;

[0062] The model prediction module deploys the trained Kalman-Transformer hybrid framework model and performs track prediction according to the candidate detection combination;

[0063] The data association and track management module performs data association and state estimation according to the predicted track to obtain a multi-target track set.

[0064] Compared with the prior art, the video multi-target tracking method and system based on inner layer-interlayer fusion provided by the technical solution disclosed by the present application introduce a transformer (Transformer) deep motion model into a traditional Kalman filter framework to form a Kalman-Transformer hybrid tracking framework (KalmanFormer): first, efficient linear motion prediction is completed by using a Kalman filter; then, an inner trajectory motion corrector learns absolute state residuals (nonlinear correction states) from historical trajectories and adaptively corrects linear prediction; a cross-trajectory attention module is used to model the space-time interaction between multiple targets to improve the correlation robustness in occlusion scenarios; and a pseudo-observation generator generates substitute observations to avoid error accumulation when observations are missing. Without extracting image appearance features, the present application only performs reasoning based on geometric trajectories, and both real-time performance and accuracy are taken into account, thereby solving the problem that the existing Kalman filter type methods are difficult to deal with nonlinear motion and long-time occlusion. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0066] Figure 1 The video multi-target tracking method based on inner layer-interlayer fusion provided by the present application is shown in the schematic diagram.

[0067] Figure 2 The Kalman-Transformer hybrid framework model structure provided by the present application is shown in the schematic diagram.

[0068] Figure 3 The cross-trajectory attention module structure provided by the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0070] The embodiments of the present application disclose a video multi-target tracking method based on inner layer-interlayer fusion, which realizes multi-target tracking of video through deep motion modeling and cross-target interaction modeling, as shown in Figure 1 The method comprises the following steps:

[0071] S1: Obtain a public or self-built video dataset containing multi-target bounding box annotations, and preprocess to construct a trajectory sequence dataset;

[0072] S2: Construct a Kalman-Transformer hybrid framework model (KalmanFormer), and train it using the trajectory sequence dataset to obtain a trained Kalman-Transformer hybrid framework model; the Kalman-Transformer hybrid framework model includes a Kalman filter prediction module, an internal trajectory motion corrector, a cross-trajectory attention module, and a pseudo-observation generator connected in sequence, and uses the prediction-update mechanism of the Kalman filter prediction module as a global outer loop; the Kalman filter prediction module is used for linear motion prediction; the internal trajectory motion corrector is used for nonlinear motion residual learning; the cross-trajectory attention module is used for cross-target interaction modeling; and the pseudo-observation generator is used for generating pseudo-observations when detection is missing;

[0073] S3: Collect an input video sequence and perform frame-by-frame target detection on the input video sequence to obtain a candidate detection set; the candidate detection set includes candidate detection boxes and corresponding confidence scores for each frame of image;

[0074] S4: Input the candidate detection set into the trained Kalman-Transformer hybrid framework model to perform data association and state estimation, and obtain a multi-target trajectory set.

[0075] In one specific embodiment, the following steps are specifically included:

[0076] S1: Collect a video to be processed, establish a multi-target trajectory dataset, and perform normalization, time window slicing, and missing frame mask preprocessing on the trajectories to construct a trajectory sequence dataset;

[0077] S2: Input the trajectory sequence in the trajectory sequence dataset into the Kalman filter prediction module in the Kalman-Transformer hybrid framework model for trajectory prediction; use the internal trajectory motion corrector for modeling to extract long-term nonlinear motion patterns; use the cross-trajectory attention module for cross-target interaction modeling; and use the pseudo-observation generator for generating pseudo-observations when detection is missing; use the prediction-update mechanism of the Kalman filter prediction module as a global outer loop to optimize the training of the Kalman-Transformer hybrid framework model;

[0078] S3: Initialize the outer loop parameters and covariance of the Kalman filter prediction module, and set the gating threshold and matching weight;

[0079] S4: Perform target detection on each frame of image in the video to be processed to obtain a candidate detection set;

[0080] S5: Perform Kalman filter prior prediction on the on-track targets through the Kalman filter prediction module to obtain prior states and adaptive covariances;

[0081] S6: input the historical time window step state into the inner trajectory motion corrector to obtain a nonlinear correction state, and calculate a hybrid prediction;

[0082] S7: input the hybrid prediction of all targets into a cross-track attention module to calculate a cross-target front K attention, obtain an interaction enhanced representation, and map it to an interaction state;

[0083] S8: if a target has no reliable detection, set a pseudo observation to participate in Kalman filter prediction module update through a pseudo observation generator;

[0084] S9: construct a combined cost matrix for the candidate trajectory and the detection frame pair and perform gated screening;

[0085] S10: according to the combined cost matrix after screening, complete the association matching of the trajectory and the detection frame by using the Hungarian algorithm, update the state and covariance of the matched trajectory, and accumulate the consecutive loss count of the unmatched trajectory;

[0086] S11: initialize a new trajectory for a high-confidence unmatched detection frame; terminate the trajectory whose consecutive loss count exceeds a threshold, and maintain the trajectory through the pseudo observation generator for the trajectory whose consecutive loss count does not exceed the threshold;

[0087] S12: output the current multi-target trajectory set and statistical information.

[0088] Further, in online inference, in S6, absolute state residual learning of the inner trajectory motion corrector is performed, which includes the following steps:

[0089] S61: trajectory embedding, linearly mapping each step state to d dimensions and adding position encoding to obtain sequence E t-T：t-1 The mapped state includes the first k-1 step observation state of all targets derived from the candidate detection frame in the candidate detection set, and the kth step prior state predicted by the Kalman filter prediction module according to the first k-1 step observation state;

[0090] S62: transformer encoding, inputting E t-T：t-1 into an L-layer transformer encoder to extract long-time dependency representation

[0091] S63: aggregation and regression, taking Z t terminal token or aggregated representation into a multi-layer perception (MLP) to regress an absolute state residual nonlinear correction state

[0092] S64: residual fusion, performing residual fusion on the prior state of each target at time t and the nonlinear correction state at time t to generate a hybrid prediction F represents a residual fusion operator, commonly element-wise addition, or gated / weighted fusion.

[0093] Further, in S7, cross-track attention module interaction enhancement is performed, specifically including the following steps:

[0094] S71: Feature preparation, embedding mixed prediction of each target into feature representation;

[0095] S72: Query / key / value projection, projecting the feature representation into query vector, key vector, and value vector;

[0096] S73: Attention score calculation, calculating the attention score between the trajectories corresponding to any two targets according to the query vector, key vector, and value vector;

[0097] S74: Top-K sparse selection, selecting the top K neighbors of each trajectory according to the attention score, and setting the rest to negative infinity;

[0098] S75: Weight aggregation, performing normalization and weighted summation on the remaining neighbors;

[0099] S76: State mapping, mapping the aggregation result through a shared multi-layer perception machine to form an interaction-enhanced state;

[0100] S77: Training constraint, taking the interaction consistency loss as the training target.

[0101] Further, in S8, pseudo-observation generation is performed, specifically including the following steps:

[0102] S81: Trigger condition, activating the pseudo-observation generator when missing, occlusion, or confidence is below the threshold;

[0103] S82: Pseudo-observation generation, taking the interaction-enhanced state as the pseudo-observation, inputting it into the Kalman filter prediction module for prediction update;

[0104] S83: Uncertainty management, performing inflation on the adaptive covariance of the Kalman filter prediction module when continuous pseudo-observation; wherein the adaptive covariance P t Inflation can be implemented as P t ←γP t (γ>1) or P t ←P t +βI, γ is the inflation coefficient, β is a positive scalar parameter, and I is the identity matrix;

[0105] S84: Termination condition, stopping the pseudo-observation when the detection is restored and the confidence is equal to or higher than the threshold, and restoring the normal Kalman filter prediction module update.

[0106] Further, S9-S11 perform gating and data association, specifically including the following steps:

[0107] S9: Combination cost gating, calculate Mahalanobis distance and 1-IoU of each pair of track and candidate detection box, and perform weighted sum to calculate combination cost, if the combination cost or Mahalanobis distance exceeds the gating threshold, the corresponding pair of track and candidate detection box is removed; the combination cost C = w m ·d M +w i ·(1-IoU);

[0108] S10: According to the combination cost, use the Hungarian algorithm to solve the minimum total cost matching track and detection box; the combination of the track and the candidate detection box corresponding to the minimum total cost is the associated matching combination;

[0109] State update, the matched track is fed back to the Kalman filter prediction module to perform Kalman filter (KF) update and adaptive covariance update, to correct the prior state and adaptive covariance;

[0110] Unmatched track processing, let the continuous loss count miss of the unmatched track be +1 and maintain the track by the pseudo observation generator (POG) combined with the pseudo observation generation strategy;

[0111] S11: New initialization, initialize a new track for the high-confidence unmatched candidate detection box.

[0112] Further, the training process in S2 specifically includes the following steps:

[0113] S21: Phase one, only train the inner track motion corrector, minimize the inner track loss function L inner , x hybrid,t represents the mixed prediction at time t, x t represents the observation state at time t;

[0114] S22: Phase two, introduce a cross-track attention module, minimize the total loss function L = λ1·L inner +λ2·L inter +λ3·L hybrid , wherein the fusion output loss x final represents the target track output by the Kalman-transformer hybrid framework model; L inter represents the interaction consistency loss function of the cross-track attention module, CrossAttn(x inner ) represents the output state of the cross-track attention module, x t represents the observation state at time t; λ1, λ2, λ3 represent the loss weights, respectively;

[0115] On the other hand, in one specific embodiment, a video multi-target tracking method based on intra-layer inter-layer fusion is used to accurately determine the position and motion state of the target when facing a complex scene multi-target tracking task, and the confidence of the target tracking is given. The multi-target tracking method is described in detail as follows:

[0116] Adjust the image size of the data set, slice the original video sequence according to a fixed time window T, and slice the trajectory data set according to a time window T = 10 or 20. As shown in Figure 2 The Kalman-transformer hybrid framework model includes a Kalman filter, an intra-track motion corrector, a cross-track attention module, and a pseudo-observation generator; based on the Kalman filter prediction module, linear motion prediction is extracted to obtain a linear prediction state; based on the intra-track motion corrector, a non-linear correction state is obtained, and a hybrid prediction is calculated; based on the cross-track attention module, cross-target spatio-temporal interaction modeling is performed to obtain an interaction enhanced state; when the target is missing or unreliable, the pseudo-observation generator generates a pseudo-observation continuous track to obtain a hybrid prediction state, and outputs the final multi-target track set. Due to the complex correlation between the extracted multiple states, in order to more effectively fuse multi-scale features and further more effectively express motion content, a tracking method based on deep motion modeling and cross-target interaction modeling is adopted.

[0117] As shown in Figure 2 The intra-track motion corrector is: inputting the historical time window step state sequence into the transformer encoder to extract long-time dependence representation, and regressing into a non-linear correction state through a multi-layer perception, and adding the linear prediction to obtain a hybrid prediction state; inputting the hybrid prediction state into the cross-track attention module to obtain an interaction enhanced state; processing the detection missing condition through the pseudo-observation generator, generating a pseudo-observation according to the interaction enhanced state to participate in Kalman filter update; evaluating the obtained prediction result through an improved loss function, and training the model.

[0118] The cross-track attention module is: inputting the hybrid prediction state into the cross-track attention module, and obtaining query, key, and value vectors through query / key / value projection respectively; then calculating attention scores, adopting Top-K sparse selection to reserve the most relevant neighbors, performing normalization and weighted summation on the reserved neighbors, mapping back to the absolute state space through a shared multi-layer perception to form an interaction enhanced state; then calculating the loss values of the intra-track loss, the interaction consistency loss, and the fusion output loss to determine the position and motion state of the target, and generating the final tracking result.

[0119] Further, the evaluation model training calculation process based on the improved total loss function is as follows:

[0120] L = λ1·L inner + λ2·L inter+ λ3·L hybri d

[0121] wherein, L inner represents the inner trajectory motion corrector inner trajectory loss function,

[0122] x hybrid,t represents the mixed prediction at time t, x t represents the observation state at time t; L inter represents the interaction consistency loss function of the cross trajectory attention module, CrossAttn(x inner ) represents the output state of the cross trajectory attention module; L hybrid represents the fusion output loss, x final represents the multi-target trajectory state finally output by the Kalman-transformer hybrid framework model, λ1, λ2, λ3 respectively represent the loss weight; in the above formula, λ1=1, λ2=1, λ3=1.

[0123] Deep motion modeling is one of the important methods to improve the performance of multi-target tracking. In the multi-target tracking method based on deep motion modeling, if only the tracking module based on linear motion hypothesis is used, there is a large gap between the motion states at different times, the learned motion representation is relatively rough, which seriously affects the performance of multi-target tracking.

[0124] By using the inner trajectory motion corrector in this embodiment, a more fine nonlinear motion representation can be extracted, and more accurate motion information can be retained. In the face of the obtained mixed prediction state, if the measurement method based on Mahalanobis distance is adopted, it is very sensitive to the change of nonlinear motion, which further affects the performance of multi-target tracking. In this embodiment, the method based on deep motion modeling and cross-target interaction modeling is used to replace Mahalanobis distance, so that the tracking of nonlinear motion is more stable. This method not only solves the problem that the traditional linear motion hypothesis cannot learn the fine representation of nonlinear motion, but also improves the unstable tracking of nonlinear motion caused by the measurement method based on Mahalanobis distance.

[0125] On the other hand, in one embodiment of the video multi-target tracking system based on inner-layer inter-layer fusion, it includes a video acquisition module, a target detection module, a model prediction module, a data association and a track management module;

[0126] The video acquisition module acquires the video dataset to construct the track sequence dataset, and acquires the input video sequence;

[0127] The target detection module performs frame-by-frame target detection on the input video sequence to obtain a candidate detection set;

[0128] A model prediction module is configured to deploy the trained Kalman-Transformer hybrid framework model to perform trajectory prediction according to the candidate detections combination;

[0129] A data association and trajectory management module is configured to perform data association and state estimation according to the predicted trajectories to obtain a multi-target trajectory set.

[0130] The various embodiments described in the specification are progressive in nature, and each embodiment highlights the differences from other embodiments. The same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method.

[0131] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video multi-target tracking method based on intra-layer and inter-layer fusion, characterized in that, Includes the following steps: Step 1: Obtain a video dataset containing multi-object bounding box annotations and preprocess it to construct a trajectory sequence dataset; Step 2: Construct a Kalman-transformer hybrid framework model and train it using a trajectory sequence dataset to obtain a trained Kalman-transformer hybrid framework model. The Kalman-transformer hybrid framework model includes a Kalman filter prediction module, an inner trajectory motion corrector, a cross-trajectory attention module, and a pseudo-observation generator connected in sequence, and uses the prediction-update mechanism of the Kalman filter prediction module as the global outer loop. Step 3: Acquire the input video sequence and perform frame-by-frame target detection on the input video sequence to obtain a candidate detection set; Step 4: Input the candidate detection set into the trained Kalman-transformer hybrid framework model to perform data association and state estimation, and obtain a multi-target trajectory set.

2. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 1, characterized in that, The Kalman filter prediction module performs linear motion prediction based on historical observations, state transition matrix, and control input matrix to generate prior states. During the prediction process, it updates the adaptive covariance and prior states. During model training, it derives the observation states of each target based on the multi-target bounding boxes in the trajectory sequence dataset and generates trajectories based on the observation states. During inference, it derives the observation states of each target based on candidate detection boxes and initializes and generates trajectories based on the observation states.

3. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 2, characterized in that, The process of absolute state residual learning by the internal trajectory motion corrector includes: Step 221: Construct absolute state vectors based on the observed states of the trajectory sequence dataset and the prior states generated by the Kalman filter prediction module. Linearly map the absolute state vectors of each target within the time window T to d dimensions and add position encoding to obtain the state sequence E. t-T：t-1 t represents time. Step 222: Convert the state sequence E of each target t-T：t-1 Input an L-layer transformer encoder to extract the long-term dependency representation Z∈R for each target. Txd ; Step 223: Extract the terminal tokens or aggregate representations Z of the long-term dependency representation Z of each target. last After regression output by a multilayer perceptron, the nonlinear correction state x is obtained. NL ; Step 224: Prior state of each target at time t and the nonlinear correction state at time t Perform residual fusion to generate hybrid predictions F represents the residual fusion operator.

4. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 3, characterized in that, The process of cross-target interaction modeling by the cross-trajectory attention module is as follows: Step 231: Embed the hybrid predictions of each target as a feature representation; Step 232: Project the feature representations of each target into query vectors, key vectors, and value vectors; Step 233: Calculate the attention score between the trajectories corresponding to any two targets in the same frame image based on the query vector, key vector, and value vector of each target; Step 234: Sort the attention scores from high to low, select the top K trajectories in the same frame for each trajectory as neighbor trajectories, and form a neighbor set. Set the trajectories in the remaining same frame to negative infinity. Step 235: Normalize and weight the value vectors corresponding to the trajectories in the neighbor set to obtain the interactive aggregated representation of the trajectories; Step 236: The interaction aggregation representation is mapped back to the absolute state space via a shared multilayer perceptron, forming an interaction-enhanced state.

5. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 4, characterized in that, The pseudo-observation generator is activated according to the pseudo-observation generation strategy, which is as follows: when there are missing or occluded bounding boxes or detection boxes, or when the confidence level is lower than the association threshold, the interaction enhancement state is activated. As pseudo-observations, pseudo-observations are used as the observation state input to the Kalman filter prediction module for prediction updates. When pseudo-observations are used continuously, adaptive covariance is expanded. When the detection box and bounding box are recovered and the confidence level is higher than or equal to the association threshold, the pseudo-observation generator stops generating pseudo-observations.

6. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 5, characterized in that, The training process for the Kalman-transformer hybrid framework model in step 2 is as follows: Step 21: Pre-train and optimize the inner trajectory motion corrector by minimizing the inner trajectory loss between the mixed prediction and the state offset; Step 22: Perform joint fine-tuning by combining the cross-track attention module. Optimize the loss by jointly optimizing the intra-track loss and the interaction consistency loss of the cross-track attention module. Step 23: Based on the pseudo-observation generation strategy, use the interactive enhancement state output by the cross-trajectory attention module as a pseudo-observation to update the Kalman filter prediction module; The total loss is calculated by weighting and summing the internal trajectory loss, interaction consistency loss, and fusion output loss calculated based on the multi-objective trajectory set output by the model. End-to-end optimization is then performed using the total loss to obtain the trained Kalman-transformer hybrid framework model.

7. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 5, characterized in that, The process of data association and state estimation for the Kalman-transformer hybrid framework model in step 4 is as follows: Step 41: Perform Kalman filter linear prediction on all candidate detection boxes corresponding to the initialized trajectories or all trajectories updated by the Kalman filter prediction module in the previous time step to obtain the prior state; Step 42: The internal trajectory motion corrector obtains the nonlinear correction state based on the known state sequence within the most recent T frames, and performs residual fusion with the prior state to obtain the hybrid prediction; Step 43: The cross-trajectory attention module calculates the attention score between each trajectory based on the mixed prediction of all trajectories and outputs the interaction enhancement state; Step 44: If the current frame image of the input video sequence has a missing detection box or the corresponding confidence is lower than the association threshold, the pseudo-observation generator generates pseudo-observations based on the interaction enhancement state, which are used as input to the Kalman filter prediction module to obtain the prior state as the updated trajectory, and then returns to step 41 until all frames of the input video sequence have been traversed. Step 45: Construct the combined cost matrix of trajectory and candidate detection boxes and perform gating screening. Based on the selected combined cost matrix, use the Hungarian algorithm to associate and assign targets, update matching trajectories and manage unmatched trajectories and detections. Step 46: Output the updated and managed trajectories to obtain a set of multi-target trajectories.

8. The video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 7, characterized in that, The specific process of step 45 is as follows: Step 451: Calculate the motion consistency cost and geometric consistency cost for each combination of trajectory and candidate detection box, and construct a combined cost matrix based on the two costs; Step 452: Use elliptical gating and confidence gating to filter out unreasonable pairings of trajectories and candidate detection boxes, and adjust the combination cost matrix according to the filtering results; Step 453: Use the Hungarian algorithm to optimize the adjusted combined cost matrix, obtain the optimal combined cost matrix, and determine the combination of associated trajectories and candidate detection boxes; Step 454: Feed back the candidate detection boxes in the associated trajectory and detection combination as observation states to the Kalman filter prediction module for prediction update, correct the state of the corresponding trajectory and update its adaptive covariance, and reset the consecutive loss count of the corresponding trajectory to zero; increment the consecutive loss count of unassociated trajectories by 1; if the current consecutive loss count exceeds the preset termination threshold, terminate the trajectory; if the current consecutive loss count does not exceed the termination threshold Nmiss, activate the pseudo-observation generator according to the pseudo-observation generation strategy to generate pseudo-observation maintenance trajectories; for unassociated candidate detection boxes with a confidence greater than the new threshold, create a corresponding new trajectory and initialize its state.

9. A video multi-target tracking method based on intra-layer and inter-layer fusion according to claim 6, characterized in that, The total loss function L used to train the Kalman-transformer hybrid framework model is expressed as: L=λ1·L inner +λ2·L inter +λ3·L hybrid Among them, L inner This represents the inner trajectory loss function of the inner trajectory motion corrector. x hybrid,t Let x represent the mixed prediction at time t. t L represents the observed state at time t; inter This represents the interaction consistency loss function for cross-trajectory attention modules. CrossAttn(x inner ) indicates the output state of the cross-track attention module; L hybrid Indicates the fusion output loss. x final λ1 represents the target trajectory output by the Kalman-transformer hybrid framework model; λ2, λ3 represent the loss weights, respectively.

10. A video multi-target tracking system based on inner-layer-inter-layer fusion, characterized in that, The video multi-target tracking method based on inner-layer-inter-layer fusion according to any one of claims 1-9 includes a video acquisition module, a target detection module, a model prediction module, and a data association and trajectory management module; The video acquisition module collects video datasets to construct trajectory sequence datasets and collects input video sequences. The target detection module performs frame-by-frame target detection on the input video sequence to obtain a candidate detection set; The model prediction module deploys a pre-trained Kalman-transformer hybrid framework model and performs trajectory prediction based on candidate detections. The data association and trajectory management module performs data association and state estimation based on the predicted trajectory to obtain a set of multi-target trajectories.

Images