Multi-target tracking method combining time convolutional network and self-attention mechanism

Abstract

The invention relates to the technical field of computer vision, in particular to a multi-target tracking method combining a time convolution network and a self-attention mechanism, which comprises a motion model based on a TCN, dynamic ReID feature updating based on track confidence, and a rematching strategy used for correcting wrong association caused by target interaction shielding. A TCN network is combined with an attention mechanism to be used for motion modeling, and nonlinear motion prediction of a track is achieved. In addition, a track confidence coefficient is provided to measure the robustness of the track, and the ReID features are dynamically updated based on the track confidence coefficient. Meanwhile, in order to make up for wrong association caused by target interaction shielding, a rematching module is designed, and the effectiveness of the rematching module is proved through experiments. In addition, each module of the system has a modular characteristic and a relatively strong generalization ability, so that the system is suitable for a wider real scene, and future research work is possibly stimulated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a multi-target tracking method combining a time convolution network and a self-attention mechanism. BACKGROUND

[0002] Multi-target tracking (MOT) is a core task in computer vision, aiming to estimate the bounding boxes of targets through video sequences and maintain the continuity of target identities, and is widely used in automatic driving, intelligent monitoring, motion analysis, robots and other fields. With the rapid development of target detection technology and deep learning models, the tracking-by-detection (TBD) framework has become the mainstream method in the current MOT field. The TBD method usually performs target detection in each frame of image, and then connects the cross-frame detection results into complete target trajectories through data association algorithms. In order to further improve the tracking accuracy, the introduction of appearance features has become an important development in recent years. Through the introduction of appearance features, MOT methods can better distinguish target identities, especially in scenes where the appearance difference between targets is large. However, in complex scenes, such as frequent occlusion, high similarity between targets, and non-linear motion, the accuracy and robustness of existing MOT methods still face significant challenges.

[0003] At present, most TBD methods rely on linear motion models such as Kalman filter for trajectory prediction, match the predicted trajectory frame with the detection frame, use the matched detection frame to update the trajectory frame, and keep the unmatched trajectory frame unchanged. However, these models have limited performance in dealing with fast motion, frequent occlusion and complex motion patterns, resulting in a significant decrease in tracking accuracy. At the same time, due to the similarity of target appearance, especially between similar targets, existing feature extraction methods cannot effectively distinguish the identities of different targets, resulting in frequent ID switching problems. Although some researches introduce appearance feature embedding and attention mechanism to improve the robustness of data association, in the case of long-term occlusion and complex motion patterns, the trajectory association and management mechanism still appears too rigid and cannot effectively respond to dynamic changes. SUMMARY

[0004] The purpose of the present application is to provide a multi-target tracking method combining a time convolution network and a self-attention mechanism to solve the problems raised in the background art.

[0005] In order to solve the above technical problems, the present application provides the following technical solutions:

[0006] A multi-target tracking method combining a time convolution network and a self-attention mechanism based on the TBD paradigm, the specific method comprising:

[0007] S100, representing the non-linear motion of the target based on the TCN motion model to generate a predicted trajectory;

[0008] S200, the robustness of each trajectory is represented by a trajectory confidence module, and the appearance feature is adaptively updated based on the trajectory confidence;

[0009] S300, the predicted trajectory is matched with the detected object, and a re-matching module is embedded in the matching process to correct ID switching caused by occlusion.

[0010] The application provides a TCN-Track model, which selects YOLOX as a detector and uses a TCN-based motion model to output a predicted trajectory. A trajectory confidence module represents the robustness of the trajectory, and the appearance feature is adaptively updated based on the trajectory confidence. Finally, the predicted trajectory is matched with the detected object by using the Hungarian algorithm. A re-matching module is embedded in the matching process to correct ID switching caused by occlusion.

[0011] Preferably, the TBD paradigm includes: first, detecting the task target independently in each video frame.

[0012] Then, the detection results between consecutive frames are matched, and the position change of the object is estimated, so as to construct a continuous trajectory.

[0013] A series of consecutive frame images in a video are obtained The detector first identifies and locates all targets in to obtain a target set Each target in the set has spatial position and detection confidence information, which is represented as | ∈ } for the subsequent data association stage.

[0014] In the data association stage, The targets in the set that exceed the confidence threshold are regarded as candidate targets, wherein each candidate target carries its position attribute , the image region corresponding to the target, and the timestamp ;

[0015] The tracker associates each target in the set, thereby establishing the trajectory of the target over time , and the specific formula is: , ;

[0016] Wherein, represents the length of the trajectory of the target .

[0017] Preferably, in the TCN motion model in S100, the target trajectory Contains a series of bounding boxes B={ , ,…, };

[0018] in, Represents the timestamp, N is the total number of frames, and the bounding box is represented as... ;

[0019] The goal of multi-object tracking (MOT) is to assign a unique ID identifier to the bounding boxes of all frames. This assignment aims to establish a comprehensive association between all bounding boxes;

[0020] By creating motion models, the location of a target can be predicted, specifically including:

[0021] When the target historical trajectory length is set to At that time, the historical trajectory is represented as a sequence. Where K is the total number of targets in all frames. It is a frame index;

[0022] time goal Represented as ,in Indicates the center coordinates of the corresponding bounding box, ( ) represent the width and height of the bounding box, respectively;

[0023] Historical trajectories are input into the motion model to predict at time [time]. bounding box coordinates ;

[0024] Preferably, the TCN motion model in S100 specifically includes a hybrid architecture of temporal convolutional network TCN and self-attention mechanism;

[0025] S101, The Temporal Convolutional Network (TCN) includes three key components: dilated causal convolution module, residual TCN block, and cross-layer connection mechanism;

[0026] The dilated causal convolution module constructs a hierarchical receptive field by progressively increasing the dilation factor (d=1,2,4) and using a fixed-size filter (k=3), ensuring that the model can capture motion patterns from local to global.

[0027] The residual TCN block is used to solve the dimension mismatch problem through 1×1 convolution, which effectively alleviates the gradient vanishing problem in deep networks.

[0028] In the cross-layer connection mechanism, the identity mapping preserves the original motion features, and the feature transformation path enhances the nonlinear expression capability.

[0029] S102. A self-attention mechanism is introduced after the Temporal Convolutional Network (TCN) module. By calculating the attention weights between the query, key, and value, long-distance spatiotemporal dependencies are modeled. The calculation formula is as follows:

[0030] ;

[0031] Q, K, and V are generated from the output features of TCN through linear transformation. The formula calculates the similarity matrix of query-key, scales and normalizes it, and then performs weighted aggregation on the value matrix to finally output a feature representation with global spatiotemporal awareness.

[0032] S103, In tracing historical trajectory Before inputting the motion model for prediction, in order to accelerate model convergence and improve prediction accuracy, the trajectory sequence needs to be Z-score normalized:

[0033] ;

[0034] ;

[0035] in, This represents the th trajectory in a trajectory sequence of length N. The original coordinates of the frame ( ), The mean of the original coordinates of this trajectory sequence ( ), Standard deviation, Represents the th in the trajectory sequence Normalized coordinates of the frame.

[0036] Preferably, the Z-Score normalization process in S103 includes:

[0037] Based on valid bounding box data, the mean and standard deviation of each coordinate component are calculated. These calculated mean and standard deviation are then used to normalize the input bounding box differences, ensuring the mean is 0 and the standard deviation is 1, thus reducing scale differences between samples. During training, standardization reduces numerical instability caused by variations in input data ranges, preventing gradient explosion or vanishing gradients and maintaining model training stability. Simultaneously, standardization reduces the difference between the training and test sets, resulting in more consistent model performance across different datasets and enhancing the model's generalization ability.

[0038] In the multi-object tracking (MOT) task, the sequence data often needs to introduce a padding mechanism due to the inconsistent length of the track, so as to ensure the uniform dimension of the batch input. However, such padding frames may introduce noise, which interferes with feature extraction and prediction accuracy. Therefore, the model introduces a padding mask, the core function of which is to identify the effective frames and padding frames in the sequence;

[0039] The padding mask is introduced by detecting the all-zero vector in the input sequence to generate a binary tensor (0 represents padding and 1 represents real), and is applied to mask the invalid data during Z-score normalization: only the mean and standard deviation are calculated for real frames, avoiding the distortion of the statistical distribution by padding frames, so as to realize accurate sequence feature standardization;

[0040] In order to ensure that the loss calculation is consistent with the original scale coordinates, an inverse normalization step is applied before the output head of the prediction model to restore the predicted boundary coordinates to the actual physical scale; specifically, for each inverse normalized predicted boundary box and the corresponding real boundary box , the L1 loss between them is calculated. The loss function is represented as:

[0041] ;

[0042] Where N is the number of samples, represents the L1 norm, that is, the sum of the absolute errors of the two in each dimension.

[0043] The design of motion TCN combined with self-attention not only retains the advantages of local feature extraction of time convolution, but also enhances the modeling ability of the model for global spatio-temporal relationship through the self-attention mechanism, providing more accurate and stable motion prediction results for multi-object tracking.

[0044] Preferably, the track confidence module in S200 includes confidence modeling and dynamic ReID feature updating.

[0045] S201, the confidence modeling includes:

[0046] Set and are the detection set and the track set respectively, , ,…, is the detection confidence of each target in the detection set, wherein the confidence is obtained from the detector output result, and the detection confidence is mainly used to classify the detection frame into a high-confidence detection set and a low-confidence detection set before association, and the detection confidence This is used to classify detection boxes into high-confidence and low-confidence detection sets before association, with the trajectory confidence representing the robustness of the trajectory.

[0047] However, trajectory It is a detection set The trajectory confidence is derived from the correlation of detection boxes at different times in the data. Directly using the detection confidence of the last frame of the trajectory as the trajectory confidence is not elegant. Therefore, this paper proposes a new trajectory confidence method. Based on experience, recently updated trajectories (i.e., those active trajectories that have just been assigned detection and trajectory prediction) should have more reliable state predictions and higher confidence. Due to higher initial prediction noise, newly created trajectories should have lower confidence, gradually increasing to 1 with successful consecutive matching detections. If a match fails, the confidence will also gradually decrease, indicating reduced robustness of the trajectory. It's important to note that the scenario of re-tracking after a tracking failure needs to be considered; a trajectory confidence abruptly dropping to 1 after re-association is clearly unreasonable. Therefore, this paper recalculates the trajectory age after re-association following a tracking failure, ensuring the trajectory confidence gradually increases with re-tracking, thus avoiding abrupt confidence abrupt changes.

[0048] The new trajectory confidence According to the formula:

[0049] ;

[0050] in, , This is a hyperparameter for trajectory confidence decay; the smaller the value, the faster the confidence decays. It is the number of frames required for the trajectory confidence to increase to 1. It is a trajectory The number of consecutive successfully matched frames. It is a trajectory The number of consecutive frames where matching fails; if the trajectory fails but is re-associated in a subsequent frame, the age needs to be recalculated. ;

[0051] S202, The dynamic ReID feature update includes:

[0052] Using trajectory confidence Optimize the updating of ReID appearance features; calculate the ReID visual features describing the trajectory using frame-by-frame exponentially weighted moving average: ;

[0053] in, It is the ReID feature updated by EMA in the previous frame. It is the ReID feature of the detection result of the current frame matching the trajectory;

[0054] A weighting factor is needed. This is used to adjust the ReID ratio between historical and current time steps. In traditional EMA methods, the weighting factor α is fixed and fails to be dynamically adjusted according to changes in trajectory or detection quality, lacking a mechanism for flexible updates based on trajectory quality changes;

[0055] Based on trajectory confidence The weighting factor α value is modified in each frame, and the matching detection results are updated accordingly. and trajectory Then, the appearance features of each trajectory are updated using the appearance features and trajectory confidence scores of the corresponding matched detection results. The specific formula is as follows:

[0056] ;

[0057] ;

[0058] Where β is a hyperparameter, 0≤β≤1, used to control the lower limit of the weighting factor; It is the trajectory confidence score matched in the current frame, 0≤ ≤1; It is a dynamically adjusted weighting factor used to replace the fixed α in the traditional exponentially weighted moving average algorithm. It is the updated ReID feature from the previous frame. It is the ReID feature of the detection result of the current frame; It is the ReID feature updated in the current frame.

[0059] Preferably, the rematching module in S300 includes:

[0060] The confidence-based two-stage matching strategy proposed by ByteTrack is used, specifically including:

[0061] S301. Calculate the similarity between high-confidence detections and all trajectories using the Hungarian matching algorithm;

[0062] To reduce ID switching caused by target occlusion and motion interaction, ReID features are added to calculate the appearance similarity of the target based on IoU similarity; the cost matrix of Hungarian matching is a weighted sum of IoU similarity and ReID feature similarity, with the specific formula as follows:

[0063] ;

[0064] ;

[0065] ;

[0066] in, , These are the weights of the IoU cost and the ReID feature cost, respectively, and satisfy the following conditions: + =1; Represents the detection box and IoU; Feature similarity is represented using cosine similarity as a metric, where... , They represent the first The first detection box and the first ReID features between trajectories; Represents bounding box and The area of ​​the intersection Represents bounding box and The area of ​​the union;

[0067] However, despite their effectiveness, these methods often result in false matches when faced with frequent occlusion and complex motion patterns. Therefore, before the second-stage low-confidence matching detection, we introduce a re-matching strategy to correct high-risk association results from the first-stage matching.

[0068] To further reduce the risk of mismatches, a risk detection box is introduced. This risk detection box refers to a detection box with a high IoU value with multiple trajectories. Such detection boxes typically appear in complex tracking environments, such as when the target is occluded. Specifically, they include:

[0069] We calculate the IoU matrix between all high-confidence detection boxes and trajectory boxes in the current frame, and identify detection boxes whose IoU values ​​with multiple trajectories exceed a threshold; specifically, we set a high IoU threshold. The system then searches for bounding boxes that highly overlap with multiple trajectories. These bounding boxes typically have a high risk of mismatch, thus requiring secondary matching.

[0070] Determine the risk detection box set using the IoU matrix. The corresponding set of trajectories and will already be in the first phase and The set of successfully associated detection boxes is With risk detection box set Merged into a secondary detection set ;

[0071] S302. For high-confidence and low-confidence detection boxes that failed to match in the first stage, the matching threshold is lowered in the second stage to perform a second match.

[0072] The second phase focuses on addressing the "risk detection boxes" from the first phase's matching process and potential mismatches; among which... It includes all detection boxes that need to be re-evaluated in the secondary matching;

[0073] This strategy allows us to more accurately re-match potential mismatched targets, ensuring the correct association of targets during multi-target tracking.

[0074] Specifically, a greedy algorithm is used for matching: Unlike the globally optimal Hungarian algorithm, the greedy algorithm achieves a local optimum by selecting the best matching pair one by one. In each matching step, the matching cost between each pair of detection boxes and trajectories is calculated. This cost also consists of two parts: IoU similarity and appearance similarity of ReID features. During the matching process, the greedy algorithm selects the detection box and trajectory with the minimum matching cost each time and updates the matching result. For successfully matched detection boxes and trajectories, the system updates their state to ensure accurate tracking in subsequent frames.

[0075] By employing a rematching strategy, mismatches caused by occlusion and target interaction can be significantly reduced, thereby improving the accuracy and robustness of tracking.

[0076] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0077] This invention enhances the ability to model temporal dependencies, effectively adapting to complex nonlinear motion patterns, especially in occluded and rapidly changing scenarios, significantly improving the accuracy of target trajectory prediction. Simultaneously, a dynamic trajectory confidence update mechanism is designed, which adjusts the trajectory confidence score in real time by combining historical matching performance to improve the stability of data association. For trajectory management after occlusion and loss, a secondary re-matching strategy based on high-confidence screening is proposed. After conventional Hungarian matching, a secondary screening and correction is performed on potential risk detection boxes, effectively reducing the risk of mismatches in occluded scenarios.

[0078] This invention achieves significant tracking performance on multiple publicly available datasets, particularly demonstrating strong robustness and stability on the SportsMOT and DanceTrack datasets. The SportsMOT dataset involves fast-moving sports scenes, testing tracking accuracy under complex motion patterns; while the DanceTrack dataset highlights the challenges of high appearance similarity between targets and frequent occlusion, validating the effectiveness of this method in these complex scenarios. Attached Figure Description

[0079] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0080] Figure 1 This is a flowchart of a multi-target tracking method combining a temporal convolutional network and a self-attention mechanism according to the present invention;

[0081] Figure 2 This is a diagram of the TCN-Track model architecture of the present invention;

[0082] Figure 3 This is a diagram of the TCN motion model of the present invention;

[0083] Figure 4 This is a diagram of the architectural elements in the TCN motion model of the present invention; where (a) is dilated causal convolution; (b) is a TCN residual block; (c) is an example of residual connection in TCN; the blue line represents the filter in the residual function, and the green line represents the identity mapping;

[0084] Figure 5 This is a comparison of the tracking performance of the ByteTrack and TCN-Track trackers before and after an occlusion event in the SportsMOT dataset. "a", "b", and "c" represent the tracking results at frames 22, 42, and 68, respectively, which represent the scene before, during, and after occlusion.

[0085] Figure 6 This is a comparison chart of the tracking performance of the ByteTrack and TCN-Track trackers before and after occlusion events in the DanceTrack dataset. "a", "b" and "c" are the tracking results of frames 131, 143 and 149, respectively, representing the scene before, during and after occlusion.

[0086] Figure 7 This is an evaluation graph on the SportsMOT test set in an embodiment of the present invention; the best results are shown in bold, and higher or lower values ​​(marked with ↑ / ↓) indicate better performance;

[0087] Figure 8 This is a graph showing the ablation experiment results on the SportsMOT validation set in this embodiment of the invention; where higher or lower values ​​(marked with ↑ / ↓) indicate better performance;

[0088] Figure 9 In this embodiment of the invention, the TCN motion model is applied to other tracker data graphs in the SportMOT test set experiment. Detailed Implementation

[0089] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0090] Please see Figures 1-9 The present invention provides the following technical solution:

[0091] Example 1: A multi-target tracking method combining temporal convolutional networks and self-attention mechanisms, based on the TBD paradigm, proposes a TCN-Track model architecture;

[0092] Specific experimental methods include:

[0093] Datasets: To verify the effectiveness of the proposed method, this paper conducted experimental evaluations on two emerging datasets, DanceTrack

[23] and SportsMOT[5], which present challenges of rapid motion changes and appearance similarity. The DanceTrack dataset contains 40 training videos, 25 validation videos, and 35 test videos. Although the targets in the dance scenes in this dataset are easy to detect, they are difficult to distinguish due to their highly similar appearances and complex and varied motion patterns, which places high demands on the tracking algorithm. SportsMOT is another newly proposed dataset that focuses on sports scenes such as basketball, football, and volleyball. It contains 45 training videos, 45 validation videos, and 150 test videos, all of which are collected from high-level sports events. Due to the rapid and varied motion characteristics of athletes, this dataset requires the tracking algorithm to have stronger robustness.

[0094] Evaluation Metrics: For evaluation metrics, we adopted a multi-dimensional comprehensive evaluation system, mainly including High-Order Tracking Accuracy (HOTA), IDF1 score, and Multi-Target Tracking Accuracy (MOTA). HOTA, as the core metric, integrates detection accuracy and association accuracy, providing a comprehensive assessment of the algorithm's performance. The IDF1 score measures the consistency of target trajectories, reflecting the accuracy of target association. MOTA focuses on the overall performance of multi-target tracking, comprehensively considering the number of missed detections (FN), false alarms (FP), and identity switching (IDs), and is primarily affected by detection quality. These metrics collectively provide a comprehensive performance evaluation of the algorithm.

[0095] Implementation Details: This research aims to develop a robust tracker based on the TCN motion model. To ensure fair comparison, we directly use the YOLOX detector pre-trained weights provided by the DanceTrack and SportsMOT benchmarks. The ReID appearance model uses a pre-trained FastReID model to extract the appearance features of the target. The TCN motion model implementation employs specific parameter configurations: the input dimension is 4 corresponding to the bounding box coordinates, the number of channels in the 3-layer TCN is set to [32, 64, 128] to achieve layer-by-layer feature expansion, the attention dimension is matched with the last TCN layer at 128, the output dimension is 4 corresponding to the predicted bounding box, and the dropout rate is set to 0.5 to prevent overfitting.

[0096] The training samples were constructed using a sliding window method (window length L=10), sampling was performed time-stamp by time starting from each video sequence. To simulate the initial tracking trajectory length being less than 10 frames, the probability of sample sampling was ( If the sampling length is less than 10 frames (L=1~9), padding with zero values ​​is used to bring it up to 10 frames. During training, the learning rate... The training consists of 100 rounds.

[0097] The key parameter configurations for trajectory confidence modeling are as follows: , When using trajectory confidence to improve ReID appearance feature updates, The experimental hardware environment consisted of an Intel(R) Xeon(R) CPU E5-2640 processor (base frequency 2.4 GHz), 188GB of RAM, and an Nvidia RTX1080Ti graphics card. The software environment was implemented using Python 3.9 and PyTorch 1.9.0 frameworks.

[0098] Benchmark dataset results: The proposed TCN-Track is compared with the officially released state-of-the-art methods on the SportsMOT test set, and the results are as follows: Figure 7 As shown. Results for other methods are derived from official benchmarks and relevant papers.

[0099] SportsMOT test results are as follows: Figure 7As shown, even without using ReID appearance information, our proposed TCN-Track outperforms similar tracking algorithms that rely solely on motion information across all metrics. Notably, on the HOTA metric, our method, also without using appearance ReID features, outperforms ByteTrack, which uses a Kalman filter, by nearly 12 percentage points, and on IDF1 and MOTA metrics, it outperforms by 5.2 and 1.8 percentage points, respectively. Furthermore, our method surpasses OC-SORT, an enhanced Kalman filter method, demonstrating superior performance. Introducing ReID appearance features significantly improves our method across HOTA, IDF1, and MOTA metrics, with an approximately 4 percentage point improvement on HOTA compared to the version using only motion information, indicating that ReID appearance features significantly enhance target discrimination and tracking accuracy. This further illustrates the importance of ReID appearance information in complex scenes, particularly its advantages in handling target occlusion and interactions. These results highlight the state-of-the-art capabilities of our method, validating its effectiveness even in scenarios involving fast and diverse athlete movements.

[0100] Ablation Experiments: In this section, we conducted ablation experiments on the SportsMOT validation set to analyze performance changes after modifying one or more components. The TCN-Track tracker consists of three main components: a TCN-based motion model, dynamic ReID feature updates based on trajectory confidence, and a re-matching strategy. Results are shown below. Figure 8 (SportsMOT dataset).

[0101] Like TCN-Track, BaseLine uses YOLOX as its detector. The ReID appearance model also uses a pre-trained FastReID model to extract appearance features, and the default appearance feature update method uses a fixed... The exponentially weighted moving average was used. The matching and association phase employed the two-stage association method proposed by ByteTrack, which divided the detection results into two stages based on the detection confidence and associated them separately. No post-processing methods, such as Gaussian interpolation, were applied to the final results in any of the experiments.

[0102] For the SportsMOT dataset, such as Figure 8As shown, the ReID model performs significantly, with a 2.3% improvement in HOTA (Hyperactivity Over Time) on the Baseline by adding ReID appearance features. This highlights the importance of a robust motion model on non-linear motion datasets. Using the same ReID model (traditional exponentially weighted moving average update method), replacing the standard Kalman filter (KF) with the TCN motion model increases HOTA by 2.9%. Applying dynamic updates to appearance features based on trajectory confidence increases HOTA by 1.1%. Due to the less crowded scene and the highly effective appearance model, the proposed re-matching strategy is less effective than dynamic ReID feature updates. The order of component importance is: TCN motion model, dynamic ReID feature updates, and re-matching strategy.

[0103] The TCN motion model was applied to other trackers. Furthermore, the TCN motion model was applied to a KF-based tracker to validate its effectiveness.

[0104] like Figure 9 As shown, to independently test the superiority of the TCN motion model in nonlinear motion prediction, the Kalman filter was replaced with the TCN motion model on the ByteTrack and OC-SORT (an improvement on the KF) trackers, respectively. Both trackers use only motion features for tracking and do not provide appearance information. On the SportsMOT test set, the TCN motion model improved the HOTA index by 10.2% and 2.4% on the two trackers, respectively.

[0105] Visual qualitative analysis: To more intuitively demonstrate the effectiveness of the proposed method, such as... Figure 5 , Figure 6 As shown, we tested TCN-Track in complex multi-object scenarios and compared it with ByteTrack. These scenarios involved athletes and dancers wearing similar clothing, performing irregular nonlinear movements, and experiencing severe occlusion, thus rigorously testing the tracking algorithm's ability to handle occlusion and nonlinear motion problems.

[0106] like Figure 5 As shown, the tracked targets are volleyball players wearing red and white jerseys. a, b, and c represent the tracking performance of each tracker before, during, and after object occlusion, respectively. In stage b, the red jersey number 2 (target 3) is occluded by teammates of the same color, target 7 and target 9. ByteTrack failed to effectively handle the occlusion and the player's movement. In stage c, the ID of the player in the red jersey number 2 changes to 9, indicating an ID switch between target 3 and target 9. Our TCN-Track, however, maintained accurate tracking of target 3 after the occlusion occurred, without any ID switch.

[0107] Figure 6The demonstration showcased tracking performance in a dance scene. In phase b, target 5 almost completely occluded target 7. After the occlusion ended, ByteTrack switched the IDs of target 5 and target 7. Although TCN-Track also lost detection of target 5 in phase b, it accurately tracked the re-emerging target 5 in phase c, demonstrating its effective tracking capability in scenes with large areas of occlusion.

[0108] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-target tracking method combining temporal convolutional networks and self-attention mechanisms, based on the TBD paradigm, characterized in that: The method includes: S100: Based on the TCN motion model, the nonlinear motion of the target is characterized, and a predicted trajectory is generated; S200: The robustness of each trajectory is characterized by the trajectory confidence module, and the appearance features are adaptively updated based on the trajectory confidence. S300: Match the predicted trajectory with the detected object. During the matching process, embed a rematching module to correct ID switching caused by occlusion.

2. The multi-target tracking method combining temporal convolutional networks and self-attention mechanisms as described in claim 1, characterized in that, The TBD paradigm includes: Acquire a series of consecutive frames from a video. The detector first identifies and locates All targets in the target set are used to obtain a target set. Each target in the set has spatial location and detection confidence information, represented as { | ∈ }, used for subsequent data association stages; During the data association phase, Targets in the set that exceed the confidence threshold will be considered as candidate targets, where each candidate target They all carry their location attributes Image region corresponding to the target and timestamp ; The tracker will Each target in the set is linked together to establish its trajectory over time. The specific formula is as follows: , ; in, Indicate target The length of the trajectory.

3. The multi-target tracking method combining temporal convolutional networks and self-attention mechanisms as described in claim 1, characterized in that, The TCN motion model in S100 includes a hybrid architecture of temporal convolutional network TCN and self-attention mechanism; S101, The Temporal Convolutional Network (TCN) includes three key components: dilated causal convolution module, residual TCN block, and cross-layer connection mechanism; The dilated causal convolution module constructs a hierarchical receptive field by progressively increasing the dilation factor and using a fixed-size filter, ensuring that the model can capture motion patterns from local to global. The residual TCN block is used to solve the dimension mismatch problem through 1×1 convolution; In the cross-layer connection mechanism, the identity mapping preserves the original motion features, and the feature transformation path enhances the nonlinear expression capability. S102. A self-attention mechanism is introduced after the Temporal Convolutional Network (TCN) module. By calculating the attention weights between the query, key, and value, long-distance spatiotemporal dependencies are modeled. The calculation formula is as follows: ； Q, K, and V are generated from the output features of TCN through linear transformation. The formula calculates the similarity matrix of query-key, scales and normalizes it, and then performs weighted aggregation on the value matrix to finally output a feature representation with global spatiotemporal awareness. S103, In tracing historical trajectory Before inputting the motion model for prediction, in order to accelerate model convergence and improve prediction accuracy, the trajectory sequence needs to be Z-score normalized: ； ； in, This represents the th trajectory in a trajectory sequence of length N. The original coordinates of the frame ( ), The mean of the original coordinates of this trajectory sequence ( ), Standard deviation, Represents the th in the trajectory sequence Normalized coordinates of the frame.

4. The multi-target tracking method combining temporal convolutional networks and self-attention mechanisms as described in claim 3, characterized in that, The Z-Score normalization process in S103 includes: Based on the valid bounding box data, calculate the mean and standard deviation of each coordinate component, and use the calculated mean and standard deviation to normalize the input bounding box difference, so that the mean of the data is 0 and the standard deviation is 1, thereby reducing the scale difference between samples. A padding mask is introduced, which generates a binary tensor by detecting all-zero vectors in the input sequence and is applied to mask invalid data during Z-score normalization: the mean and standard deviation are calculated only for the real frames, avoiding the distortion of the statistical distribution by the padding frames, thereby achieving accurate sequence feature normalization; To ensure that the loss calculation matches the coordinates at the original scale, an inverse normalization step is applied before the output head of the prediction model to restore the predicted boundary coordinates to the actual physical scale; specifically, for each inverse normalized predicted bounding box... and the corresponding true bounding box Calculate the L1 loss between them. The loss function is expressed as: ； Where N is the number of samples, This represents the L1 norm, which is the sum of the absolute errors of the two in each dimension.

5. The multi-target tracking method combining temporal convolutional networks and self-attention mechanisms as described in claim 1, characterized in that, The trajectory confidence module in S200 includes: confidence modeling and dynamic ReID feature update; S201, The confidence modeling includes: set up and They are the detection set and the trajectory set, respectively. , ,…, It is the detection confidence score for each target in the detection set, wherein the confidence score is obtained from the detector output. This is used to classify detection boxes into high-confidence and low-confidence detection sets before association, with the trajectory confidence representing the robustness of the trajectory. To improve trajectory confidence, a new trajectory confidence method is proposed. According to the formula: ； in, , It is the decay hyperparameter of trajectory confidence. It is the number of frames required for the trajectory confidence to increase to 1. It is a trajectory The number of consecutive successfully matched frames. It is a trajectory The number of consecutive frames where matching fails; if the trajectory fails but is re-associated in a subsequent frame, the age needs to be recalculated. ; S202, The dynamic ReID feature update includes: Using trajectory confidence Optimize the updating of ReID appearance features; calculate the ReID visual features describing the trajectory using frame-by-frame exponentially weighted moving average: ; in, It is the ReID feature updated by EMA in the previous frame. It is the ReID feature of the detection result of the current frame matching the trajectory; Based on trajectory confidence The weighting factor α value is modified in each frame, and the matching detection results are updated accordingly. and trajectory Then, the appearance features of each trajectory are updated using the appearance features and trajectory confidence scores of the corresponding matched detection results. The specific formula is as follows: ； ； Where β is a hyperparameter, 0≤β≤1, used to control the lower limit of the weighting factor; It is the trajectory confidence score matched in the current frame, 0≤ ≤1; It is a dynamically adjusted weighting factor used to replace the fixed α in the traditional exponentially weighted moving average algorithm. It is the updated ReID feature from the previous frame. It is the ReID feature of the detection result of the current frame; It is the ReID feature updated in the current frame.

6. The multi-target tracking method combining temporal convolutional networks and self-attention mechanisms as described in claim 1, characterized in that, The rematching module in S300 includes: The confidence-based two-stage matching strategy proposed by ByteTrack is used, specifically including: S301. Calculate the similarity between high-confidence detections and all trajectories using the Hungarian matching algorithm; To reduce ID switching caused by target occlusion and motion interaction, ReID features are added to calculate the appearance similarity of the target based on IoU similarity; the cost matrix of Hungarian matching is a weighted sum of IoU similarity and ReID feature similarity, with the specific formula as follows: ； ； ； in, , These are the weights of the IoU cost and the ReID feature cost, respectively, and satisfy the following conditions: + =1; Represents the detection box and IoU; Feature similarity is represented using cosine similarity as a metric, where... , They represent the first The first detection box and the first ReID features between trajectories; Represents bounding box and The area of ​​the intersection Represents bounding box and The area of ​​the union; To further reduce the risk of mismatches, a risk detection box is introduced, which refers to a detection box that has a high IoU value with multiple trajectories; specifically including: Calculate the IoU matrix between all high-confidence detection boxes and trajectory boxes in the current frame, and identify detection boxes whose IoU values ​​with multiple trajectories exceed the threshold; Determine the risk detection box set using the IoU matrix. The corresponding set of trajectories and will already be in the first phase and The set of successfully associated detection boxes is With risk detection box set Merged into a secondary detection set ; S302. For high-confidence and low-confidence detection boxes that failed to match in the first stage, the matching threshold is lowered in the second stage to perform a second match. The second phase focuses on addressing the "risk detection boxes" from the first phase's matching process and potential mismatches; among which... It includes all detection boxes that need to be re-evaluated in the secondary matching; Specifically, a greedy algorithm is used for matching: In each matching step, the matching cost between each pair of detection boxes and trajectories is calculated. This cost also consists of two parts: IoU similarity and appearance similarity of ReID features. During the matching process, the greedy algorithm selects the detection box and trajectory with the minimum matching cost each time and updates the matching result. For successfully matched detection boxes and trajectories, the system updates their state to ensure accurate tracking in subsequent frames.

Images