Multi-target tracking method and system based on dynamic cue gating and temporal augmentation

Abstract

The application relates to the technical field of computer vision, and discloses a multi-target tracking method and system based on dynamic clue gating and timing enhancement, which comprises the following steps: acquiring a target video, forming a detection set through target detection and feature extraction, constructing a historical trajectory set, and each trajectory in the set comprising an appearance clue and a motion clue; inputting the appearance clue and the motion clue into a TFEN to obtain multi-clue embedding of a detection end and a trajectory end; inputting the multi-clue embedding of the detection end and the trajectory end into a CGAF, and under the constraint of a spatial neighborhood, performing dynamic gating fusion on different clue embeddings to obtain fused detection embedding and trajectory embedding; constructing a trajectory-detection similarity matrix to obtain a matching result of the trajectory and the detection; updating a target trajectory state and maintaining a target identity ID according to the matching result; and outputting a multi-target tracking result. The application can adaptively adjust the contribution of the appearance clue and the motion clue, suppress the influence of noise, and improve the accuracy of target tracking in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-target tracking method and system based on dynamic cue gating and temporal enhancement. Background Technology

[0002] Multi-object tracking (MOT) aims to continuously locate multiple targets in a video sequence and maintain a consistent identity (ID) for each target across frames, forming a stable target trajectory. MOT has significant application value in fields such as autonomous driving, intelligent security, sports event analysis, industrial inspection, behavior understanding, and video retrieval.

[0003] In existing technologies, the Tracking-by-Detection (TbD) paradigm is widely used due to its clear structure and strong modular substitutability. The TbD paradigm includes: performing target detection and appearance feature extraction on each frame, constructing a cost matrix based on motion consistency and appearance similarity, using the Hungarian algorithm or a greedy strategy to complete the trajectory-detection association, and maintaining the trajectory lifecycle through a trajectory state machine.

[0004] However, multi-target tracking still faces several challenges in complex dynamic scenarios with frequent occlusion, nonlinear motion, and appearance drift. These challenges include: dense target density leading to frequent occlusion; nonlinear motion such as rapid turns, sudden stops, and starts rendering motion cues unreliable; and appearance cue drift caused by changes in illumination, scale, and similar appearances, leading to false positives and false negatives by the detector. Furthermore, existing temporal modeling methods tend to densely aggregate historical trajectories to obtain long-range context, but this aggregation process can weaken short-term, high-frequency discriminative information, making it difficult to make stable association decisions at critical moments such as occlusion recovery and rapid maneuvers, thus affecting target tracking accuracy. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multi-target tracking method and system based on dynamic cue gating and temporal enhancement, which can adaptively adjust the contribution of appearance cues and motion cues, suppress the influence of noise historical fragments, improve the association robustness and identity consistency in complex scenes, and improve the accuracy of target tracking.

[0006] To address the aforementioned technical problems, this invention provides a multi-target tracking method based on dynamic cue gating and temporal enhancement, comprising: Acquire the target video and preprocess it to obtain a frame-by-frame image sequence. The image sequence is subjected to target detection and feature extraction to form a detection set, the target historical trajectory is obtained and a historical trajectory set is constructed, and each detection in the detection set and each trajectory in the historical trajectory set includes appearance cues and motion cues; The appearance cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the appearance embedding of the detection end. The motion cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the motion embedding of the detection end. The appearance embedding and motion embedding of the detection end are used as the multi-cue embedding of the detection end. The appearance cues in the historical trajectory set are input into a temporal feature enhancement network for encoding to obtain the trajectory-end appearance embedding. The motion cues in the historical trajectory set are input into a temporal feature enhancement network for encoding to obtain the trajectory-end motion embedding. The trajectory-end appearance embedding and the trajectory-end motion embedding are used as the trajectory-end multi-cue embedding. The multi-clue embeddings at the detection end and the multi-clue embeddings at the trajectory end are input into the cue-gated attention fusion module. Under the spatial neighborhood constraint, different cue embeddings are dynamically gated and fused to obtain the fused detection embedding and the fused trajectory embedding. A trajectory-detection similarity matrix is ​​constructed based on the fused detection embedding and the fused trajectory embedding. The trajectory-detection similarity matrix is ​​then used to perform matching to obtain the matching result between the trajectory and the detection. Update the target trajectory status and maintain the target identity ID based on the matching results, and output the multi-target tracking results.

[0007] Further, the appearance cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the appearance embedding at the detection end, including: The input appearance cues are normalized and nonlinearly mapped to obtain basic features. The basic features are then multi-scale pooled to generate multi-scale summary feature units. The multi-scale summary feature units are superimposed with position encoding and scale encoding and then concatenated with the appearance cues to form a hybrid sequence. Temporal self-attention encoding is performed on the hybrid sequence, the saliency of the feature units in the encoded hybrid sequence is scored, and a set of key feature units is obtained by filtering. The [CLS] feature units are concatenated with the set of key feature units and then attention aggregation is performed to obtain the [CLS] aggregated embedding. Baseline features are extracted based on the input appearance cues. A gating vector is learned based on the baseline features. The [CLS] aggregate embedding and the baseline features are weighted and fused using the gating vector to obtain the appearance embedding of the detection end.

[0008] Furthermore, the appearance of the detection end is embedded as follows: , In the formula, For embedding in the appearance of the detection end, Represents a linear mapping. For the gated vector, [CLS] aggregation embedding, Baseline characteristics, This indicates element-wise multiplication.

[0009] Furthermore, the step of scoring the saliency of feature units in the encoded mixed sequence and filtering to obtain a set of key feature units includes: Each feature unit is scored for saliency using a scorer to obtain a scalar score. Invalid locations are masked during the saliency scoring process. Through Top- Hard selection or soft selection during the training phase filters scalar scores, and the feature units corresponding to the filtered scalar scores are used as the set of key feature units.

[0010] Furthermore, the aforementioned Top- The scalar score for hard selection or soft selection during the training phase is as follows: When using Top- When using hard selection to filter scalar scores , For the number of key segments, To retain the proportion, The length of the mixed sequence, To round up; When using soft selection to filter scalar scores during the training phase, a softmax operation with temperature scaling is performed on the scalar scores to obtain a weight vector, and the feature units are weighted according to the weight vector; during temperature scaling, the temperature anneals as the training process progresses.

[0011] Furthermore, the multi-cue embedding at the detection end and the multi-cue embedding at the trajectory end are input to the cue-gated attention fusion module, and dynamic gating fusion of different cue embeddings is performed under spatial neighborhood constraints to obtain the fused detection embedding and the fused trajectory embedding, including: The multi-clue embeddings at the detection end and the multi-clue embeddings at the trajectory end are concatenated according to the object dimension to form a unified feature unit sequence; Calculate the set of object center points formed by the detection center point and the trajectory center point, and select the k nearest neighbors in the spatial neighborhood for each object center point to construct an attention mask; Under the attention mask constraint, neighborhood self-attention interaction is performed on the feature unit sequence to obtain a context fusion representation; Gated attention fusion is performed on the context fusion representation in the multi-cue dimension to obtain the fusion representation. The fusion representation is then divided into fused detection embedding and fused trajectory embedding according to the object dimension.

[0012] Furthermore, the step of performing gated attention fusion on the context fusion representation in the multi-cue dimension to obtain the fusion representation specifically involves: The context fusion representation is used as the Q value, and the multi-clue embedding at the detection end is used as the K value and V value; Based on the compatibility of the context fusion representation with the multi-clue embedding of the detection end, the output gate bias is used to obtain the attention score according to the Q value and K value. The gate bias and the attention score are added together to obtain the cue dimension score. The clue dimension score is subjected to a single softmax normalization to obtain the clue weight. The clue weight is then used to perform clue dimension weighted aggregation on the V value to obtain the fused representation.

[0013] Furthermore, the elements of the trajectory-detection similarity matrix are the cosine similarity or normalized inner product between the fused trajectory embedding and the fused detection embedding. Before matching, a threshold constraint or distance gating constraint is applied to the trajectory-detection similarity matrix to eliminate unreasonable ones.

[0014] Furthermore, when matching based on the trajectory-detection similarity matrix, the Hungarian algorithm or a greedy strategy is used for matching.

[0015] This invention also provides a multi-target tracking system based on dynamic cue gating and temporal enhancement, comprising: The video input module acquires the target video. The preprocessing module preprocesses the target video to obtain a frame-by-frame image sequence; The detection module performs target detection on the image sequence and outputs detection boxes; The feature extraction module extracts features based on the target detection results to obtain appearance cues and motion cues, forming a detection set, wherein each detection in the detection set includes appearance cues and motion cues; The trajectory management module acquires the target's historical trajectory and constructs a historical trajectory set, maintaining the trajectory status and historical cache. Each trajectory in the historical trajectory set includes appearance cues and motion cues. The TFEN encoding module performs temporal enhancement and sparse aggregation on appearance cues and motion cues in the detection set and appearance cues and motion cues in the historical trajectory set to obtain detection end appearance embedding, detection end motion embedding, trajectory end appearance embedding and trajectory end motion embedding. The detection end appearance embedding and detection end motion embedding are used as multi-cue embeddings of the detection end, and the trajectory end appearance embedding and trajectory end motion embedding are used as multi-cue embeddings of the trajectory end. The CGAF fusion module performs dynamic gating fusion of the multi-cue embeddings at the detection end and the multi-cue embeddings at the trajectory end under spatial neighborhood constraints, to obtain the fused detection embedding and the fused trajectory embedding. The matching module constructs a trajectory-detection similarity matrix based on the fused detection embedding and the fused trajectory embedding, and performs matching based on the trajectory-detection similarity matrix to obtain the matching result between the trajectory and the detection. The interaction and configuration module updates the target trajectory status and maintains the target identity ID based on the matching results; The visualization and export module outputs multi-target tracking results.

[0016] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention acquires temporal sequences of appearance and motion cues from a detection set and historical trajectories. These sequences are then processed by a temporal feature enhancement network to obtain multi-cue embeddings at both the detection and trajectory ends. The multi-cue embeddings from both ends are input into a cue-gated attention fusion module, where they are dynamically weighted and fused under spatial neighborhood constraints to obtain fused detection and trajectory embeddings. Based on this, a trajectory-detection similarity matrix is ​​obtained for association, enabling adaptive adjustment of the contributions of appearance and motion cues and suppressing the influence of noisy historical segments. Matching is performed based on the trajectory-detection similarity matrix, and the target trajectory state is updated using the matching results. The target identity ID is maintained, and multi-target tracking results are obtained. This improves the robustness and identity consistency of association in complex scenarios such as frequent occlusion, nonlinear motion, or appearance drift, thereby enhancing the accuracy of target tracking. Attached Figure Description

[0017] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 The flowchart is shown in the preferred embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the trajectory lifecycle management and ID maintenance process.

[0019] Figure 3 This is an architecture diagram of the system in a preferred embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0021] Reference Figure 1As shown, this invention discloses a multi-target tracking method based on dynamic cue gating and temporal enhancement, comprising the following steps: S1: Obtain the target video and running parameters, and preprocess the target video to obtain a frame-by-frame image sequence.

[0022] The target video is the path to the video file or the video stream to be processed. Video file formats include mp4, avi, mkv, or mov, with a frame rate ranging from 10 to 60 fps; resolution can be standard definition or high definition.

[0023] Optionally, runtime parameters can be obtained, including: detector type, appearance feature extractor type, and historical trajectory window length. Multi-scale pooling length set Retention ratio CGAF neighborhood size Similarity threshold, missing / retained frame threshold, output directory, etc.

[0024] Preprocessing includes: decoding the video to obtain frame-by-frame images; performing scaling, color space conversion, and normalization on each frame; and optionally performing denoising, sharpening, or histogram equalization on the frames to adapt to different lighting and weather conditions. Timestamp and frame rate information are parsed, and detection, feature extraction, correlation, and visualization are pipelined and scheduled in a multi-threaded or asynchronous queue to improve overall throughput.

[0025] S2: Perform target detection and feature extraction on the image sequence to form a detection set, obtain the target historical trajectory and construct a historical trajectory set. Each detection in the detection set and each trajectory in the historical trajectory set includes appearance cues and motion cues.

[0026] S2-1: Target detection.

[0027] "Detection" refers to the target candidate bounding box output by the detector in the current frame. The detection set includes several detected objects in each frame, denoted as . , , This is the i-th detection object. Each detection object contains at least: bounding box coordinates (denoted as...). ), detection confidence score (denoted as ), appearance embedding vector, detection index (i.e. The bounding box coordinates can be represented by the top-left corner and width / height, or the top-left corner and bottom-right corner. The detector can be a single-stage or two-stage detector, and the detection output can be filtered according to a confidence threshold. A non-maximum suppression threshold can be configured to control the number of overlapping boxes.

[0028] S2-2: Extract appearance cues and construct motion cues.

[0029] For each detection box, a target image patch is cropped and input into an appearance feature extractor, which outputs appearance cues. Simultaneously, motion cues are constructed for the bounding box coordinates, including center point, width and height, area, aspect ratio, velocity estimation, and historical displacement. The appearance feature extractor can be a ReID network, a contrastive learning encoder, or a visual Transformer encoder, and the motion cues can be obtained by Kalman filtering to predict the state or by simple difference estimation. Furthermore, the appearance cues can be processed... Normalization facilitates cosine similarity calculation; motion cues can be normalized to a uniform scale to reduce the impact of different resolutions on the threshold.

[0030] "Clues" can refer to appearance cues, motion cues, or other signals that can characterize the consistency of a target's identity. Appearance cues include: target appearance feature vectors obtained by cropping from the target detection bounding boxes, the target's historical appearance sequence, and its confidence level. Motion cues include: bounding box position and scale, target center point, velocity estimation, acceleration estimation, historical displacement sequence, trajectory age, and number of lost frames.

[0031] S2-3: Historical trajectory maintenance and feature caching.

[0032] Record the collection of historical trajectories , , Let j be the j-th historical trajectory; for each trajectory, maintain the trajectory status, trajectory age, number of lost frames, and historical window length (denoted as ). The appearance clues within (denoted as) ) and motion cue sequences (denoted as ), fill in the missing positions with a mask and record the validity mask (denoted as ), ).like Figure 2 As shown, trajectory lifecycle management includes states such as Tentative (trajectory initialization), Confirmed (trajectory confirmation), Lost (trajectory loss retention), and Removed (trajectory termination and deletion). An ID pool is maintained to ensure consistency of identity for the same target across frames. When consecutive trajectory matches reach a threshold, the trajectory transitions from Tentative to Confirmed; when consecutive unmatched trajectories exceed the threshold, the trajectory transitions from Lost to Removed. "Historical trajectory" refers to the trajectory in the most recent... Intra-frame state sequence and feature sequence.

[0033] S3: The appearance cues from the detection set are input into a Temporal Feature Enhancement Network (TFEN) for encoding to obtain the appearance embedding of the detection end. Similarly, the motion cues from the detection set are input into the TFEN for encoding to obtain the motion embedding of the detection end. The appearance embedding and motion embedding of the detection end are then used as the multi-cue embedding of the detection end. For each detection set, the appearance cues and motion cues are input into the corresponding encoder or mapping layer to obtain the appearance embedding of the detection end (denoted as...). ) and motion embedding at the detection end (denoted as ). (Can be adopted) Degenerate forms or short-window buffers can be used to improve stability.

[0034] S3-1: Normalize and nonlinearly map the input appearance cues to obtain the basic features.

[0035] S3-2: Multi-scale pooling is performed on the basic features to generate multi-scale summary feature units. These multi-scale summary feature units are then superimposed with positional and scale codes and concatenated with appearance cues to form a hybrid sequence. Each "feature unit" can correspond to a token within the deep network. In this embodiment, the length set of the multi-scale pooling is... ,in, is the scale number and , Let be the length of the i-th scale pooling. It is a positive integer; specifically, the length set of multi-scale pooling can be... Alternatively, combinations can be used to simultaneously cover long-range trends and short-range dynamics. Pooling operations can be adaptive average pooling or weighted pooling to obtain summary representations at different time scales. Position encoding can be fixed sinusoidal encoding or learnable encoding, and scale encoding is used to distinguish the sources of different pooling scales.

[0036] S3-3: Perform temporal self-attention encoding on the mixed sequence.

[0037] S3-4: Score the saliency of the feature units in the encoded mixed sequence and select the key feature unit set.

[0038] S3-4-1: Each feature unit is scored for saliency using a scorer to obtain a scalar score. Invalid locations are masked during saliency scoring. The scorer can be a linear layer or an attention scorer.

[0039] S3-4-2: Through Top- Hard selection or soft selection during the training phase filters scalar scores, and the feature units corresponding to the filtered scalar scores are used as the set of key feature units.

[0040] When using Top- When using hard selection to filter scalar scores , For the number of key segments, To retain the proportion, , The length of the mixed sequence, This is for rounding up.

[0041] When using soft selection to filter scalar scores during the training phase, a softmax operation with temperature scaling is applied to the scalar scores to obtain a weight vector. Feature units are then weighted according to this weight vector to approximate a Top-order distribution. Hard selection; during temperature scaling, the temperature is annealed as training progresses to gradually approach the Top-order temperature. Hard choice.

[0042] S3-5: After concatenating the [CLS] feature (i.e. classification label feature) unit with the key feature unit set, attention aggregation is performed to obtain the [CLS] aggregated embedding.

[0043] S3-6: Extract baseline features based on the input appearance cues, learn a gating vector based on the baseline features, and use the gating vector to perform a weighted fusion of the [CLS] aggregate embedding and the baseline features to obtain the appearance embedding at the detection end: , In the formula, For embedding in the appearance of the detection end, Represents a linear mapping. For the gated vector, , Representing dimension, [CLS] aggregation embedding, As the baseline feature, the mean value of the input appearance cues is obtained. , This indicates element-wise multiplication.

[0044] By employing high-resolution feedback gating, short-term discriminative information can be enhanced, and short-term discriminative details weakened by the aggregation process can be recovered. The encoding method for motion embedding at the detection end is the same as that for appearance embedding at the detection end.

[0045] S4: Input the appearance cues from the historical trajectory set into the temporal feature enhancement network for encoding to obtain the trajectory appearance embedding, and input the motion cues from the historical trajectory set into the temporal feature enhancement network for encoding to obtain the trajectory motion embedding. Use the trajectory appearance embedding and trajectory motion embedding as the trajectory multi-cue embedding.

[0046] For a set of historical trajectories, the appearance cues and motion cues of each trajectory are constructed into temporal feature unit sequences and input into TFEN to obtain the trajectory-end appearance embedding (denoted as ). ) and trajectory-end motion embedding (denoted as The encoding methods for trajectory-end appearance embedding and trajectory-end motion embedding are the same as those for detection-end appearance embedding. The multi-cusp embedding shape for the trajectory end is... Or flattened ,in, For batch size, For the number of trajectories, For historical length, For feature dimensions.

[0047] S5: Input the multi-cue embeddings from the detection end and the trajectory end into the cue-gated attention fusion module (CGAF), and perform dynamic gating fusion of different cue embeddings under spatial neighborhood constraints to obtain the fused detection embedding and the fused trajectory embedding; then, through CGAF, perform dynamic weighted fusion of the cue embeddings under spatial neighborhood constraints to obtain the fused embedding used for association.

[0048] S5-1: The multi-clue embeddings at the detection end and the multi-clue embeddings at the trajectory end are concatenated according to the object dimension to form a unified feature unit sequence.

[0049] S5-2: Calculate the set of object center points formed by the detection center point and the trajectory center point. Based on each object center point, select its spatial neighborhood k nearest neighbors and construct an attention mask, ensuring that the interaction of the attention mask is limited to the spatial neighborhood. In this embodiment, the distance to invalid objects is assigned to infinity, and the k nearest neighbors of each object are selected. For each neighboring object, a Boolean mask is generated to prevent attention between non-neighboring objects. When an object is invalid, its attention is allowed to degenerate to a valid object to avoid numerical anomalies.

[0050] When calculating the center point of an object based on the detection box, the center point is calculated from the top left corner and width and height of the bounding box; when calculating the center point of an object based on the trajectory position, the center point is calculated from the last valid bounding box in the historical trajectory. When there are no valid boxes in the historical trajectory, the center point can be degenerated into the zero vector or the center point of the predicted box.

[0051] The attention mask is a Boolean matrix, denoted as . The element in the i-th row and j-th column of a Boolean matrix is ​​denoted as , Indicates prohibited objects For objects Attention; the distance to invalid object pairs is set to infinity to avoid them being selected as nearest neighbors, when the number of objects is less than Time to take , Number of objects.

[0052] S5-3: Perform neighborhood self-attention interaction on the feature unit sequence under attention mask constraints to obtain a context fusion representation.

[0053] S5-4: Perform gated attention fusion on the context fusion representation in the multi-cue dimension to obtain the fusion representation. Then, divide the fusion representation into the fusion detection embedding and the fusion trajectory embedding according to the object dimension.

[0054] S5-4-1: Use the context fusion representation as the Q value, and use the multi-clue embedding at the detection end as the K and V values.

[0055] S5-4-2: Based on the compatibility of context fusion representation and multi-clue embedding at the detection end, output gate bias is used to obtain attention scores based on Q and K values ​​(in this embodiment, the attention score can be Q·K or...). (where Q represents the Q-value and K represents the K-value), the cue dimension score is obtained by adding the gating bias and the attention score; in this embodiment, the gating bias is obtained by gating a multilayer perceptron. The mapping process shows that the last layer parameter of the gating branch can be initialized to zero, making the initial gating bias zero and degenerating to be determined solely by the attention score. As training progresses, the gating bias is gradually learned as a cue reliability adjustment term to improve early training stability and degenerate the initial stage into an approximately uniform fusion.

[0056] S5-4-3: Perform a single softmax normalization on the cue dimension score to obtain cue weights. Use the cue weights to perform cue dimension weighted aggregation on the V value to obtain the fused representation. Before gating fusion, the embeddings of each cue can be linearly projected to a unified dimension, and the initial fused representation can be obtained by splicing the projection and normalization. After gating fusion, one or more layers of self-attention can be performed to refine the fusion result.

[0057] Neighborhood self-attention only allows spatially nearest neighbor interactions to highlight locally conflicting targets, while gating attention can reduce the contribution of unreliable cues and improve robustness in occluded or abrupt motion scenarios.

[0058] S6: Construct a trajectory-detection similarity matrix based on the fused detection embedding and the fused trajectory embedding, and perform matching based on the trajectory-detection similarity matrix to obtain the matching result between the trajectory and the detection.

[0059] The elements of the trajectory-detection similarity matrix are the cosine similarity or normalized inner product between the fused trajectory embedding and the fused detection embedding. A similarity threshold can be set to filter low-confidence matches, and category consistency constraints or size consistency constraints can be introduced to further narrow down the candidate set. Before matching, threshold constraints or distance gating constraints are applied to the trajectory-detection similarity matrix; elements that do not meet the distance gating constraints or threshold constraints are invalidated to eliminate unreasonable matches.

[0060] When matching based on the trajectory-detection similarity matrix, the Hungarian algorithm or a greedy strategy is used. The matching results include a set of successfully matched detection and trajectory pairs, a set of unmatched trajectories, and a set of unmatched detections.

[0061] S7: Update the target trajectory status and maintain the target identity ID based on the matching results, specifically: For successfully matched detections and trajectories, update the trajectory's position, appearance features, and historical trajectories based on the detection results, while maintaining the original ID. Truncate historical trajectories to maintain a fixed length. The system updates the trajectory age and the number of lost frames, and attenuates or enhances the trajectory confidence. For unsuccessful detections, a new trajectory is initialized and assigned a new ID. For unsuccessful trajectories, a hold or termination operation is performed based on the number of lost frames threshold. Prediction and loss count accumulation are performed; when the loss count exceeds the threshold, the trajectory is terminated and removed. New trajectories remain in an initialized state until a confirmation threshold is reached for consecutive matches, to reduce interference from short trajectories caused by false detections; the trajectory termination threshold can be configured based on the frame rate and scene occlusion level.

[0062] S8: Output multi-target tracking results. In this embodiment, outputting visualized multi-target tracking results includes drawing target bounding boxes, target IDs, and trajectory tails on the original video frames, encoding the rendered frame sequence into a video file and saving it, and exporting a tracking result table (including fields such as frame number, ID, bounding box, and confidence level) and running logs for reproduction and evaluation. It supports drawing different IDs under different color schemes, and the trajectory tail length is configurable; the exported results can conform to MOTCHalllenge format or a custom format.

[0063] This invention also discloses a multi-target tracking system based on dynamic cue gating and temporal enhancement, such as... Figure 3 As shown, it includes: The video input module acquires the target video and running parameters; The preprocessing module preprocesses the target video to obtain a frame-by-frame image sequence; The detection module performs target detection on the image sequence and outputs detection boxes; The feature extraction module extracts features based on the target detection results to obtain appearance cues and motion cues, forming a detection set, wherein each detection in the detection set includes appearance cues and motion cues; The trajectory management module acquires the target's historical trajectory and constructs a historical trajectory set, maintaining the trajectory status and historical cache. Each trajectory in the historical trajectory set includes appearance cues and motion cues. The TFEN encoding module performs temporal enhancement and sparse aggregation on appearance and motion cues in the detection set and appearance and motion cues in the historical trajectory set to obtain appearance embedding, motion embedding, appearance embedding and motion embedding at the detection end, trajectory end, and motion embedding at the trajectory end. The appearance embedding and motion embedding at the detection end are used as multi-cue embeddings at the detection end, and the appearance embedding and motion embedding at the trajectory end are used as multi-cue embeddings at the trajectory end. The CGAF fusion module performs dynamic gating fusion of the multi-cue embeddings at the detection end and the multi-cue embeddings at the trajectory end under spatial neighborhood constraints, to obtain the fused detection embedding and the fused trajectory embedding. The matching module constructs a trajectory-detection similarity matrix based on the fused detection embedding and the fused trajectory embedding, and performs matching based on the trajectory-detection similarity matrix to obtain the matching result between the trajectory and the detection. The interaction and configuration module updates the target trajectory status and maintains the target identity ID based on the matching results; The visualization and export module outputs multi-target tracking results.

[0064] In this embodiment, the visualization and export module includes a graphical user interface, which can improve project usability and user experience.

[0065] The graphical user interface includes at least the following: video selection controls, inference start controls, running status display controls, progress display controls, log display controls, and result playback controls. After selecting a video, the user starts inference. The interface displays the progress of the detection, feature extraction, trajectory association, and visualization generation stages in real time, and provides a quick playback entry for the result video after inference is complete. The inference engine supports loading running status from a status file and passing video paths and output parameters via configuration overwrite; inference logs can be written to a fixed location for reproduction; the inference output directory is organized by running time for managing results from multiple runs.

[0066] One implementation of the graphical user interface includes: displaying the system title, video input area, status and progress display area, scrolling log display area, start inference button, and play result button; parsing the output log and updating the stage status and progress in real time during inference; and automatically locating the result video in the output directory and providing a playback entry point after inference is completed.

[0067] The present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multi-target tracking method based on dynamic cue gating and timing enhancement.

[0068] The present invention also discloses an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-target tracking method based on dynamic cue gating and timing enhancement.

[0069] TFEN and CGAF are merely feasible implementations. Those skilled in the art can make various modifications, substitutions, or combinations to the multi-scale pooling strategy, saliency selection strategy, gating bias form, neighborhood construction strategy, matching strategy, trajectory state machine threshold, and visualization method without departing from the spirit of the present invention. All such modifications, substitutions, or combinations should fall within the protection scope of the present invention.

[0070] The description of the technical solution, optional implementation methods and beneficial effects is not intended to limit the scope of protection. Without departing from the spirit of the present invention, those skilled in the art can make equivalent substitutions or modifications to the clue composition, timing window length, network structure of TFEN and CGAF, matching strategy and threshold selection, visualization output format, etc., and all such substitutions or modifications should fall within the scope of protection of the present invention.

[0071] To make the implementation of this invention easier to understand, the functional modules and key processing flows are further described in detail below. The following sections are supplementary descriptions of the embodiments and do not constitute a limitation on the scope of protection of the claims.

[0072] Regarding the organization of video input and project operation, the system organizes the output using a project-level operation directory: each run creates an independent task output directory to save runtime configuration snapshots, logs, visualization videos, and intermediate results; the output directory allows for tracing input videos, parameters, and results, thereby meeting the needs of project reproduction and auditing.

[0073] The system supports loading runtime states from state files. State files are used to record the tracker's internal structured states, including trajectory cache, model parameter paths, or inference configurations. State files can be generated during offline preparation or the previous run to reduce initialization overhead and improve stability.

[0074] The system supports passing video path and status file path in configuration override mode. Configuration override mode is used to quickly switch video data and running parameters without modifying the main configuration file. This mode can be used for batch processing or graphical interface driven inference calls.

[0075] Regarding the detection module, it outputs a set of detection boxes for each frame and filters the detection boxes according to a confidence threshold. The threshold can be adaptively adjusted according to the scene density: the threshold is increased in densely populated scenes to reduce false detection interference, and the threshold is decreased in severely occluded scenes to reduce trajectory breaks caused by missed detections.

[0076] The detection module supports outputting category labels, allowing the system to filter or track targets of different categories separately. For example, in pedestrian tracking tasks, only the pedestrian category is retained, while in vehicle tracking tasks, both vehicle and cyclist categories are retained.

[0077] The detection module can output additional information such as the size, aspect ratio, and key points of the detection box. This additional information can be used as motion cues or quality assessment cues to input into subsequent modules to enhance robustness under occlusion and blur conditions.

[0078] The detection module can use batch inference: group several consecutive frames into a batch to improve GPU utilization; and maintain the frame sequence mapping during output to align with the trajectory management module.

[0079] Regarding the feature extraction module, each detection box is cropped and the cropped region is expanded by a fixed ratio to include the context; the expansion ratio can be... To cover neighborhood background information, additional discriminative cues are provided between similar-looking targets. Affine alignment, size normalization, and color normalization are performed on the cropped image patch, and the resulting image is input into an appearance feature extractor to output an appearance embedding. The appearance embedding can be performed... Normalization facilitates cosine similarity calculation.

[0080] The appearance feature extractor can be a lightweight convolutional network or a visual Transformer network; when deployment resources are limited, a smaller embedding dimension can be used to reduce computational cost; when occlusion and similar appearance issues are prominent, a stronger appearance feature extractor can be used to enhance discriminative ability.

[0081] The appearance feature extraction module can introduce a quality assessment branch and output an appearance reliability score. The appearance reliability score can be used to adjust the gating bias of appearance cues in CGAF, thereby reducing the weight of appearance cues when they are occluded or blurred.

[0082] Appearance cues can be temporally smoothed: for the same trajectory in consecutive frames, the appearance embedding is performed using exponential moving average or window averaging to suppress noise, and the smoothing coefficient can be adaptively adjusted according to the trajectory confidence and occlusion degree.

[0083] Motion features include at least the bounding box coordinates and its center point, and further include velocity and scale variations. Motion features can be input as "motion cues" into TFEN and CGAF.

[0084] Motion features are predicted by a Kalman filter: the trajectory state vector is updated over time to obtain the predicted bounding box, and corrections are performed using detection observations; the geometric consistency between the predicted bounding box and the detection bounding box is used for candidate gating to reduce unreasonable matching.

[0085] Candidate gating includes distance gating and scale gating: candidates are removed when the distance between the detected center point and the predicted center point exceeds a threshold, and candidates are removed when the aspect ratio difference is too large. The threshold can be normalized according to image resolution or adaptively adjusted according to trajectory speed.

[0086] The reliability score of motion cues can be jointly estimated by trajectory velocity, acceleration and prediction residual. When the trajectory undergoes abrupt motion or the prediction residual increases significantly, the weight of motion cues in CGAF is reduced, thereby avoiding motion cues from misleading associations.

[0087] Historical trajectory window length Can be taken Preferred ;when Smaller sizes can improve response speed but lack long-range information. Larger values ​​can enhance stability but may introduce more noise fragments, so sparse polymerization of TFEN can be combined to balance the two.

[0088] Invalid positions in the historical trajectory are marked by Boolean masks. TFEN applies a mask to invalid positions during self-attention and saliency scoring to prevent invalid features from participating in aggregation, and sets back to valid positions for fully invalid sequences when necessary to avoid numerical anomalies.

[0089] TFEN first performs two layers of nonlinear mapping on the input sequence and uses residual connections to enhance nonlinear representation and stabilize feature distribution. Then, it constructs multi-scale summarization feature units: it performs adaptive average pooling on the sequence in the temporal dimension to different lengths and superimposes scale encoding on the pooling results for each scale; at the same time, it superimposes position encoding according to the center index of the pooling window to approximately represent the temporal position corresponding to the pooling feature.

[0090] Multi-scale summarization feature units are concatenated with the original sequence to form a hybrid sequence, which is then input into several Transformer blocks for self-attention encoding. The Transformer blocks include multi-head self-attention and feedforward networks, and layer normalization and residual connections are used to stabilize training.

[0091] The scorer outputs a score for each feature unit and sets invalid locations to negative infinity; in hard selection mode, it selects the top-ranked feature units. Index set; in soft selection mode The weights are obtained by performing a softmax operation. The score is used to weight the feature units for approximate selection.

[0092] TFEN uses [CLS] feature unit aggregation: [CLS] feature units are concatenated with key feature units and input into the aggregation attention module, and the output [CLS] vector is used as the final embedding; this embedding is then projected onto the output dimension through linear mapping so that it can be connected with subsequent modules.

[0093] Retention ratio Annealing can be performed with each training round: In the early stages of training near By fully utilizing the context, the sparsity is gradually reduced to the target sparse ratio in the later stages of training to focus on key segments. This strategy can reduce early training instability and improve final robustness.

[0094] TFEN uses a shared structure but different parameters for different cues (appearance and motion), or uses shared parameters but different input projections; this invention is not limited to these methods. When the statistical differences in cue features are large, using independent parameters can improve the fitting ability.

[0095] CGAF first projects each cue embedding onto a unified dimension through a linear layer, then concatenates the projection results according to the cue dimension, maps them back to the unified dimension through a fusion projection layer, and performs normalization to obtain an initial fusion representation. The initial fusion representation serves as the basic context for subsequent neighborhood self-attention and gating fusion.

[0096] Neighborhood self-attention adopts space Nearest neighbor masking limits the attention span: For each object, calculate the distance to the center point of other objects and select... The nearest neighbor; attention is focused only within the neighborhood, thus highlighting locally conflicting targets and suppressing globally irrelevant noise.

[0097] The context representation obtained through neighborhood interactions and the multi-threaded embedding representations corresponding to each object are used together as input for gating attention fusion. Specifically, the context representation is linearly mapped to obtain the query vector, and each thread embedding representation is linearly mapped to obtain the corresponding key vector and value vector. For the ... For each clue, first calculate the clue dimension score: , For the first Each clue is scored. No. The query vector corresponding to each clue. For the first The key vector corresponding to each clue For vector dimensions, The gating branch is represented by the context and is related to the first branch. The gated bias generated by concatenating the embeddings of each clue is used to reflect the degree of fit of the clue to the target object under the current context. Subsequently, the scoring corresponding to all clues is performed once on the single clue dimension. Normalization yields the weights of each clue, denoted as... ,Right now Then, based on the clue weights, the value vectors of each clue are weighted and aggregated to obtain the fused representation: ,in, For fusion representation, For the first The value vectors corresponding to each clue. The fusion representation is used to characterize the comprehensive multi-clue features of an object under the constraints of the current spatial neighborhood and the influence of contextual information, and serves as the basic representation for subsequent feature refinement and trajectory-detection similarity calculation.

[0098] Gated branch outputs can be broadcast along multiple heads, allowing different attention heads to share the same cue gating bias or use different biases. Sharing biases can reduce the number of parameters and improve stability, while not sharing biases can improve expressiveness.

[0099] After gating fusion, CGAF performs several layers of self-attention refinement: during the refinement stage, it can choose to continue using neighborhood masks or adopt global attention. When the number of scene targets is large and neighborhood conflicts dominate, neighborhood attention is preferred. When global consistency constraints are required, global attention can be used.

[0100] CGAF in calculation When using nearest neighbor masking, invalid objects are handled as follows: if an object is invalid, its attention is allowed to degenerate to a valid object to avoid softmax anomalies caused by all negative infinity, and the output is cleaned up with nan to improve numerical robustness.

[0101] The trajectory-detection similarity matrix is ​​calculated from the cosine similarity of the fused embedding, and elements that do not meet the distance gate are invalidated; a similarity threshold is used during matching. Filter low-confidence matching pairs to reduce false associations.

[0102] threshold It can be a fixed value, or it can be adaptively calculated based on the number of targets in the current frame, the degree of occlusion, and the historical matching distribution. Adaptive calculation can determine the threshold by statistically analyzing the similarity distribution (mean, variance, quantiles) to adapt to different scenarios.

[0103] Matching algorithms include the Hungarian algorithm, greedy strategies, or hierarchical matching. Hierarchical matching first uses strict gating and high thresholds to complete high-confidence matching, and then uses more relaxed gating and more complex clues to complete supplementary matching for the remaining targets.

[0104] The trajectory object includes: unique trajectory ID, current status, last matching time, historical appearance sequence, historical movement sequence, trajectory confidence, trajectory age, and loss count; the trajectory management module maintains a global ID pool to ensure that new trajectories obtain unused IDs.

[0105] Trajectory initialization includes: assigning a temporary ID to the new detection and entering the Tentative state; when the trajectory is continuous If a match is successful within a frame, the frame is converted to Confirmed; if a continuous non-match exceeds the threshold during the initialization phase, the frame is removed to reduce short-track interference caused by false detections. The value should be adjusted according to the actual situation.

[0106] Trajectory loss retention includes: when the number of consecutive unmatched frames does not exceed a certain threshold. The frame remains Lost until it is occluded and reappears; motion prediction can still be updated and historical trajectories maintained during the Lost phase; when the number of unmatched frames exceeds Then terminate and release the resources. The value should be adjusted according to the actual situation.

[0107] Trajectory updates involve truncation and alignment of historical sequences: the latest appearance and motion features are written to the end of the sequence, and for sequences exceeding a certain length... The sequence is discarded from the beginning of the sequence, and the mask is updated synchronously to ensure the consistency of sequence validity.

[0108] The visualization and export module assigns colors to targets by ID and draws bounding boxes and ID text on video frames. Track wakes can be generated by recording a sequence of historical center points and drawing a polyline; the wake length can be configured to balance readability and information content.

[0109] The export results include: exporting the resulting video, exporting frame-by-frame result text, exporting intermediate state files, and exporting the runtime log. The runtime log includes information on key stages and exceptions to help locate errors and evaluate performance.

[0110] The output directory is organized by date and time, and each directory contains at least a subdirectory of visual videos. The filenames of the resulting videos can be fixed or associated with the input video names for batch processing.

[0111] The graphical interface includes: system title display, video input path display, file browsing button, start reasoning button, playback result button, status label, progress label and scrolling log window, and supports centering of the window and button status linkage.

[0112] The graphical interface starts the inference process in a background thread and reads the inference output in real time. It identifies stages such as "object detection", "feature extraction", "trajectory association" and "visualization generation" by parsing keywords in the output text, and updates the current status and progress percentage on the interface to avoid interface blocking.

[0113] The graphical interface detects GPU availability and displays device information at the start of inference. If the GPU is unavailable, it automatically reverts to CPU execution and displays a message in the log to improve availability under different hardware environments.

[0114] After the reasoning is complete, the graphical interface automatically searches for the result video file in the output directory. If a default filename exists, it is used directly; otherwise, any MP4 file is searched as the result. The "Play Result Video" button is available for quick playback.

[0115] Regarding configuration management and reproducibility, the system uses a configuration management framework to organize dataset paths, model hyperparameters, training strategies, and inference options, and saves configuration snapshots to the output directory each time it runs, so that different experimental settings can be recorded and reproduced.

[0116] The system supports selecting different pipeline modules through configuration overlay, such as selecting to execute only the tracing module or add an evaluation module; and supports setting the number of cores and whether to enable rich text logging to adapt to different deployment environments.

[0117] Regarding training and evaluation, the system can be trained and validated on public multi-target tracking datasets, and the tracking performance can be evaluated using metrics such as IDF1, HOTA, AssA, and MOTA; among them, IDF1 focuses on identity consistency and is suitable for evaluating the robustness of associated modules.

[0118] During the inference phase, the system can use detection results and appearance features as input and only run on the correlation module. This approach can be used to ablate and evaluate the contribution of the correlation module under fixed detector conditions.

[0119] The above description of the optional implementation methods of each functional module and key processing flow is only used to explain the technical solution and engineering implementation ideas of the present invention. Those skilled in the art can make equivalent substitutions or modifications to the detector, appearance feature extractor, motion model, TFEN and CGAF structure, gating and neighborhood strategy, matching strategy, trajectory state machine threshold, and interface interaction method without departing from the spirit of the present invention, and all such substitutions or modifications should fall within the protection scope of the present invention.

[0120] This invention addresses several technical problems in existing technologies, including the difficulty in adaptively adjusting the reliability of multi-threaded tracking as it changes with time and scenario, insufficient modeling of local neighborhood conflicts, easy propagation of historical trajectory noise fragments, and incomplete engineering deployment and output. Compared with existing technologies, this invention has the following advantages: 1. This invention, within a detection-based tracking paradigm, obtains temporal sequences of appearance and motion cues by acquiring a detection set and constructing historical trajectories. Multi-scale modeling is then performed using a temporal feature enhancement network, followed by sparse aggregation through saliency filtering to obtain multi-cue embeddings at both the detection and trajectory levels. Furthermore, TFEN is used for multi-scale context modeling in the temporal dimension, and highly saliency segments are sparsely aggregated to suppress the propagation of noisy historical segments and enhance short-term discriminative information.

[0121] 2. The multi-cue embedding at the detection end and the multi-cue embedding input cue gating attention fusion module at the trajectory end are combined. Under spatial neighborhood constraints, a gating bias is introduced to achieve dynamic weighted fusion of cue dimensions, resulting in fused detection embeddings and fused trajectory embeddings. By introducing cue-dimensional gating attention under spatial neighborhood constraints through CGAF, dynamic weighted fusion of the reliability of cues such as appearance and motion is achieved.

[0122] 3. Based on the fused detection embedding and the fused trajectory embedding, a trajectory-detection similarity matrix for association is obtained and matched. The target trajectory state is updated and the target identity ID is maintained based on the matching results to obtain multi-target tracking results. This can improve the robustness of association and identity consistency in complex scenarios such as frequent occlusion, nonlinear motion, or appearance drift, thereby improving the accuracy of target tracking.

[0123] 4. This invention provides a deployable, systematic implementation method that supports video import, inference monitoring, result visualization and export, and has good engineering application value.

[0124] To further demonstrate the beneficial effects of this invention, multi-object tracking simulation experiments were conducted on the DanceTrack dataset using both the method of this invention and the existing DiffMOT method (see the paper "Lv W, Huang YH, Zhang N, et al. DiffMOT: A real-time diffusion-based multiple object tracker with non-linear prediction[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2024: 18656-18665."). The IDF1, HOTA, and AssA scores obtained by this invention were 67.0, 62.8, and 48.7, respectively. Compared to the DiffMOT method, this invention improves IDF1 by 4.0 points, HOTA by 0.5 points, and AssA by 1.5 points. The simulation experiments show that this invention can adaptively adjust the contributions of different cues, suppress the influence of noisy historical fragments, and improve the robustness of association and identity consistency under crowded interaction conditions, thus demonstrating the beneficial effects of this invention.

[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-target tracking method based on dynamic cue gating and temporal enhancement, characterized in that, include: Acquire the target video and preprocess it to obtain a frame-by-frame image sequence. The image sequence is subjected to target detection and feature extraction to form a detection set, the target historical trajectory is obtained and a historical trajectory set is constructed, and each detection in the detection set and each trajectory in the historical trajectory set includes appearance cues and motion cues; The appearance cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the appearance embedding of the detection end. The motion cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the motion embedding of the detection end. The appearance embedding and motion embedding of the detection end are used as the multi-cue embedding of the detection end. The appearance cues in the historical trajectory set are input into a temporal feature enhancement network for encoding to obtain the trajectory-end appearance embedding. The motion cues in the historical trajectory set are input into a temporal feature enhancement network for encoding to obtain the trajectory-end motion embedding. The trajectory-end appearance embedding and the trajectory-end motion embedding are used as the trajectory-end multi-cue embedding. The multi-clue embeddings at the detection end and the multi-clue embeddings at the trajectory end are input into the cue-gated attention fusion module. Under the spatial neighborhood constraint, different cue embeddings are dynamically gated and fused to obtain the fused detection embedding and the fused trajectory embedding. A trajectory-detection similarity matrix is ​​constructed based on the fused detection embedding and the fused trajectory embedding. The trajectory-detection similarity matrix is ​​then used to perform matching to obtain the matching result between the trajectory and the detection. Update the target trajectory status and maintain the target identity ID based on the matching results, and output the multi-target tracking results.

2. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 1, characterized in that: The appearance cues in the detection set are input into a temporal feature enhancement network for encoding to obtain the appearance embedding at the detection end, including: The input appearance cues are normalized and nonlinearly mapped to obtain basic features. The basic features are then multi-scale pooled to generate multi-scale summary feature units. The multi-scale summary feature units are superimposed with position encoding and scale encoding and then concatenated with the appearance cues to form a hybrid sequence. Temporal self-attention encoding is performed on the hybrid sequence, the saliency of the feature units in the encoded hybrid sequence is scored, and a set of key feature units is obtained by filtering. The [CLS] feature units are concatenated with the set of key feature units and then attention aggregation is performed to obtain the [CLS] aggregated embedding. Baseline features are extracted based on the input appearance cues. A gating vector is learned based on the baseline features. The [CLS] aggregate embedding and the baseline features are weighted and fused using the gating vector to obtain the appearance embedding of the detection end.

3. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 2, characterized in that: The detection end is embedded in the following form: ， In the formula, For embedding in the appearance of the detection end, Represents a linear mapping. For the gated vector, [CLS] aggregation embedding, Baseline features This indicates element-wise multiplication.

4. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 2, characterized in that: The process of scoring the saliency of feature units in the encoded mixed sequence and filtering them to obtain a set of key feature units includes: Each feature unit is scored for saliency using a scorer to obtain a scalar score. Invalid locations are masked during the saliency scoring process. Through Top- Hard selection or soft selection during the training phase filters scalar scores, and the feature units corresponding to the filtered scalar scores are used as the set of key feature units.

5. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 4, characterized in that: The above through Top- The scalar score for hard selection or soft selection during the training phase is as follows: When using Top- When using hard selection to filter scalar scores , For the number of key segments, To retain the proportion, The length of the mixed sequence, To round up; When using soft selection to filter scalar scores during the training phase, a softmax operation with temperature scaling is performed on the scalar scores to obtain a weight vector, and the feature units are weighted according to the weight vector; during temperature scaling, the temperature anneals as the training process progresses.

6. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 1, characterized in that: The multi-cue embeddings from the detection end and the trajectory end are input into the cue-gated attention fusion module. Under spatial neighborhood constraints, different cue embeddings are dynamically gated and fused to obtain the fused detection embedding and the fused trajectory embedding, including: The multi-clue embeddings at the detection end and the multi-clue embeddings at the trajectory end are concatenated according to the object dimension to form a unified feature unit sequence; Calculate the set of object center points formed by the detection center point and the trajectory center point, and select the k nearest neighbors in the spatial neighborhood for each object center point to construct an attention mask; Under the attention mask constraint, neighborhood self-attention interaction is performed on the feature unit sequence to obtain a context fusion representation; Gated attention fusion is performed on the context fusion representation in the multi-cue dimension to obtain the fusion representation. The fusion representation is then divided into fused detection embedding and fused trajectory embedding according to the object dimension.

7. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 6, characterized in that: The process of performing gated attention fusion on the context fusion representation across multiple cue dimensions to obtain the fused representation specifically involves: The context fusion representation is used as the Q value, and the multi-clue embedding at the detection end is used as the K value and V value; Based on the compatibility of the context fusion representation with the multi-clue embedding of the detection end, the output gate bias is used to obtain the attention score according to the Q value and K value. The gate bias and the attention score are added together to obtain the cue dimension score. The clue dimension score is subjected to a single softmax normalization to obtain the clue weight. The clue weight is then used to perform clue dimension weighted aggregation on the V value to obtain the fused representation.

8. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 1, characterized in that: The elements of the trajectory-detection similarity matrix are the cosine similarity or normalized inner product between the fused trajectory embedding and the fused detection embedding. Before matching, a threshold constraint or distance gating constraint is applied to the trajectory-detection similarity matrix to eliminate unreasonable ones.

9. The multi-target tracking method based on dynamic cue gating and temporal enhancement according to claim 1, characterized in that: When matching based on the trajectory-detection similarity matrix, the Hungarian algorithm or a greedy strategy is used.

10. A multi-target tracking system based on dynamic cue gating and temporal enhancement, characterized in that, include: The video input module acquires the target video. The preprocessing module preprocesses the target video to obtain a frame-by-frame image sequence; The detection module performs target detection on the image sequence and outputs detection boxes; The feature extraction module extracts features based on the target detection results to obtain appearance cues and motion cues, forming a detection set, wherein each detection in the detection set includes appearance cues and motion cues; The trajectory management module acquires the target's historical trajectory and constructs a historical trajectory set, maintaining trajectory status and historical cache. Each trajectory in the historical trajectory set includes appearance cues and motion cues. The TFEN encoding module performs temporal enhancement and sparse aggregation on appearance cues and motion cues in the detection set and appearance cues and motion cues in the historical trajectory set to obtain detection end appearance embedding, detection end motion embedding, trajectory end appearance embedding and trajectory end motion embedding. The detection end appearance embedding and detection end motion embedding are used as multi-cue embeddings of the detection end, and the trajectory end appearance embedding and trajectory end motion embedding are used as multi-cue embeddings of the trajectory end. The CGAF fusion module performs dynamic gating fusion of the multi-cue embeddings at the detection end and the multi-cue embeddings at the trajectory end under spatial neighborhood constraints, to obtain the fused detection embedding and the fused trajectory embedding. The matching module constructs a trajectory-detection similarity matrix based on the fused detection embedding and the fused trajectory embedding, and performs matching based on the trajectory-detection similarity matrix to obtain the matching result between the trajectory and the detection. The interaction and configuration module updates the target trajectory status and maintains the target identity ID based on the matching results; The visualization and export module outputs multi-target tracking results.

Images