A robust multi-target tracking method based on high recall and strong temporal association

Abstract

The application discloses a robust multi-target tracking method based on high recall and strong time sequence correlation, comprising the following steps: extracting shared feature maps of a current video frame through a backbone network and an encoder; a main detection branch generates a high-confidence candidate set, and an auxiliary detection branch generates a potential candidate set through splicing a noise query; after performing multi-dimensional consistency filtering on the potential candidate to remove noise and redundancy, the current frame target query is formed together with the main branch candidate; inputting the current frame target query into a spatial attention module to realize intra-frame feature enhancement; inputting the enhanced candidate as a key and a value, and a historical trajectory query as a query item into a time span attention module, updating the historical trajectory representation according to the attention weight, and completing identity correlation; according to the correlation result, the identity of the candidate is inherited, a new trajectory is initialized or a mismatched trajectory is terminated, and a tracking result is output. The application improves the recall rate and cross-frame correlation stability in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of video target tracking technology, and in particular to a robust multi-target tracking method based on high recall and strong temporal correlation. Background Technology

[0002] Multi-object tracking is a core task in computer vision aimed at providing continuous localization and maintaining consistent identity associations for targets of interest in video sequences. It is widely used in real-world scenarios such as transportation systems, video surveillance, autonomous driving, and motion analysis. With the exponential growth of video data, improving the performance of multi-object tracking methods in complex environments has become a crucial research topic. Current multi-object tracking methods functionally consist of two core aspects: candidate target identification and historical trajectory association. The former provides a set of potential trackable targets, while the latter assigns correct identity representations to candidate targets based on current and historical information, ultimately generating a complete trajectory for each target. However, in complex scenarios such as dense target distribution, rapid movement, prolonged occlusion, and drastic changes in appearance, existing methods still face significant performance challenges.

[0003] Existing multi-target tracking methods mainly follow two technical routes. The first is a two-stage tracking paradigm based on detection and re-association. This paradigm relies on high-performance target detectors to provide a candidate target set and uses linear motion models such as Kalman filtering to predict the target's future trajectory. Subsequent work has introduced advanced cues such as deep appearance embedding and pseudo-depth information to further alleviate the identity switching problem. The second is an end-to-end tracking paradigm based on attention mechanisms. This paradigm uses query representation to simultaneously perform target detection and tracking. It simulates object continuity through trajectory queries that maintain identity in TrackFormer, handles target appearance and disappearance through the evolutionary dynamic tracking query in the MOTR series, and achieves temporal association through long-term memory modeling mechanisms in MeMOTR. This type of method naturally completes cross-frame matching of targets and trajectories using temporal attention mechanisms.

[0004] However, the aforementioned existing technologies still have two problems: First, the single-branch detection structure leads to insufficient recall. Existing methods usually rely on a single detection branch to output high-confidence target candidates to ensure accuracy. However, in scenarios with dense targets, a high proportion of small targets, or occlusion, it is easy to overlook a large number of potential effective targets. Moreover, in detection-tracking interaction methods, the detection capability gradually weakens during the optimization process, causing some real targets to be completely missing in the association stage. Second, the unstable historical trajectory update leads to weak association capability. Detection-based methods rely on linear modeling and can only handle simple uniform motion. Query-based methods rely on the inference module to predict future trajectories, and the update performance is unstable. When the target is occluded for a long time or moves quickly, the historical trajectory information modeling is not accurate enough. In dense scenes, mismatches occur frequently, eventually leading to identity switching and trajectory interruption, which seriously affects the stability and reliability of the tracking results. Summary of the Invention

[0005] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a robust multi-target tracking method based on high recall and strong temporal correlation.

[0006] The technical solution adopted in this invention is a robust multi-target tracking method based on high recall and strong temporal correlation, comprising the following steps: S1, extracting shared feature maps from the current video image frame via a backbone network and encoder; S2, the main detection branch generates a high-confidence candidate set through the main target query via the main branch self-attention module, the shared cross-attention module, and the main branch feedforward network; the auxiliary detection branch concatenates the auxiliary target query with the noise query and generates a potential candidate set via the auxiliary branch self-attention module, the shared cross-attention module, and the auxiliary branch feedforward network; S3, performing multi-dimensional consistency filtering on the potential candidate set to remove noise and redundancy, and combining the filtered auxiliary candidates with the high-confidence... The candidate set together forms the target query for the current frame; S4, the target query for the current frame is input into the spatial attention module, and spatial features are enhanced through self-attention interaction between candidate targets within the same frame; S5, the spatially enhanced current frame candidate is used as a key and value, and the historical trajectory query is used as a query item and input into the time-cross attention module, and the current frame observation information is aggregated according to the attention weight to update the historical trajectory representation and complete the cross-frame identity association; S6, according to the association result, the historical trajectory identity identifier is inherited for the successfully matched current frame candidate, a new trajectory is initialized for the unmatched current frame candidate that meets the conditions, the unmatched historical trajectory is retained or terminated according to the continuous mismatch duration, and the multi-target tracking result of the current frame is output.

[0007] Furthermore, in S2, when the auxiliary detection branch concatenates the auxiliary target query and the noise query and processes them through the auxiliary branch self-attention module, it follows the... Construct an enhanced query set, in which This indicates the auxiliary target query. To assist in querying the number of queries, For feature dimension, This indicates the injected noise query. This represents the enhanced query set obtained after concatenation; after self-attention calculation, the enhanced query set discards the noisy query part and retains the auxiliary query with attention bias for cross-attention interaction with the encoder output.

[0008] Furthermore, the multi-dimensional consistency filtering in S3 includes appearance consistency verification for each candidate object in the potential candidate set. Extracting feature representations from encoder output and neighboring space Cosine similarity is calculated for targets with medium to high confidence levels, and appearance consistency scores are determined according to... Confirmed, among which Indicates candidate objects The center point is the spatial neighborhood within the preset pixel radius of the circle. This is the high-confidence candidate set. For the neighboring region Feature representation of a high-confidence target.

[0009] Furthermore, the multi-dimensional consistency filtering in S3 includes geometric consistency verification; for the detection targets in the potential candidate set, a physical representation is obtained through a multilayer perceptron. ,in Indicates the coordinates of the bounding box center. Indicates width, Indicates height; geometric consistency score is based on... Confirmed, among which and These are the lower and upper bounds of the effective aspect ratio. and These are the lower and upper bounds of the relative area. and These are the width and height of the current video image frame, respectively. This is an indicator function.

[0010] Furthermore, the multi-dimensional consistency filtering in S3 includes candidate consistency verification, for each auxiliary candidate object in the potential candidate set. Extract its feature representation And calculate the cross-attention score with all main branch candidates in the high-confidence candidate set, based on The normalized weights are obtained, where This represents the function for calculating feature similarity. and These are the main branch candidate objects in the high-confidence candidate set. The cardinality of the high-confidence candidate set; candidate consistency score based on Determine that only auxiliary candidates with a candidate consistency score higher than the preset threshold will be retained.

[0011] Furthermore, the main detection branch and the auxiliary detection branch adopt asymmetric parameter configuration, the main branch self-attention module and the auxiliary branch self-attention module are independent network parameters, the main branch feedforward network and the auxiliary branch feedforward network are independent feedforward parameters, and the weight parameters of the shared cross-attention module are shared between the two branches; the spatial attention module adopts a multi-head self-attention mechanism with relative position encoding, the relative position encoding is calculated based on the relative displacement between the center coordinates of the candidate target bounding box; the temporal cross-attention module adopts a multi-head cross-attention mechanism, wherein the number of heads is set as a trainable hyperparameter; historical trajectory query is terminated when the continuous mismatch duration exceeds the preset trajectory survival threshold, and is temporarily retained when it is below the trajectory survival threshold, waiting for re-association in the next time frame.

[0012] Further, in S2, the main detection branch generates a high-confidence candidate set through the main target query via the main branch self-attention module, the shared cross-attention module, and the main branch feedforward network, including the following sub-steps: S211, the main target query generates a main query vector and a primary key vector through linear projection, and outputs the main branch enhanced query by capturing the dependency relationship between queries through self-attention calculation; S212, the main branch enhanced query interacts with the shared feature map as a query item through the shared cross-attention module, and aggregates visual information from the feature map according to the attention distribution to generate the main branch decoding feature; S213, the main branch decoding feature is nonlinearly transformed by the main branch feedforward network to generate the main branch output feature; S214, the main branch output feature predicts the target category probability and bounding box parameters through the classification branch and the regression branch respectively, and filters the predicted boxes with confidence higher than a preset threshold to form the high-confidence candidate set.

[0013] Further, the auxiliary detection branch in S2, which concatenates the auxiliary target query and the noise query, generates a potential candidate set via the auxiliary branch self-attention module, the shared cross-attention module, and the auxiliary branch feedforward network, includes the following sub-steps: S221, the auxiliary target query and the noise query are concatenated and linearly projected to generate an auxiliary query vector and an auxiliary key vector. Self-attention calculation is used to guide the auxiliary query towards the low-confidence region using the noise query as an attention anchor, outputting an auxiliary enhancement query; S222, after discarding the noise query portion of the auxiliary enhancement query, the remaining auxiliary query is used as a query item and interacts with the shared feature map via the shared cross-attention module to generate auxiliary decoding features; S223, the auxiliary decoding features undergo nonlinear transformation via the auxiliary branch feedforward network to generate auxiliary output features; S224, the auxiliary output features predict the target category probability and bounding box parameters via the classification branch and regression branch respectively, extracting all predicted boxes to form the potential candidate set.

[0014] Further, step S4 includes the following sub-steps: S41, obtaining a spatial query vector and a spatial key vector from the current frame target query through a learnable linear projection layer, and calculating a relative position encoding bias term based on the bounding box center coordinates of each candidate target; S42, superimposing the relative position encoding bias term onto the dot product result of the spatial query vector and the spatial key vector, and obtaining a spatial attention weight matrix after scaling and normalization; S43, performing weighted aggregation on the value vector corresponding to the current frame target query according to the spatial attention weight matrix to generate intermediate features that fuse neighborhood spatial layout information; S44, adding the intermediate features to the original current frame target query through a residual connection to obtain the spatially enhanced candidate feature representation of the current frame.

[0015] Further, step S5 includes the following sub-steps: S51, obtaining a time query vector by linear projection of the historical trajectory query, and obtaining a time key vector and a time value vector by linear projection of the spatially enhanced current frame candidate; S52, calculating and scaling the dot product between the time query vector and the time key vector, and obtaining a time attention weight matrix by normalizing the exponential function, where each element represents the degree of matching between the historical trajectory and the current frame candidate; S53, performing weighted aggregation of the time value vector according to the time attention weight matrix to obtain the updated trajectory features fused with the current frame observation information; S54, selecting the current frame candidate with the largest attention weight for each historical trajectory from the time attention weight matrix as the association matching result, and dynamically updating the motion state and appearance representation of the historical trajectory according to the updated trajectory features.

[0016] Beneficial Effects: This invention proposes a robust multi-target tracking method based on high recall and strong temporal correlation. By constructing a dual-branch recall structure combining a main detection branch and an auxiliary detection branch, it solves the deficiency of insufficient recall in complex scenes of existing single-branch detection structures. The main detection branch ensures accurate localization and high-confidence candidate output, while the auxiliary detection branch expands the detection search space through differentiated query generation and noise injection mechanisms, specifically mining low-confidence, occluded, and motion-blurred targets that are easily missed by the main branch. The combination of the two significantly improves the coverage of potential real targets. At the same time, a multi-dimensional consistency filtering mechanism is introduced to jointly screen auxiliary candidates from multiple dimensions such as appearance features, geometric attributes, and cross-branch semantic consistency, ensuring that noise and redundant candidates are controlled within an acceptable range while improving recall, thus structurally solving the tracking failure problem caused by missed detections. This method constructs a Transformer-based spatiotemporal joint attention association mechanism. The spatial attention module characterizes the relative layout relationships in dense scenes through self-attention interactions between candidate targets within the same frame, enhancing the distinguishability between similar-looking targets. The temporal cross-attention module performs cross-attention calculation by using historical trajectory queries as query terms and spatially enhanced current frame candidates as keys and values. Based on the attention weights, it dynamically aggregates current frame observation information to update the historical trajectory representation. This addresses the instability of association in traditional linear motion models and simple appearance matching when targets are occluded for extended periods, move rapidly, or undergo drastic appearance changes. It enables trajectory queries to adaptively adjust state representations, maintaining high identity consistency in occlusion recovery and dense interaction scenarios. Ultimately, this method significantly reduces the false negative rate while effectively suppressing the probability of identity switching and trajectory interruption. It maintains relatively stable tracking performance in complex scenes with dense targets, rapid movement, motion blur, and long-term occlusion, improving the robustness and reliability of multi-target tracking systems. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall steps of the method of the present invention; Figure 2 This is a schematic diagram of the multi-dimensional consistency filtering algorithm of the present invention; Figure 3 This is a schematic diagram of the spatiotemporal attention target association algorithm of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] A robust multi-target tracking method based on high recall and strong temporal correlation includes the following steps: S1, extracting shared feature maps from the current video image frame via a backbone network and encoder; S2, the main detection branch generates a high-confidence candidate set through a main target query via a main branch self-attention module, a shared cross-attention module, and a main branch feedforward network; the auxiliary detection branch concatenates the auxiliary target query with a noise query and generates a potential candidate set via an auxiliary branch self-attention module, the shared cross-attention module, and the auxiliary branch feedforward network; S3, performing multi-dimensional consistency filtering on the potential candidate set to remove noise and redundancy, and combining the filtered auxiliary candidates with the high-confidence candidate set to form a... S4. The current frame target query is generated; the current frame target query is input into the spatial attention module, and spatial feature enhancement is performed through self-attention interaction between candidate targets in the same frame; S5. The spatially enhanced current frame candidate is used as a key and value, and the historical trajectory query is used as a query item and input into the time cross-attention module. The current frame observation information is aggregated according to the attention weight to update the historical trajectory representation and complete the cross-frame identity association; S6. According to the association result, the historical trajectory identity identifier is inherited for the successfully matched current frame candidate, a new trajectory is initialized for the unmatched current frame candidate that meets the conditions, the unmatched historical trajectory is retained or terminated according to the continuous mismatch duration, and the current frame multi-target tracking result is output.

[0020] like Figure 1As shown in the flowchart, the overall steps of the method of this invention include six core steps connected sequentially. The top of the diagram is the input stage, labeled as the current video image frame. The arrow below it points to the first step S1. The box in this step indicates that the backbone network and encoder extract shared feature maps, meaning that the input video frame is subjected to multi-stage downsampling feature extraction through the backbone network with a residual network structure, and then context modeling is performed through a multi-layer Transformer encoder. The final output is a shared feature map with 2048 channels, which serves as the common input for subsequent dual-branch detection. Below S1, there are two parallel branches, left and right. The left branch is the main detection branch path, and the right branch is the auxiliary detection branch path, which together constitute the second step S2. The main detection branch path passes through the main target query, the main branch self-attention module, the shared cross-attention module, and the main branch feedforward network in sequence, and finally outputs a high-confidence candidate set. The auxiliary detection branch path starts from the concatenation of the auxiliary target query and the noise query, and passes through the auxiliary branch self-attention module, the same shared cross-attention module, and the auxiliary branch feedforward network in sequence, and outputs a potential candidate set. The parameter sharing relationship of the shared cross-attention module between the two branches is marked by a dashed box in the diagram. Below S2 is the third step, S3, marked with a box indicating multi-dimensional consistency filtering. Its input is the potential candidate set output by the auxiliary branch. After joint filtering based on appearance, geometry, and candidate dimensions, the retained auxiliary candidates are merged with the high-confidence candidates from the main branch to form the target query for the current frame. Below S3 are the fourth step, S4, spatial attention module; the fifth step, S5, temporal cross-attention module; and the sixth step, S6, trajectory management and output. S4 enhances the intra-frame spatial features of the target query for the current frame. S5 performs cross-frame attention interaction between the enhanced candidates and historical trajectory queries to complete identity association. S6 performs identity inheritance, trajectory initialization, and termination operations based on the association results, finally outputting the multi-target tracking result for the current frame at the bottom of the graph. The overall flowchart clearly indicates the data flow and dependencies between modules through arrow directions, intuitively presenting the technical architecture of this invention that combines dual-branch recall and spatiotemporal attention association.

[0021] When step S1 is implemented, the current video image frame is input into the backbone network. The backbone network adopts a residual network structure with 101 layers. The initial convolutional kernel size is 7x7 and the stride is 2. Subsequently, it goes through 4 stages of downsampling operations. Each stage includes 3, 4, 23 and 3 residual blocks respectively. The size of the output feature map decreases to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image respectively. Finally, the 1 / 32 size feature map output by the 4th stage is selected as the shared feature map. The spatial resolution of the shared feature map is about 1 / 32 of the side length of the input image, and the number of channels is expanded to 2048 dimensions. The encoder part adopts a 6-layer Transformer encoder structure. Each layer includes a multi-head self-attention module and a feedforward network module. The number of self-attention heads is set to 8, and the hidden layer dimension of the feedforward network is set to 4096. The encoder performs further feature transformation and context modeling on the shared feature map output by the backbone network. The feature stability is maintained through inter-layer residual connections and layer normalization operations. Finally, the encoder-enhanced shared feature map is output for use by the subsequent main detection branch and auxiliary detection branch.

[0022] When step S2 is implemented, the main detection branch first initializes a set of main target queries, the number of which is set to 300, and the vector dimension of each query is 256. This set of queries is modeled for the dependency relationship between queries through the main branch self-attention module, the number of self-attention heads is set to 8, and then input to the shared cross-attention module. Attention is calculated using the main target query as the query item and the shared feature map output by the encoder as the key and value. The main branch decoded features are output and finally nonlinearly transformed through the main branch feedforward network. Its hidden layer dimension is set to 1024, and the main branch output features are obtained. The auxiliary detection branch initializes 150 auxiliary target queries with a vector dimension of 256 and generates a noise query vector. Each element of this noise query is randomly sampled from a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. The auxiliary target queries and this noise query are concatenated along the feature dimension to obtain an enhanced query set of 151. After processing by the auxiliary branch's self-attention module, the noise query portion is discarded, retaining 150 auxiliary queries with attention bias. These are then processed by the shared cross-attention module and the auxiliary branch feedforward network to obtain the auxiliary branch output features. The concatenation of the auxiliary target queries and the noise queries is then processed by the auxiliary branch's self-attention module, making the noise query an attention anchor point in the calculation of inter-query similarity and weight normalization. This guides the attention distribution of the auxiliary target queries to shift towards low-confidence feature regions. After discarding the noise query portion, this attention bias is embedded in the retained auxiliary query feature representation, providing a query representation biased towards low-confidence regions for subsequent cross-attention interactions with the shared feature map, thereby uncovering potential targets that the main detection branch may easily miss.

[0023] In step S3, multi-dimensional consistency filtering is performed on all potential candidates obtained after processing the auxiliary branch output features through the classification and regression branches. During appearance consistency verification, a spatial neighborhood is defined for each auxiliary candidate, with a radius of 50 pixels centered on the candidate target's center point. High-confidence main branch candidates are searched within this neighborhood for feature similarity comparison. During geometric consistency verification, the aspect ratio of the auxiliary candidate's bounding box is checked, with a lower limit of 0.2 and an upper limit of 5.0. Simultaneously, the ratio of the bounding box area to the entire frame image area is checked, with a lower limit of 0.001 and an upper limit of 0.5. Candidates that do not meet these range requirements are eliminated. During candidate consistency verification, the cross-attention similarity between each auxiliary candidate and all main branch candidates is calculated. A redundancy threshold of 0.85 is set. Auxiliary candidates with similarity higher than this threshold are considered to highly overlap with the main branch detection results and are eliminated, retaining only auxiliary candidates with similarity lower than 0.85. After the above three-dimensional joint screening, the retained auxiliary candidates are merged with the high-confidence candidate set generated by the main branch to form the target query set of the current frame. The total number of these sets changes dynamically but usually does not exceed 400.

[0024] In step S4, the current frame target query set is input into the spatial attention module for intra-frame feature enhancement. This module employs a multi-head self-attention mechanism with 8 self-attention heads, maintaining the output dimension of the learnable linear projection layer at 256. During self-attention calculation, a relative position code is calculated for each candidate target based on its bounding box center coordinates, using a sine-cosine position coding function with a wavelength parameter set to 10000. The relative position code bias term is superimposed on the dot product of the query vector and the key vector, scaled by a scaling factor equal to the square root of the query vector dimension, i.e., 16. The spatial attention weight matrix is ​​the number of candidates in the current frame multiplied by the number of candidates in the current frame. Each weight value is calculated using a normalized exponential function, with the candidate region having the highest attention score receiving a larger aggregation weight. After weighted aggregation of the value vectors based on this weight matrix, the aggregation result is added to the original candidate features via a residual connection. The retention ratio of the residual connection is set to 1.0, and the epsilon parameter for layer normalization is set to 1e-5. The final output is the spatially enhanced candidate feature representation of the current frame.

[0025] In step S5, the spatially enhanced candidate features of the current frame are used as keys and values, maintaining a dimension of 256. Historical trajectory queries are input as query terms into the temporal cross-attention module. The maximum capacity of the historical trajectory query set is set to 500 entries, with each trajectory query vector having a dimension of 256. Historical trajectory queries with more than 30 consecutive mismatched frames are temporarily retained but not included in the association calculation. The number of attention heads in the temporal cross-attention module is set to 8, and the scaling factor is also set to 16. The calculated temporal attention weight matrix is ​​the size of the number of historical trajectories multiplied by the number of current frame candidates, with each weight value representing the degree of matching between the corresponding historical trajectory and the current candidate. For each historical trajectory, the current candidate corresponding to the position of the maximum attention weight in its corresponding row is selected as the association matching result. Simultaneously, the value vectors of all current candidates are weighted and aggregated according to the attention weights to obtain the updated trajectory feature representation. During the update process, the fusion ratio coefficient of the current frame observation information is set to 0.7, and the retention ratio coefficient of the original historical trajectory information is set to 0.3.

[0026] In step S6, trajectory management is performed based on the association matching results output by the temporal attention module. For current frame candidates that successfully match historical trajectories, the corresponding historical trajectory's identifier number is directly inherited, and this number is assigned sequentially starting from 1. For current frame candidates that fail to match but have a classification confidence score higher than 0.6, a new target trajectory is initialized, a new identifier number is assigned, and the current frame is recorded as the starting frame for that trajectory. For unmatched historical trajectories, their consecutive mismatch frame counter is checked. If the number of consecutive mismatch frames is less than 15, the trajectory is temporarily retained until the next frame to continue waiting for matching, and its status is marked as pending recovery. If the number of consecutive mismatch frames reaches or exceeds 15, the trajectory is terminated, removed from the active trajectory set, and its identifier number is released for reassignment. After all retained historical trajectories are queried and associated with the current frame, the multi-target tracking result of the current frame is output. This result includes the bounding box position coordinates, identifier number, and trajectory status identifier of each target in the current frame.

[0027] Preferably, in S2, when the auxiliary detection branch concatenates the auxiliary target query and the noise query and processes them through the auxiliary branch self-attention module, it follows that... Construct an enhanced query set, in which This indicates the auxiliary target query. To assist in querying the number of queries, For feature dimension, This indicates the injected noise query. This represents the enhanced query set obtained after concatenation; after self-attention calculation, the enhanced query set discards the noisy query part and retains the auxiliary query with attention bias for cross-attention interaction with the encoder output.

[0028] Specifically, the auxiliary detection branch concatenates the auxiliary target query and the noisy query, and then processes the concatenation using the auxiliary branch's self-attention module. This involves introducing an additional noisy query vector to guide the attention distribution of the auxiliary target query towards low-confidence feature regions. This formula is based on the principle of calculating the dot product similarity between query vectors and key vectors in the self-attention mechanism. The purpose of the concatenation operation is to use the noisy query as a fixed anchor point in the calculation of the self-attention weight matrix. Thus, under the action of the attention normalization exponential function, the attention weight of the auxiliary target query is partially allocated to the virtual region represented by the noisy query. In practice, the number of auxiliary target queries is set to 150, with each query vector having a feature dimension of 256. The number of noisy queries is fixed at 1, also with a dimension of 256, resulting in 151 enhanced query sets after concatenation. During the self-attention calculation process, each enhanced query calculates a similarity score with all 151 queries, including the noisy query. The scores are scaled by a scaling factor of 16 and then transformed into an attention weight distribution using a normalization exponential function. Since the initial element values ​​of the noisy queries are randomly sampled from a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, they gradually learn feature patterns representing low-confidence regions during the optimization process. This causes the auxiliary target queries to acquire an attention bias towards low-confidence regions when interacting with the noisy queries. After the attention calculation is completed, the feature vectors corresponding to the noisy query positions in the output features are discarded, and only the feature vectors corresponding to the original 150 auxiliary target query positions are retained. These are used for subsequent cross-attention calculations with the shared feature map output by the encoder, thereby embedding the attention bias information into the feature representation of the auxiliary queries while maintaining the number of auxiliary queries.

[0029] like Figure 2The diagram illustrates the multi-dimensional consistency filtering algorithm of this invention. It presents the complete processing flow of joint three-dimensional screening of the potential candidate set output by the auxiliary detection branch in a modular block diagram form, aiming to effectively suppress noise and redundant candidates while improving recall. The left side of the diagram represents the input portion, labeled as the potential candidate set. This set consists of all predicted bounding boxes output by the classification and regression branches of the auxiliary detection branch, totaling 150. Each candidate includes two types of information: feature representation and bounding box geometric parameters. Three parallel processing branches extend to the right from the input box, corresponding to three independent consistency verification dimensions: appearance consistency verification branch, geometric consistency verification branch, and candidate consistency verification branch, from top to bottom. In the first appearance consistency verification branch, the diagram shows two sub-modules: neighborhood space search and cosine similarity calculation. The neighborhood space search module defines a circular neighborhood with a preset pixel radius of 50 pixels, centered on the center point of the bounding box of each auxiliary candidate. Within this range, it searches for high-confidence main branch candidate targets. The cosine similarity calculation module performs a dot product normalization operation on the 256-dimensional feature vector of the auxiliary candidate and the feature vector of each main branch candidate in the neighborhood, and takes the maximum cosine similarity value as the appearance consistency score. Candidates with scores lower than the threshold of 0.65 are judged as having abnormal appearances and are removed. The second geometric consistency verification branch includes two sub-modules: bounding box parameter decoding and physical rationality judgment. The bounding box parameter decoding module extracts the center coordinates, width, and height information of candidate boxes from the regression branch output. The physical rationality judgment module simultaneously examines two indicators: aspect ratio and relative area. The effective range for aspect ratio is set to 0.2 to 5.0, and the effective range for relative area is set to 0.001 to 0.5. The two indicators are jointly judged by multiplying them using an indicator function. If either indicator exceeds the reasonable range, the geometric consistency score is zero, and the candidate is directly eliminated. The third candidate consistency verification branch includes two sub-modules: cross-attention calculation and redundancy judgment. The cross-attention calculation module performs a scaled dot product similarity operation on the features of each auxiliary candidate and all high-confidence main branch candidate features, and obtains the attention weight distribution through a normalized exponential function. The redundancy judgment module takes the maximum attention weight as a measure of redundancy. The candidate consistency score is obtained by subtracting the maximum weight value from 1. Candidates with a score below 0.15, i.e., a maximum weight greater than 0.85, are considered highly redundant with the main branch and are eliminated. The filtering results from the three branches converge in the joint filtering output module on the right side of the graph. The final retained auxiliary candidates are merged with the high-confidence candidates from the main branch to form the target query set for the current frame.

[0030] Preferably, the multi-dimensional consistency filtering in S3 includes appearance consistency verification for each candidate object in the potential candidate set. Extracting feature representations from encoder output and neighboring space Cosine similarity is calculated for targets with medium to high confidence levels, and appearance consistency scores are determined according to... Confirmed, among which Indicates candidate objects The center point is the spatial neighborhood within the preset pixel radius of the circle. This is the high-confidence candidate set. For the neighboring region Feature representation of a high-confidence target.

[0031] Specifically, the appearance consistency verification in multi-dimensional consistency filtering evaluates the appearance rationality of auxiliary candidates by calculating the maximum cosine similarity between auxiliary candidate features and high-confidence main branch candidate features in the neighborhood. This is based on the fundamental assumption that similar targets in the feature space should have similar feature representations. The dot product operation in the cosine similarity calculation formula measures the consistency of two feature vectors in direction. The denominator is the product of the magnitudes of the two feature vectors, which serves to normalize the similarity value range to between -1 and +1. In practice, the feature representation of the auxiliary candidate is extracted from the shared feature map output by the encoder through bilinear interpolation. The extraction location is the feature map coordinates corresponding to the center point of the auxiliary candidate's bounding box, and the feature vector dimension is 256. The neighborhood space is a circular region with the center point of the auxiliary candidate and a pixel radius of 50 pixels. All high-confidence main branch candidate targets are searched within this region. For each main branch candidate within the neighborhood, its 256-dimensional feature vector is element-wise multiplied with the 256-dimensional feature vector of the auxiliary candidate, and the results are summed to obtain the dot product. Simultaneously, the magnitudes of the two feature vectors are calculated separately and multiplied. The cosine similarity is obtained by dividing the dot product by the magnitude product. The maximum cosine similarity score among all main branch candidates within the neighborhood is selected as the appearance consistency score for that auxiliary candidate. A score close to 1 indicates that the auxiliary candidate is highly consistent in appearance with a reliable target within the neighborhood, while a score close to 0 or a negative value indicates a significant appearance difference. In practice, an appearance consistency score threshold of 0.65 is set, and only auxiliary candidates with scores higher than this threshold are retained for subsequent filtering stages.

[0032] Preferably, the multi-dimensional consistency filtering in S3 includes geometric consistency verification, whereby a physical representation of the detection target in the potential candidate set is obtained via a multilayer perceptron. ,in Indicates the coordinates of the bounding box center. Indicates width, Indicates height; geometric consistency score is based on... Confirmed, among which and These are the lower and upper bounds of the effective aspect ratio. and These are the lower and upper bounds of the relative area. and These are the width and height of the current video image frame, respectively. This is an indicator function.

[0033] Specifically, the geometric consistency verification in multi-dimensional consistency filtering simultaneously checks whether the aspect ratio and relative area of ​​the auxiliary candidate bounding boxes fall within a reasonable range by multiplying indicator functions. This is based on the prior knowledge that real targets in images should have physically reasonable shapes and sizes. The aspect ratio is calculated as the ratio of the bounding box width to its height, which has different reasonable ranges for different target categories. The relative area is calculated as the ratio of the bounding box area to the area of ​​the entire frame image, used to exclude outlier detection boxes that are too small or too large. In practice, the width and height of the auxiliary candidate bounding boxes are obtained by decoding the bounding box parameters output from the regression branch, in pixels. The lower limit of the effective aspect ratio range is set to 0.2, and the upper limit is set to 5.0, covering most reasonable target shapes from horizontal to vertical stripes. The width W and height H of the entire frame image are determined by the input image resolution; the width is set to 1920 pixels, the height to 1080 pixels, and the frame image area W multiplied by H is 2,073,600 square pixels. The lower limit of the effective range for relative area is set to 0.001, corresponding to a bounding box area of ​​at least 2073.6 square pixels. The upper limit of the relative area is set to 0.5, corresponding to a bounding box area of ​​at most 1036800 square pixels. The indicator function returns a value of 1 when the conditions are met and a value of 0 when the conditions are not met. The product of two indicator functions is only 1 if both conditions are met simultaneously; otherwise, the product is 0. The geometric consistency score takes a value of 0 or 1. A value of 1 indicates that the candidate's geometry and size conform to reasonable target features, while a value of 0 indicates the presence of geometric anomalies and the candidate is rejected.

[0034] Preferably, the multi-dimensional consistency filtering in S3 includes candidate consistency verification, for each auxiliary candidate object in the potential candidate set. Extract its feature representation And calculate the cross-attention score with all main branch candidates in the high-confidence candidate set, based on The normalized weights are obtained, where This represents the function for calculating feature similarity. and These are the main branch candidate objects in the high-confidence candidate set. The cardinality of the high-confidence candidate set; candidate consistency score based on Determine that only auxiliary candidates with a candidate consistency score higher than the preset threshold will be retained.

[0035] Specifically, in the candidate consistency verification of multi-dimensional consistency filtering, a cross-attention mechanism is used to calculate the similarity distribution between auxiliary candidates and all main branch candidates, and the maximum similarity is used to measure the degree of redundancy. The cross-attention score calculation in the formula adopts the basic form of scaled dot product attention. The numerator calculates the similarity index between the auxiliary candidate feature and a certain main branch candidate feature, while the denominator sums the similarity indices of all main branch candidates to achieve normalization, ensuring that the sum of the attention weights of all main branch candidates is 1. The similarity calculation function uses a dot product approach. Both the auxiliary candidate feature and the main branch candidate feature are 256-dimensional vectors. The dot product result is used directly without scaling because the feature vectors have already undergone layer normalization to keep their magnitude within a reasonable range. In the formula for calculating the candidate consistency score, the maximum attention weight among all main branch candidates is subtracted from the value of 1. The maximum attention weight reflects the degree of overlap between the auxiliary candidate and the most similar main branch candidate. The closer this value is to 1, the more likely the auxiliary candidate is to completely overlap with a certain main branch candidate. In this case, the candidate consistency score is close to 0, indicating a high degree of redundancy, and it should be eliminated. Conversely, the closer the maximum attention weight is to 0, the more likely the auxiliary candidate is to be dissimilar to all main branch candidates. The candidate consistency score is close to 1, indicating a high degree of complementarity, and it should be retained. In specific implementation, the redundancy judgment threshold is set to 0.85. That is, when the maximum attention weight is greater than or equal to 0.85, the candidate consistency score is less than or equal to 0.15, and the auxiliary candidate is considered redundant and eliminated. When the maximum attention weight is less than 0.85, the candidate consistency score is higher than 0.15, and the auxiliary candidate is considered to have supplementary value and is retained.

[0036] Preferably, the main detection branch and the auxiliary detection branch adopt asymmetric parameter configuration, the main branch self-attention module and the auxiliary branch self-attention module are independent network parameters, the main branch feedforward network and the auxiliary branch feedforward network are independent feedforward parameters, and the weight parameters of the shared cross-attention module are shared between the two branches; the spatial attention module adopts a multi-head self-attention mechanism with relative position encoding, the relative position encoding is calculated based on the relative displacement between the center coordinates of the candidate target bounding box; the temporal cross-attention module adopts a multi-head cross-attention mechanism, wherein the number of heads is set as a trainable hyperparameter; historical trajectory query is terminated when the continuous mismatch duration exceeds a preset trajectory survival threshold, and is temporarily retained when it is below the trajectory survival threshold, waiting for re-association in the next time frame.

[0037] like Figure 3The diagram illustrates the spatiotemporal attention target association algorithm of this invention. It showcases the overall architecture of the spatial attention module and the temporal cross-attention module working collaboratively to achieve intra-frame feature enhancement and inter-frame identity association, presented in a two-layer cascaded structure. The upper part of the diagram represents the spatial attention processing layer, and the lower part represents the temporal cross-attention processing layer, connected sequentially by data flow arrows. The left input of the spatial attention processing layer is labeled as the target query for the current frame. This query set is composed of high-confidence candidates from the main branch and auxiliary candidates filtered through multiple dimensions, with a dynamically changing number but not exceeding 400. Each candidate has a feature dimension of 256. The input branches to the right into three parallel linear projection paths, generating a spatial query vector, a spatial key vector, and a spatial value vector, respectively, all maintaining a projection dimension of 256. Simultaneously, a branch extends from below the input box to the relative position encoding module. This module applies a sine and cosine position encoding function based on the horizontal and vertical differences of the center coordinates of each candidate target's bounding box, generating a scalar bias term after linear projection dimensionality reduction. The spatial query vector and spatial key vector are fed into the dot product calculation module. Their output is added element-wise with the relative position encoding bias term in the overlay module. The overlay result is scaled by a scaling factor of 16 and then input into the normalization exponential function module to generate a spatial attention weight matrix with a size equal to the number of candidates in the current frame multiplied by the number of candidates in the current frame. The sum of each row of this matrix is ​​1. The spatial attention weight matrix and spatial value vector are input into the weighted aggregation module, which outputs intermediate features that fuse neighborhood spatial layout information. These intermediate features are then added to the original current frame target query through a residual connection. After layer normalization, the spatially enhanced current frame candidate feature representation is obtained and passed down as the output of the spatial attention layer to the temporal cross-attention processing layer. In the temporal cross-attention processing layer, the upper input is a historical trajectory query set with a maximum capacity of 500 entries, each trajectory query having a 256-dimensional dimension. A temporal query vector is generated through linear projection. The lower input is the spatially enhanced current frame candidate from the upper layer, which is used to generate a temporal key vector and a temporal value vector through two linear projections, respectively. The time query vector and time key vector are fed into the dot product and scaling module. After processing with a normalized exponential function, a time attention weight matrix with a size equal to the number of historical trajectories multiplied by the number of current frame candidates is generated. Each row of this matrix corresponds to the matching probability distribution of a historical trajectory across all current frame candidates. The time attention weight matrix and time value vector are input into the weighted aggregation module, which outputs the updated trajectory features fused with current frame observation information. Simultaneously, a branch is drawn from the time attention weight matrix to the maximum weight index module. This module selects the current frame candidate with the highest attention weight for each historical trajectory as the association matching result. Finally, the trajectory update and identity association results are output in the lower right corner of the graph.

[0038] Preferably, in step S2, the main detection branch generates a high-confidence candidate set through the main target query via the main branch self-attention module, the shared cross-attention module, and the main branch feedforward network, including the following sub-steps: S211, the main target query generates a main query vector and a primary key vector through linear projection, and outputs the main branch enhanced query by capturing the dependency relationship between queries through self-attention calculation; S212, the main branch enhanced query interacts with the shared feature map as a query item through the shared cross-attention module, and aggregates visual information from the feature map according to the attention distribution to generate the main branch decoding feature; S213, the main branch decoding feature is nonlinearly transformed by the main branch feedforward network to generate the main branch output feature; S214, the main branch output feature predicts the target category probability and bounding box parameters through the classification branch and the regression branch respectively, and filters the predicted boxes with confidence higher than a preset threshold to form the high-confidence candidate set.

[0039] Specifically, in step S21, the number of main target queries is set to 300, and each query vector has a dimension of 256. Query vectors, key vectors, and value vectors are generated through three independent linear projection layers, with the output dimension of each projection layer remaining at 256. The self-attention weight matrix is ​​obtained by calculating the dot product of the query vector and all key vectors, dividing by a scaling factor of 16, and then applying a normalized exponential function. This matrix has a size of 300 x 300. The value vectors are weighted and aggregated, then processed through residual connections and layer normalization to output the main branch enhanced query. In step S22, the main branch enhanced query is used as a query item and undergoes cross-attention calculation with the shared feature map output by the encoder. The spatial size of the shared feature map is 1 / 32 of the input image, and the number of channels is 2048. After flattening the shared feature map along its spatial dimensions, a feature sequence with a length equal to the input image size divided by 32 and then squared is obtained. The feature vector at each position has 2048 dimensions. After being reduced to 256 dimensions by linear projection, it serves as the key and value. After cross-attention calculation, the main branch decoded features are output. In step S23, the decoded features of the main branch are processed by the main branch feedforward network. This feedforward network includes two fully connected layers. The first layer maps 256 dimensions to 1024 dimensions and is activated using a modified linear unit. The second layer maps 1024 dimensions back to 256 dimensions, outputting the main branch output features. In step S24, the main branch output features are input to the classification branch and the regression branch, respectively. The classification branch is a linear layer that maps 256 dimensions to the number of target categories plus 1 dimension, with the number of categories set to 80. The regression branch is a linear layer that maps 256 dimensions to 4-dimensional output bounding box parameters. Predicted boxes with a classification confidence score higher than 0.5 are selected to form a high-confidence candidate set.

[0040] Preferably, the auxiliary detection branch in S2, which concatenates the auxiliary target query and the noise query and generates a potential candidate set via the auxiliary branch self-attention module, the shared cross-attention module, and the auxiliary branch feedforward network, includes the following sub-steps: S221, the auxiliary target query and the noise query are concatenated and linearly projected to generate an auxiliary query vector and an auxiliary key vector. Self-attention calculation is used to guide the auxiliary query towards the low-confidence region using the noise query as an attention anchor, outputting an auxiliary enhancement query; S222, after discarding the noise query portion of the auxiliary enhancement query, the remaining auxiliary query is used as a query item and interacts with the shared feature map via the shared cross-attention module to generate auxiliary decoding features; S223, the auxiliary decoding features undergo nonlinear transformation via the auxiliary branch feedforward network to generate auxiliary output features; S224, the auxiliary output features predict the target category probability and bounding box parameters via the classification branch and regression branch respectively, and all predicted boxes are extracted to form the potential candidate set.

[0041] Specifically, in step S21, the number of auxiliary target queries is set to 150, with a vector dimension of 256. The number of noise queries is 1, also with a dimension of 256. After concatenation, the number of enhanced query sets is 151. Query, key, and value vectors are generated through three linear projection layers, maintaining a projection dimension of 256. During the self-attention calculation process, the noise query serves as an attention anchor point, participating in the attention weight calculation between all 151 positions. This causes the attention distribution of the auxiliary target queries to be biased towards the low-confidence region feature pattern represented by the noise query. After the self-attention output, the feature vectors corresponding to the noise query positions are discarded, retaining only the feature vectors of the 150 auxiliary query positions as auxiliary enhanced queries. In step S22, the retained 150 auxiliary enhanced queries are used as query items and cross-attention calculations are performed with the shared feature map output by the encoder. The shared feature map is reduced to 256 dimensions through linear projection and used as keys and values ​​to calculate auxiliary decoding features. This step shares the same cross-attention module weight parameters as the main branch. In step S23, the auxiliary decoding features are processed by an auxiliary branch feedforward network. This feedforward network has the same structure as the main branch feedforward network but its weight parameters are independent. It includes two fully connected layers. The first layer maps 256 dimensions to 1024 dimensions and is activated. The second layer maps back to 256 dimensions and outputs auxiliary output features. In step S24, the auxiliary output features are input to the classification branch and the regression branch respectively. The classification branch outputs 80 classes plus a 1-dimensional class probability, and the regression branch outputs 4-dimensional bounding box parameters. Unlike the main branch, the auxiliary branch does not set a screening threshold for classification confidence, but instead extracts all 150 predicted boxes to form a potential candidate set.

[0042] Preferably, step S4 includes the following sub-steps: S41, obtaining a spatial query vector and a spatial key vector from the current frame target query through a learnable linear projection layer, and calculating a relative position encoding bias term based on the bounding box center coordinates of each candidate target; S42, superimposing the relative position encoding bias term onto the dot product result of the spatial query vector and the spatial key vector, and obtaining a spatial attention weight matrix after scaling and normalization; S43, performing weighted aggregation on the value vector corresponding to the current frame target query according to the spatial attention weight matrix to generate intermediate features that fuse neighborhood spatial layout information; S44, adding the intermediate features to the original current frame target query through a residual connection to obtain a spatially enhanced candidate feature representation of the current frame.

[0043] Specifically, in step S41, the current frame target query set is input into the spatial attention module. The number of this set changes dynamically but does not exceed 400. Each query vector has a dimension of 256. A learnable linear projection layer generates spatial query vectors and spatial key vectors, maintaining a projection dimension of 256. Simultaneously, a relative position encoding bias term is calculated based on the bounding box center coordinates of each candidate target. The calculation method involves applying sine and cosine position encoding functions to the difference between the horizontal and vertical center coordinates of the two candidate targets, respectively. The wavelength parameter is set to 10000. The encoded vectors in the horizontal and vertical directions are concatenated and then linearly projected to reduce the dimension to 1 dimension to obtain a scalar bias value. In step S42, the dot product of the spatial query vector and the spatial key vector is divided by a scaling factor of 16. The relative position encoding bias term is then element-wise superimposed onto the corresponding positions of the dot product matrix. The superimposed matrix is ​​normalized along the row direction using a normalized exponential function to obtain the spatial attention weight matrix. The size of this matrix is ​​the number of candidates in the current frame multiplied by the number of candidates in the current frame, and the sum of the elements in each row of the matrix is ​​1. In step S43, the value vector corresponding to the target query in the current frame is weighted and aggregated according to the spatial attention weight matrix. The value vector is generated by the target query in the current frame through the third linear projection layer, and has a dimension of 256. During weighted aggregation, the output of each position is the weighted sum of the corresponding row of the weight matrix and all value vectors. In step S44, the intermediate features obtained by weighted aggregation are added to the original target query in the current frame through a residual connection. The retention ratio of the residual connection is 1.0. After addition, the feature is normalized by layer, and the epsilon parameter of the normalization is set to 1e-5. Finally, the spatially enhanced candidate feature representation of the current frame is output, and the dimension remains 256.

[0044] Preferably, step S5 includes the following sub-steps: S51, obtaining a time query vector by linear projection of the historical trajectory query, and obtaining a time key vector and a time value vector by linear projection of the spatially enhanced current frame candidate; S52, calculating and scaling the dot product between the time query vector and the time key vector, and obtaining a time attention weight matrix by normalizing the exponential function, where each element represents the degree of matching between the historical trajectory and the current frame candidate; S53, performing weighted aggregation of the time value vector according to the time attention weight matrix to obtain the updated trajectory features fused with the current frame observation information; S54, selecting the current frame candidate with the largest attention weight for each historical trajectory from the time attention weight matrix as the association matching result, and dynamically updating the motion state and appearance representation of the historical trajectory according to the updated trajectory features.

[0045] Specifically, in step S51, the number of historical trajectory queries dynamically changes, but the maximum capacity is set to 500. Each trajectory query vector has a dimension of 256. A time query vector is generated through linear projection, maintaining a projection dimension of 256. The number of current frame candidates after spatial enhancement does not exceed 400, and each candidate feature has a dimension of 256. Time key vectors and time value vectors are generated through linear projection, both with a projection dimension of 256. In step S52, the dot product of the time query vector and the time key vector yields a score matrix whose size is the number of historical trajectories multiplied by the number of current frame candidates. Each element in the score matrix is ​​divided by a scaling factor of 16. Then, a normalized exponential function is applied along the column direction, i.e., the current frame candidate dimension, to obtain a time attention weight matrix. Each row in the matrix corresponds to the attention distribution of a historical trajectory on all current frame candidates, and the sum of the elements in each row is 1. In step S53, the time value vectors are weighted and aggregated based on the time attention weight matrix. The output for each historical trajectory is the weighted sum of that row of the weight matrix and all candidate value vectors, resulting in the updated trajectory features fused with the current frame observation information. In step S54, the column index corresponding to the maximum weight value is selected from each row of the temporal attention weight matrix. The candidate of the current frame pointed to by the column index is the best association matching result of the historical trajectory. At the same time, the motion state representation of the historical trajectory is updated according to the updated trajectory features. During the update, the fusion ratio coefficient of the current frame observation information is set to 0.7, and the retention ratio coefficient of the original information of the historical trajectory is set to 0.3. Finally, the cross-frame identity association is completed and the target identity information of the current frame matched by each historical trajectory is output.

[0046] A robust multi-target tracking method based on high recall and strong temporal correlation is proposed, which improves detection recall by introducing a recall-oriented auxiliary detection branch outside the main detection branch. Differential query initialization and noise query injection strategies are used to expand the detection search space, enabling the model to effectively uncover low-confidence potential targets that are easily missed by the main branch in dense crowds, small targets, occlusion, and motion blur scenes. Simultaneously, a multi-dimensional consistency filtering mechanism is used to jointly screen candidates with blurred appearance features, abnormal geometric dimensions, and high redundancy with the main branch, achieving a better balance between high recall and low noise. Enhanced cross-frame correlation stability is also achieved by constructing a Transformer-based spatiotemporal joint attention mechanism. The spatial attention module utilizes the relative positional relationships between candidate targets within the same frame for local context enhancement, mitigating the confusion problem of similar-looking targets in dense scenes. The temporal cross-attention module dynamically interacts with historical trajectory queries and current frame candidates, adaptively fusing current observation information to update the trajectory representation based on attention weights, enabling the tracker to maintain stable identity consistency even under conditions of rapid target movement and long-term occlusion.

[0047] This invention addresses the issue of insufficient recall in single-branch detection by abandoning the strategy of relying solely on the high-confidence output of a single detection branch. Instead, it employs a dual-branch collaborative approach to broaden the candidate coverage. Simultaneously, multi-dimensional consistency constraints control the noise introduced by the auxiliary branch to a reasonable level, fundamentally reducing the risk of missing true targets during the association phase due to missed detections. Regarding the issue of association errors caused by unstable historical trajectory updates, this method no longer relies on linear motion models that can only handle simple uniform motion scenarios or inference modules with unstable predictions. Instead, it utilizes a spatiotemporal attention mechanism to comprehensively leverage intra-frame spatial context and inter-frame temporal dependencies. This allows trajectory queries to dynamically adjust the state representation based on the current frame observation, effectively suppressing identity switching and trajectory interruption phenomena in scenarios with dense interactions and occlusion re-enactment, thus improving the overall robustness and reliability of multi-target tracking systems in complex real-world environments.

[0048] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A robust multi-target tracking method based on high recall and strong temporal correlation, characterized in that, Includes the following steps: S1, extract the shared feature map of the current video image frame through the backbone network and encoder; S2, the main detection branch generates a high-confidence candidate set through the main target query via the main branch self-attention module, the shared cross-attention module and the main branch feedforward network. The auxiliary detection branch concatenates the auxiliary target query and the noise query and generates a potential candidate set through the auxiliary branch self-attention module, the shared cross-attention module and the auxiliary branch feedforward network. S3, perform multi-dimensional consistency filtering on the potential candidate set to remove noise and redundancy, and combine the filtered auxiliary candidates with the high-confidence candidate set to form the target query for the current frame; S4, input the target query of the current frame into the spatial attention module, and enhance spatial features through self-attention interaction between candidate targets in the same frame; S5 uses the spatially enhanced current frame candidate as the key and value, and the historical trajectory query as the query item input to the time-cross attention module. Based on the attention weight, the current frame observation information is aggregated to update the historical trajectory representation and complete the cross-frame identity association. S6: Based on the association results, inherit the historical trajectory identity of the current frame candidate that is successfully matched, initialize a new trajectory for the current frame candidate that is not matched but meets the conditions, retain or terminate the unmatched historical trajectory according to the continuous mismatch duration, and output the multi-target tracking result of the current frame.

2. The robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, In S2, the auxiliary detection branch concatenates the auxiliary target query and the noise query, and then processes them through the auxiliary branch self-attention module, based on... Construct an enhanced query set, in which This indicates the auxiliary target query. To assist in querying the number of queries, For feature dimension, This indicates the injected noise query. This represents the enhanced query set obtained after concatenation; after self-attention calculation, the enhanced query set discards the noisy query part and retains the auxiliary query with attention bias for cross-attention interaction with the encoder output.

3. The robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, The multi-dimensional consistency filtering in S3 includes appearance consistency verification, for each candidate object in the potential candidate set. Extracting feature representations from encoder output and neighboring space Cosine similarity is calculated for targets with medium to high confidence levels, and appearance consistency scores are determined according to... Confirmed, among which Indicates candidate objects The center point is the spatial neighborhood within the preset pixel radius of the circle. This is the high-confidence candidate set. For the neighboring region Feature representation of a high-confidence target.

4. The robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, The multi-dimensional consistency filtering in S3 includes geometric consistency verification, whereby a physical representation of the target in the potential candidate set is obtained via a multilayer perceptron. ,in Indicates the coordinates of the bounding box center. Indicates width, Indicates height; geometric consistency score is based on... Confirmed, among which and These are the lower and upper bounds of the effective aspect ratio. and These are the lower and upper bounds of the relative area. and These are the width and height of the current video image frame, respectively. This is an indicator function.

5. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, The multi-dimensional consistency filtering in S3 includes candidate consistency verification, for each auxiliary candidate object in the potential candidate set. Extract its feature representation And calculate the cross-attention score with all main branch candidates in the high-confidence candidate set, based on The normalized weights are obtained, where This represents the function for calculating feature similarity. and These are the main branch candidate objects in the high-confidence candidate set. The cardinality of the high-confidence candidate set; Candidate consistency score based on Determine that only auxiliary candidates with a candidate consistency score higher than the preset threshold will be retained.

6. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, The main detection branch and the auxiliary detection branch adopt asymmetric parameter configuration. The self-attention module of the main branch and the self-attention module of the auxiliary branch are independent network parameters. The feedforward network of the main branch and the feedforward network of the auxiliary branch are independent feedforward parameters. The weight parameters of the shared cross-attention module are shared between the two branches. The spatial attention module adopts a multi-head self-attention mechanism with relative position encoding. The relative position encoding is calculated based on the relative displacement between the center coordinates of the candidate target bounding box. The time-cross attention module adopts a multi-head cross-attention mechanism, wherein the number of heads is set as a trainable hyperparameter; historical trajectory queries are terminated when the duration of continuous mismatch exceeds a preset trajectory survival threshold, and are temporarily retained when the duration is below the trajectory survival threshold, waiting for re-association in the next time frame.

7. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, In S2, the main detection branch generates a high-confidence candidate set through the main target query, the main branch self-attention module, the shared cross-attention module, and the main branch feedforward network, including the following steps: The main target query is generated into a main query vector and a primary key vector through linear projection, and the main branch is output to enhance the query by capturing the dependencies between queries through self-attention calculation. The main branch enhanced query, as a query item, interacts with the shared feature map through the shared cross-attention module, and aggregates visual information from the feature map based on the attention distribution to generate the main branch decoding feature; The main branch decoding features are nonlinearly transformed by the main branch feedforward network to generate the main branch output features; The main branch output features predict the target category probability and bounding box parameters through the classification branch and regression branch respectively, and the predicted boxes with confidence scores higher than a preset threshold are selected to form the high-confidence candidate set.

8. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, The auxiliary detection branch in S2 concatenates the auxiliary target query and the noise query, and then generates a potential candidate set through the auxiliary branch self-attention module, the shared cross-attention module, and the auxiliary branch feedforward network, including the following steps: The auxiliary target query and the noise query are concatenated and linearly projected to generate an auxiliary query vector and an auxiliary key vector. Through self-attention calculation, the noise query is used as an attention anchor point to guide the auxiliary query to the low confidence region, and the auxiliary enhanced query is output. After discarding the noisy query portion of the auxiliary enhancement query, the retained auxiliary query is used as a query item and interacts with the shared feature map through the shared cross-attention module to generate auxiliary decoding features; The auxiliary decoding features are nonlinearly transformed by the auxiliary branch feedforward network to generate auxiliary output features; The auxiliary output features predict the target category probability and bounding box parameters through classification and regression branches, respectively, and extract all predicted boxes to form the potential candidate set.

9. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, S4 includes: The target query of the current frame is processed through a learnable linear projection layer to obtain a spatial query vector and a spatial key vector, and a relative position encoding bias term is calculated based on the bounding box center coordinates of each candidate target; the relative position encoding bias term is superimposed on the dot product result of the spatial query vector and the spatial key vector, and the result is scaled and normalized to obtain a spatial attention weight matrix. The value vector corresponding to the target query in the current frame is weighted and aggregated according to the spatial attention weight matrix to generate intermediate features that fuse neighborhood spatial layout information; the intermediate features are added to the original target query in the current frame through residual connection to obtain the spatially enhanced candidate feature representation of the current frame.

10. A robust multi-target tracking method based on high recall and strong temporal correlation according to claim 1, characterized in that, S5 includes: The historical trajectory query is linearly projected to obtain a time query vector, and the spatially enhanced current frame candidate is linearly projected to obtain a time key vector and a time value vector. The dot product between the time query vector and the time key vector is calculated and scaled, and the time attention weight matrix is ​​obtained by normalizing the exponential function, where each element represents the degree of matching between the historical trajectory and the current frame candidate; the time value vector is weighted and aggregated according to the time attention weight matrix to obtain the updated trajectory features that fuse the observation information of the current frame. For each historical trajectory, the candidate frame with the largest attention weight is selected from the temporal attention weight matrix as the association matching result, and the motion state and appearance representation of the historical trajectory are dynamically updated based on the updated trajectory features.

Images