A trajectory pair association method based on time sequence and fusion module

Abstract

The application belongs to the technical field of trajectory pair association, and discloses a trajectory pair association method based on a time sequence and a fusion module, which comprises the following steps: step 1, a time sequence module composed of a time sequence convolution, a channel normalization layer and a channel-level topology optimization graph convolution is constructed to extract dynamic features of trajectories changing over time; step 2, a fusion module composed of a fusion convolution, an adaptive graph channel attention module and a batch normalization layer is constructed to fuse different features of objects in the same time segment; step 3, a trajectory similarity calculation module composed of a multi-layer time sequence module, a fusion module, an adaptive average pooling and a full connection classifier is constructed to calculate a trajectory pair similarity score; and step 4, the similarity between each trajectory pair in the preliminary tracking result is extracted, and the Hungarian algorithm is used for matching to output a final multi-target tracking result. The application can fuse trajectory features containing time and space information, and improve the stability and accuracy of trajectory association.

Description

Technical Field

[0001] This invention belongs to the field of trajectory pair association technology, and particularly relates to a trajectory pair association method based on time series and fusion modules. Background Technology

[0002] The core task of multi-target tracking is to identify each target of interest in a video sequence and maintain its trajectory information across consecutive frames. Therefore, it is widely used in fields such as intelligent surveillance and security, autonomous driving and assisted driving, and sports analytics and motion recognition. However, during multi-target tracking, issues such as target occlusion, appearance changes, and detection errors can easily lead to trajectory fragmentation and identity switching, resulting in decreased accuracy in subsequent trajectory association and missed detection correction. Therefore, research on trajectory pair association techniques has significant theoretical and practical value for improving the performance of multi-target tracking.

[0003] In multi-object tracking methods, trajectory pair association techniques serve as a post-processing method. By associating and reconstructing short or incomplete trajectories, missing trajectories can be effectively repaired, thereby improving the continuity and stability of tracking results. Common trajectory pair association techniques often rely on appearance features. For example, the Globalinformation and Optimizing Tracker (Giaotracker) proposed by Du et al. uses an improved ResNet50-TP model to encode trajectory appearance features and combines temporal and spatial losses to associate trajectory pairs. The Refinement Multi-Object Tracker (ReMOT) proposed by Yang et al. decomposes imperfect trajectories into shorter trajectories and re-merges the decomposed trajectories based on appearance features. However, these methods rely on trajectory appearance features, resulting in high computational costs. Therefore, Du et al. proposed a simple online real-time tracking method (StrongSimple Online and Realtime Tracking, StrongSORT) that enhances trajectory association using only motion features. This method encodes temporal and spatial features using two multilayer perceptrons respectively and combines pooling operations to achieve feature compression and extraction, thereby obtaining trajectory representations for association. Although this method can reduce some of the trajectory fragmentation problem, its performance is still unsatisfactory due to the lack of adaptive modeling of the temporal relationship of the trajectory. Summary of the Invention

[0004] The technical problem this invention aims to solve is the insufficient correlation accuracy caused by trajectory fragmentation and identity switching in multi-target tracking.

[0005] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows: A trajectory pair association method based on temporal sequence and fusion module includes the following steps: Step 1: Construct a temporal module consisting of temporal convolution, channel normalization layer, and channel-level topology optimization graph convolution to extract dynamic features of trajectory changes over time; Step 2: Construct a fusion module consisting of fused convolutions, an adaptive graph channel attention module, and a batch normalization layer to fuse different features of objects in the same time segment; Step 3: Construct a trajectory similarity calculation module consisting of a multi-layer temporal module, a fusion module, an adaptive average pooling module, and a fully connected classifier to calculate the trajectory pair similarity score; Step 4: Use the trained trajectory similarity calculation module to extract the similarity between each trajectory pair in the preliminary tracking results, and use the Hungarian algorithm to perform matching to output the final multi-target tracking results.

[0006] Furthermore, step 1 specifically involves: Step 1.1, Construction of the timing module: Step 1.1.1, construct temporal convolution; Step 1.1.2: Construct 5 channel normalization layers to process the 5 dimensions respectively; Step 1.1.3: Construct a channel-level topology-optimized graph convolution; Step 1.2, Using the timing module: Step 1.2.1: Input the information tensor of each trajectory into the time series module; Step 1.2.1: Input the tensor into the temporal convolution to extract temporal features; Step 1.2.2: Input the tensor after temporal convolution into the 5-channel normalization layer dimension by dimension; Step 1.2.3: Correct the linear units of the normalized tensor using a nonlinear activation function; Step 1.2.4: Convolve the tensor that has been corrected by the linear unit through the channel-level topology-optimized graph.

[0007] Furthermore, step 2 specifically involves: Step 2.1, Construction of the fusion module: Step 2.1.1, construct the fused convolution; Step 2.1.2: Construct the adaptive graph channel attention module; Step 2.1.3: Construct the batch normalization layer; Step 2.2, Using the fusion module: Step 2.2.1: The information tensor of each trajectory is processed by multiple time-series modules as input to the fusion module; Step 2.2.2: Input the input tensor into the temporal convolution to fully fuse the different features of objects in the same time segment; Step 2.2.3: Input the tensor after temporal convolution into the adaptive graph channel attention module; Step 2.2.3, convert the tensor after the adaptive graph channel attention module The input is fed into a batch normalization layer, and the activation function is modified by linear units to further enhance the discriminative power of the features.

[0008] Furthermore, step 3 specifically involves: Step 3.1, Construction of the trajectory similarity calculation module: Step 3.1.1: Construct multidimensional temporal feature extraction modules for the two trajectories respectively. The multidimensional temporal feature extraction modules for the two trajectories have the same structure and do not share parameters. Step 3.1.2: Construct the fusion module for the two trajectories respectively; Step 3.1.3: Construct adaptive average pooling; Step 3.1.4: Construct a fully connected classifier; Step 3.2, Using the trajectory similarity calculation module: Step 3.2.1: Input the information tensors of the two trajectories into the trajectory similarity calculation module; Step 3.2.2: Input the information tensors of the two trajectories into the corresponding multidimensional temporal feature extraction modules to obtain information tensors with the same structure as those in step 2.2.1; Step 3.2.3: Input the information tensor obtained in the previous step into the corresponding fusion module constructed in step 3.1.2 and the adaptive average pooling constructed in step 3.1.3 respectively, and remove all dimensions with channels of 1 to obtain two trajectory features; Step 3.2.4: Input the two trajectory features into the fully connected classifier defined in step 3.1.4 to calculate the similarity between the trajectories.

[0009] Step 3.2.5: Input the obtained similarity into a Softmax function with dimension 1 for normalization to obtain the final similarity score of the trajectory pairs.

[0010] Furthermore, step 4 specifically involves: Step 4.1: Extract each trajectory based on the preliminary tracking results; Step 4.2: Use the trained trajectory similarity calculation module to extract the similarity between each trajectory pair in the preliminary tracking results; Step 4.3: Check the time interval and spatial distance of each pair of trajectories in the tracking results. For the trajectory pairs that meet the requirements, calculate the similarity using the trajectory similarity calculation module. and put 1- Fill it into the cost matrix; Step 4.4: Use the Hungarian algorithm to match the cost matrix and update the trajectory identifiers to obtain the final tracking result.

[0011] The present invention has the following advantages: it can integrate trajectory features containing time and space information; it can effectively reduce identity changes during the tracking process and improve the stability and accuracy of trajectory association. Attached Figure Description

[0012] Figure 1 The timing module model structure designed for this invention.

[0013] Figure 2 The fusion module model structure designed for this invention.

[0014] Figure 3 The trajectory similarity calculation module model structure designed for this invention.

[0015] Figure 4 The trajectory pair association algorithm designed for this invention. Detailed Implementation

[0016] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0017] This patent designs a trajectory pair association technology based on temporal and fusion modules to associate trajectory pairs using the temporal and positional information of targets in the trajectory. The approach is as follows: First, a temporal module is constructed, consisting of temporal convolution, channel normalization layers, and channel-wise topology refinement graph convolution (CTRGC), to extract dynamic features of the trajectory changing over time. Then, a fusion module is constructed, consisting of fusion convolution, adaptive graph channel attention (AGCA), and batch normalization layers, to fuse different features of objects in the same time segment. Next, a trajectory similarity calculation module is constructed, consisting of multi-layer temporal modules, a fusion module, adaptive average pooling, and a fully connected classifier, to calculate the trajectory pair similarity score. Finally, the trained trajectory similarity calculation module is used to extract the similarity between each trajectory pair in the preliminary tracking results, and the Hungarian algorithm is used for matching to output the final multi-target tracking result.

[0018] Step 1, Timing Module The construction and usage process is as follows Figure 1 As shown, where They represent The input dimensions and the expected output dimensions: Step 1.1, Construction of the timing module: Step 1.1.1, Construct Temporal Convolution The input channel size of this convolution is The output channel size is The kernel size is (7, 1).

[0019] Step 1.1.2: Construct 5 channel normalization layers Each of the five dimensions is processed separately.

[0020] Step 1.1.3: Construct a channel-level topology optimization graph convolution. The input and output dimensions of this convolutional layer are both 1. .

[0021] Step 1.2, the usage flow of the timing module: Step 1.2.1: The input to this module is the information tensor of each trajectory. ,in Indicates batch size, Indicates the number of input channels. Represents time frame number and This indicates information about the current frame of the trajectory. The five dimensions represent the target's current frame information, the x-coordinate of the top-left corner of the target's bounding box, the y-coordinate, the target's length, and the target's width, respectively.

[0022] Step 1.2.1, convert the tensor Input to temporal convolution In order to extract time-domain features.

[0023] Step 1.2.2, convert the tensor after temporal convolution... Dimensional Input to 5-channel normalization layer This ensures a balanced distribution of the current frame information, x-coordinate, y-coordinate, length, and width values ​​of the trajectory object, thereby improving model convergence.

[0024] Step 1.2.3, normalize the tensor The nonlinear representation of features is enhanced by modifying the linear unit (ReLU) using a nonlinear activation function.

[0025] Step 1.2.4, convert the ReLU tensor Channel-level topology-optimized graph convolution CTRGC is used to further model the dynamic correlation between trajectory points and enhance feature representation.

[0026] Step 2, Fusion Module The construction and usage process is as follows Figure 2 As shown, where They represent The input dimensions and the expected output dimensions: Step 2.1, Construction of the fusion module: Step 2.1.1, Construct fused convolutions The input channel size of this convolution is The output channel size is The kernel size is (1, 5).

[0027] Step 2.1.2: Construct the Adaptive Graph Channel Attention (AGCA) module, where the input and output dimensions of the channels are both 256.

[0028] Step 2.1.3, Construct the batch normalization layer , The number of channels is .

[0029] Step 2.2, the usage process of the fusion module: Step 2.2.1: The input to this module is the information tensor of each trajectory processed by multiple time-series modules. ,in Indicates batch size, Indicates the number of input channels. Represents time frame number and This indicates information about the current frame of the trajectory. The five dimensions represent the target's current frame information, the x-coordinate of the top-left corner of the target's bounding box, the y-coordinate, the target's length, and the target's width, respectively.

[0030] Step 2.2.2, convert the tensor Input to temporal convolution In parentheses (.), different features of objects in the same time segment are fully integrated.

[0031] Step 2.2.3, convert the tensor after temporal convolution... The input is fed into the Adaptive Graph Channel Attention (AGCA) module to establish global dependencies of trajectory features along the channel dimension.

[0032] Step 2.2.3, convert the tensor after the adaptive graph channel attention module Input to batch normalization layer Furthermore, the ReLU activation function is used to further enhance the discriminative power of the features.

[0033] Step 3, the construction and usage process of the trajectory similarity calculation module (Tracklet Similarity Network, TSNetwork) is as follows: Figure 3 As shown: Step 3.1, Construction of the trajectory similarity calculation module: Step 3.1.1: Construct multi-dimensional temporal feature extraction modules TemporalModule_1 and TemporalModule_2 for trajectory 1 and trajectory 2 respectively. The module structures of trajectory 1 and 2 are identical, and their parameters are not shared. Each TemporalModule contains four modules. , cin The dimensions are 1, 32, 64, and 128 respectively. cout The dimensions are 32, 64, 128, and 256, respectively.

[0034] Step 3.1.2: Construct the FusionBlock fusion module for trajectory 1 and trajectory 2 respectively. cin and cout All dimensions are 256.

[0035] Step 3.1.3: Construct AdaptiveAvgPool2d(1, 1), where (1, 1) represents that the height and width after compression are both 1.

[0036] Step 3.1.4: Construct a fully connected classifier Classifier(256), where 256 indicates that the classifier receives a 256-dimensional trajectory feature vector and the output of the classifier is a binary classification probability.

[0037] Step 3.2, Using the trajectory similarity calculation module: Step 3.2.1: The input to this module is the information tensor of two trajectories. and ,in and The structure and steps in 1.2.1 Consistent.

[0038] Step 3.2.2, respectively and Input TemporalModule_1 and TemporalModule_2 to get the same result as in step 2.2.1. Consistent in structure and .

[0039] Step 3.2.3, will and Input the FusionBlock constructed in step 3.1.2 and the AdaptiveAvgPool2d constructed in step 3.1.3 respectively, and remove all dimensions with a channel of 1 to obtain the features of each trajectory. and ,in and All dimensions are 256 is the feature dimension. Indicates the batch size.

[0040] Step 3.2.4, will and The data is then input into the fully connected classifier (Classifier) ​​defined in step 3.1.4 to calculate the similarity between the two trajectories.

[0041] Step 3.2.5: Input the obtained similarity scores into a 1-dimensional Softmax function for normalization to obtain the final similarity score for the trajectory pairs. y .

[0042] Step 4, the trajectory pair association algorithm is as follows: Figure 4 As shown: Step 4.1: The input to the trajectory pair association algorithm is the preliminary tracking results and the trained TSNetwork.

[0043] Step 4.2: Extract each trajectory based on the preliminary tracking results and the trajectory identifier (IDentity, ID).

[0044] Step 4.3: Check the time interval and spatial distance for each pair of trackers in the tracking results, and select track pairs that meet the requirements. and Calculating similarity using TSNetwork and put 1- Fill it into the cost matrix cost_matrix.

[0045] Step 4.4: Use the Hungarian algorithm to match based on cost_matrix and update the trajectory ID to obtain the final tracking result.

[0046] The target tracking method proposed in this invention was tested using sequences from the MOT17 dataset, and the results are shown in Table 1. To highlight the advantages of this method, a comparison with StrongSORT was made. Compared with StrongSORT's trajectory pair association algorithm, this method not only integrates the positional information of targets in each frame of the trajectory in the temporal and fusion modules, but also comprehensively considers the length and width information of the targets. In addition, CTRGC and AGCA are introduced in the temporal and fusion modules respectively, realizing adaptive modeling of the temporal relationship of the trajectory. As shown in Table 1, the method in this invention outperforms StrongSORT in all three main evaluation metrics of multi-target tracking (high-order tracking accuracy, multi-target tracking precision, and identity F1 score), demonstrating the effectiveness of the improvements made in this invention.

[0047] Table 1 Experimental Results Method evaluation indicators StrongSORT The method in this invention High-order tracking accuracy 70.828 70.931 Multi-target tracking accuracy 78.670 78.955 Identity F1 score 83.353 83.378 Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art will be able to make various modifications and improvements without departing from the principles of the present invention, and these modifications and improvements should also be considered to fall within the scope of protection of the present invention.

Claims

1. A trajectory pair association method based on temporal sequence and fusion modules, characterized in that, Includes the following steps: Step 1: Construct a temporal module consisting of temporal convolution, channel normalization layer, and channel-level topology optimization graph convolution to extract dynamic features of trajectory changes over time; Step 2: Construct a fusion module consisting of fused convolutions, an adaptive graph channel attention module, and a batch normalization layer to fuse different features of objects in the same time segment; Step 3: Construct a trajectory similarity calculation module consisting of a multi-layer temporal module, a fusion module, an adaptive average pooling module, and a fully connected classifier to calculate the trajectory pair similarity score; Step 4: Use the trained trajectory similarity calculation module to extract the similarity between each trajectory pair in the preliminary tracking results, and use the Hungarian algorithm to perform matching to output the final multi-target tracking results.

2. The trajectory pair association method based on time series and fusion module according to claim 1, characterized in that, Step 1 is as follows: Step 1.1, Construction of the timing module: Step 1.1.1, construct temporal convolution; Step 1.1.2: Construct 5 channel normalization layers to process the 5 dimensions respectively; Step 1.1.3: Construct channel-level topology-optimized graph convolution; Step 1.2, Using the timing module: Step 1.2.1: Input the information tensor of each trajectory into the time series module; Step 1.2.1: Input the tensor into the temporal convolution to extract temporal features; Step 1.2.2: Input the tensor after temporal convolution into the 5-channel normalization layer dimension by dimension; Step 1.2.3: Correct the linear units of the normalized tensor using a nonlinear activation function; Step 1.2.4: Convolve the tensor that has been corrected by the linear unit through the channel-level topology-optimized graph.

3. The trajectory pair association method based on time series and fusion module according to claim 2, characterized in that, Step 2 is as follows: Step 2.1, Construction of the fusion module: Step 2.1.1, construct the fused convolution; Step 2.1.2: Construct the adaptive graph channel attention module; Step 2.1.3: Construct the batch normalization layer; Step 2.2, Using the fusion module: Step 2.2.1: The information tensor of each trajectory is processed by multiple time-series modules as input to the fusion module; Step 2.2.2: Input the input tensor into the temporal convolution to fully fuse the different features of objects in the same time segment; Step 2.2.3: Input the tensor after temporal convolution into the adaptive graph channel attention module; Step 2.2.3, convert the tensor after the adaptive graph channel attention module The input is fed into a batch normalization layer, and the activation function is modified by linear units to further enhance the discriminative power of the features.

4. The trajectory pair association method based on time series and fusion module according to claim 3, characterized in that, Step 3 specifically involves: Step 3.1, Construction of the trajectory similarity calculation module: Step 3.1.1: Construct multidimensional temporal feature extraction modules for the two trajectories respectively. The multidimensional temporal feature extraction modules for the two trajectories have the same structure and do not share parameters. Step 3.1.2: Construct the fusion module for the two trajectories respectively; Step 3.1.3: Construct adaptive average pooling; Step 3.1.4: Construct a fully connected classifier; Step 3.2, Using the trajectory similarity calculation module: Step 3.2.1: Input the information tensors of the two trajectories into the trajectory similarity calculation module; Step 3.2.2, respectively Input the corresponding multidimensional temporal feature extraction module to obtain a structure consistent with that in step 2.2.

1. ; Step 3.2.3: Input the information tensor obtained in the previous step into the corresponding fusion module constructed in step 3.1.2 and the adaptive average pooling constructed in step 3.1.3 respectively, and remove all dimensions with channels of 1 to obtain two trajectory features; Step 3.2.4, will The data is then fed into the fully connected classifier defined in step 3.1.4 to calculate the similarity between trajectories. Step 3.2.5: Input the obtained similarity into a Softmax function with dimension 1 for normalization to obtain the final similarity score of the trajectory pairs.

5. The trajectory pair association method based on time series and fusion module according to claim 4, characterized in that, Step 4 is as follows: Step 4.1: Extract each trajectory based on the preliminary tracking results; Step 4.2: Use the trained trajectory similarity calculation module to extract the similarity between each trajectory pair in the preliminary tracking results; Step 4.3: Check the time interval and spatial distance of each pair of trajectories in the tracking results. For the trajectory pairs that meet the requirements, calculate the similarity using the trajectory similarity calculation module. and put 1- Fill it into the cost matrix; Step 4.4: Use the Hungarian algorithm to match the cost matrix and update the trajectory identifiers to obtain the final tracking result.

Images