A multi-target tracking trajectory association method based on a graph neural network

Abstract

The application provides a multi-target tracking trajectory association method based on a graph neural network, constructs a data set for graph neural network training composed of four trajectory types, the data set can enable the network to learn various condition modes when the target moves, so that the trajectories can still be associated when the trajectories are blocked or missed. The graph neural network can fully aggregate the target information of multiple frames before and after, express the potential association between the target and the target, and find out the position of the cut-off and whether the two trajectories need to be spliced together when the existing trajectory has an identity switching problem. The application uses the same graph neural network to process trajectory cutting and trajectory splicing, reduces the number of times of training the trajectory association model, and eliminates the need to input a fixed trajectory length to the graph neural network. The experiment on the MOT17 public data set combined with multiple trackers shows that the performance of the existing tracker is improved.

Description

Technical Field

[0001] This invention relates to multi-target tracking technology, and particularly to multi-target tracking trajectory association technology based on graph neural networks. Background Technology

[0002] The main task of multi-object tracking is to detect multiple targets in a video frame and effectively correlate them across frames to obtain the continuous motion trajectory of each target in the video. Multi-object tracking lays the foundation for subsequent high-level tasks such as cross-camera multi-object tracking, abnormal behavior recognition, and biometric counting. Currently, multi-object tracking has demonstrated its development potential and economic benefits in multiple fields such as intelligent video surveillance, sports analytics, and autonomous driving.

[0003] In recent years, a series of tracking algorithms have emerged, but they still erroneously associate the trajectories of different targets together or assign different identities to trajectories from the same target due to the complexity of target motion and mutual occlusion between targets. To address these issues, a trajectory association method has been proposed. This method is applied to already generated trajectories, first truncating incorrectly associated trajectories to ensure that each short trajectory no longer suffers from identity switching problems, and then re-associating all short trajectories, thereby ultimately reducing the occurrence of identity switching problems. However, existing methods use zero-padding or sliding window methods to ensure that the trajectory length input to the network is consistent. But since the trajectory length is not fixed, using a fixed input length method is not reasonable. At the same time, existing methods treat trajectory truncation and trajectory concatenation as two independent tasks, thus requiring two separate networks to perform these two tasks, which increases the training cost. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multi-target tracking trajectory association method that can complete both trajectory truncation and trajectory splicing using the same trajectory association model.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is a multi-target tracking trajectory association method based on graph neural networks, comprising the following steps:

[0006] Training data construction steps: Construct training datasets for four trajectory types, including time-continuous identity-consistent trajectories, time-discontinuous identity-consistent trajectories, time-discontinuous identity-inconsistent trajectories, and time-continuous identity-inconsistent trajectories; Each frame of the target data for each trajectory in the training dataset contains the true association state between the target and the targets in the preceding and following frames; the association state includes truncation and continuity;

[0007] The trajectory association model training steps are as follows: The trajectory association model receives and extracts image features of the target in each frame from the input trajectory. These image features, along with the target's position and time features, form initial node features. Each target in the trajectory is treated as a node, and edges are constructed between adjacent targets to form initial edge features, transforming the input trajectory into a graph structure. The initial node and edge features in the graph structure are then encoded. These initial node and edge feature encodings are iteratively executed through a graph convolutional network using a series of message passing operations to obtain node features that perceive information about connecting edges and edge features that perceive information about the nodes at both ends. Edge features are then used for edge prediction to obtain prediction scores. These prediction scores determine the association state of each edge in the graph structure, thereby determining whether to truncate or splice the current edge. During training, the training process of the trajectory association model is constrained by the real association state of the training dataset as the training objective.

[0008] Tracking trajectory association steps: Input the trajectory data to be predicted into the trained trajectory association model. The trajectory association model outputs the prediction score of the edge in the graph structure corresponding to the trajectory. The association state of each edge in the graph structure is determined by the prediction score, thereby determining whether to truncate or splice the trajectory containing the edge.

[0009] This invention constructs a dataset for training graph neural networks, consisting of four trajectory types. This dataset enables the network to learn various status patterns of target movement, allowing it to maintain correlation even when trajectories are occluded or missed. The graph neural network effectively aggregates target information from multiple frames to express potential relationships between targets. When existing trajectories experience identity switching issues, the network can identify the truncation point and indicate whether two trajectories need to be joined together.

[0010] Specifically, the construction methods for the four trajectory types are as follows:

[0011] 1) Time-continuous identity consistency trajectory:

[0012] 1-1) Randomly select a trajectory from a video;

[0013] 1-2) Randomly extract this trajectory within the first set frame length range; the first set frame length is the frame length corresponding to the trajectory in the training process.

[0014] 1-3) In the graph structure corresponding to the trajectory, the associated state of each edge is set to continuous;

[0015] 2) Consistent identity trajectories with discontinuous time:

[0016] 2-1) Randomly select a trajectory from a video;

[0017] 2-2) Randomly extract this trajectory within half the length of the first set frame;

[0018] 2-3) Skip the second set frame length at the end of the intercepted trajectory, then select a trajectory that is half the length of the first set frame, and stitch the two trajectories together into one trajectory; the second set frame length is the empirical value of the set trajectory break length.

[0019] 2-4) Set the association state of each edge in the graph structure corresponding to the spliced ​​trajectory to continuous;

[0020] 3) Temporally discontinuous identity inconsistency trajectories: These represent trajectories where there is a clear spatiotemporal discontinuity and significant differences in appearance before and after the discontinuity, indicating a change in identity. The trajectory association model learns that these types of trajectories do not need to be truncated.

[0021] The construction method is as follows:

[0022] 3-1) Randomly select a video;

[0023] 3-2) First, randomly select a trajectory from the video whose length is within half the length of the first set frame. Then, randomly select another trajectory whose length is within half the length of the first set frame. Modify the time information of the two trajectories to ensure that the beginning and end of the two trajectories do not exceed the length of the second set frame. Then, stitch the two trajectories together into one trajectory.

[0024] 3-3) The trajectory of the splicing is set to truncated for the edge in the graph structure corresponding to the splicing point, and the association state of other edges is set to continuous.

[0025] 4) Time-continuous identity inconsistency trajectories:

[0026] 4-1) Filter out single targets in a single frame whose occlusion rate is greater than the occlusion ratio threshold based on the occlusion rate;

[0027] 4-2) Using the selected target as a benchmark, find another target with the largest coverage area;

[0028] 4-3) Randomly select the trajectory of the reference target and the coverage target within half the length of the first set frame, one within half the length of the first set frame, and the other within half the length of the first set frame. Then, stitch the two trajectories together to form a single trajectory.

[0029] 4-4) The edge association state of the spliced ​​trajectory in the graph structure corresponding to the splicing point is set to truncated, and the association state of other edges is set to continuous.

[0030] In the trajectory association step, when the trajectory association model is used to predict trajectory truncation, the input trajectory to be predicted is first constructed as a graph structure. The association state of each edge is obtained by predicting the score of each edge in the graph structure. Trajectory truncation is performed at the position corresponding to the association state of truncation.

[0031] In the trajectory association step, when the trajectory association model is used to predict trajectory splicing, the two trajectories to be predicted are first connected end to end in chronological order to form a spliced ​​trajectory and the splicing position is recorded before being input into the trajectory association model. The trajectory association model constructs the spliced ​​trajectory into a graph structure, obtains the association state of each edge by predicting the score of each edge in the graph structure, and finally determines whether the two trajectories need to be spliced ​​together by judging the association state of the splicing position.

[0032] The beneficial effects of this invention are that it utilizes the same graph neural network to handle trajectory truncation and trajectory splicing, reducing the number of training iterations for the trajectory association model and removing the limitation of requiring a fixed trajectory length as input to the graph neural network. Experiments on the MOT17 public dataset using multiple trackers demonstrate that the method of this invention can improve the performance of existing trackers. Attached Figure Description

[0033] Figure 1 This is a graph of the trajectory association framework based on graph neural networks;

[0034] Figure 2 This is a neural network structure diagram of the trajectory association graph. Detailed Implementation

[0035] The following provides a detailed description of how to implement this invention.

[0036] I. Construct a training dataset for training a trajectory association model based on a graph neural network.

[0037] There are four trajectory types: time-continuous identity-consistent trajectory, time-discontinuous identity-consistent trajectory, time-discontinuous identity-inconsistent trajectory, and time-continuous identity-inconsistent trajectory. Each trajectory is assigned a unique identifier ID.

[0038] 1) Temporally continuous identity-consistent trajectories: These patterns represent trajectories where movement is normal and there is no identity switching. These trajectories constitute the majority in videos, and the trajectory association model learns that they should not be truncated.

[0039] The construction method is as follows:

[0040] 1-1) In the MOT17 training dataset, randomly select a trajectory from a video;

[0041] 1-2) Randomly extract this trajectory within the frame length range of [10, 30]; wherein, the frame length range can be adjusted according to actual needs, and in this embodiment, the length of each trajectory used in the training process is set to [10, 30];

[0042] 1-3) The training objective for graph nodes is the ID of the trajectory, and the training objective for edges is their associated state set to 0, which means no truncation. Setting the ID is to supervise the features of each graph node during training, so that they are constrained to their own features, and the edge features after message passing are more discriminative.

[0043] 2) Temporally discontinuous identity-consistent trajectories: These represent trajectories belonging to the same identity that break due to detector misses or complete target occlusion. The trajectory association model learns that these trajectories do not need to be truncated.

[0044] The construction method is as follows:

[0045] 2-1) In the MOT17 training dataset, randomly select a trajectory from a video;

[0046] 2-2) Randomly extract this trajectory within the frame length range of [5,15];

[0047] 2-3) Skip frames [2,50] at the end of the intercepted trajectory, then select a trajectory of length [5,15] frames, and stitch the two trajectories together as a single trajectory; [2,50] is the normal frame length range of the observed trajectory break. First, intercept [5,15] frames, then intercept [5,15] frames again so that the frame length of the constructed trajectory is still [10,30].

[0048] 2-4) The training objective for graph nodes is the ID of the trajectory, and the training objective for edges is their associated state, which is set to 0, representing no truncation.

[0049] 3) Temporally discontinuous identity inconsistency trajectories: These represent trajectories where there is a clear spatiotemporal discontinuity and significant differences in appearance before and after the discontinuity, indicating a change in identity. The trajectory association model learns that these types of trajectories do not need to be truncated.

[0050] The construction method is as follows:

[0051] 3-1) Randomly select a video from the MOT17 training dataset;

[0052] 3-2) First, randomly select a trajectory from the video with a length controlled within [5,15] frames, and then randomly select another trajectory with a length also controlled within [5,15] frames. Modify the time information t of the two trajectories to ensure that the first and last trajectories of the two trajectories do not exceed the length of [2,50] frames, and then stitch the two trajectories together into one trajectory.

[0053] 3-3) The training objective of the graph node is the ID corresponding to the trajectory before splicing. The training objective of the edge needs to be set to 1 at the splicing point, which means that it needs to be truncated. The other associated states are set to 0, which means that it is not truncated.

[0054] 4) Temporally Continuous Identification Inconsistent Trajectories: The goal is for the network to learn these difficult-to-truncate samples. The trajectories are temporally and spatially continuous, and as targets approach each other, their appearance features become more similar. However, the network should have sufficient hard-sample discrimination capability to truncate these trajectories at occlusion points. The construction method is as follows:

[0055] 4-1) In the MOT17 training dataset, select single targets in a single frame with an occlusion rate greater than 0.5 based on the occlusion rate.

[0056] 4-2) Using the selected target as a benchmark, find another target with the largest coverage area. The benchmark target is denoted as Target 1, and the covered target is denoted as Target 2.

[0057] 4-3) Randomly search forward [5,15] frames to capture the trajectory of target 1 and target 2, and search backward [5,15] frames to capture the trajectory of target 2. Then, the two trajectories are spliced ​​together to form a single trajectory.

[0058] 4-4) The training objective of the graph node is the ID corresponding to the trajectory before splicing. The training objective of the edge needs to be set to 1 in the splicing association state, which means that it needs to be truncated. The other association states are set to 0, which means that it is not truncated.

[0059] II. Design of a trajectory association model based on graph neural networks.

[0060] The overall framework of the model is as follows Figure 1 As shown, it includes a Convolutional Neural Network (CNN) for graph construction, an encoder (Enc) for feature encoding, and a Graph Neural Network (GNN) for message passing and edge prediction.

[0061] CNN receives the input trajectory, extracts the image features of the target in each frame of the trajectory, and combines the image features with the target's position and time features to form the initial node features. Each target in the trajectory is used as a node, and edges are constructed between two adjacent targets to form the initial edge features, thereby transforming the input trajectory into a graph structure.

[0062] Enc receives the graph structure of the trajectory, encodes the initial node features and initial edge features to ensure that the feature dimensions are basically consistent, and outputs the initial node feature codes and initial edge feature codes.

[0063] GNN receives the initial node feature encoding and initial edge feature encoding in the graph structure, and then iteratively executes a series of message passing steps to obtain information including the node-aware connection edge and the edge-aware features of the two end nodes. It performs edge prediction on the edge features in the graph structure that contain the context information of the position, outputs the associated state value of each edge, and finally determines whether to truncate or splice based on the associated state value.

[0064] The implementation steps of the trajectory association model are as follows: Figure 2 As shown:

[0065] (1) Graph construction

[0066] First, the trajectory is constructed as a graph. For each target o in the trajectory t, image features are extracted using a CNN to obtain the appearance feature vector f for each target. app Based on the location of each target in the image and its timestamp in the video, the target's location feature f can be obtained. pos and time features f time Therefore, each target in the trajectory can be uniquely represented by its appearance, motion, and temporal features. Where f app f is the 2048-dimensional appearance feature vector of the target. pos The temporal feature f is represented by the target's x and y coordinates in the image and its frame number in the video. time .

[0067] We treat each target in a single trajectory as a node, and the initial node features are the appearance features f of the target. app An edge is constructed between two adjacent targets, and the initial edge features describe the relationship between the targets. The initial node features and initial edge features are constructed as follows:

[0068]

[0069]

[0070] Where i and j represent the i-th node and the j-th node, respectively. Represents the i-th node n i The initial node characteristics, The edge e between the i-th node and the j-th node represents... i,j The initial edge features, Let represent the appearance feature vector of the i-th node. x i y iLet x and y represent the temporal features and x and y coordinates of the i-th node, respectively. Our initial edge features use the time difference between consecutive frames as the temporal distance between adjacent targets, the positional difference between consecutive targets as the distance between targets, and the cosine distance metric of the appearance features of consecutive targets as the appearance similarity between targets.

[0071] (2) Feature encoding

[0072] To construct the features of nodes and edges in a graph, we should first keep the dimensions of node features and edge features similar. However, the initial node features extracted by CNN have 2048 dimensions, while the constructed edge features only have 4 dimensions. This dimensional imbalance will affect our message aggregation effect.

[0073]

[0074]

[0075] Therefore, as shown in the above equation, we will encode the features using the Multilayer Perceptron (MLP) encoding algorithm. and Features of the initial nodes respectively and initial edge features Encoding is performed to obtain the initial edge feature code. and initial edge feature encoding Used as input for layer 0 of a graph neural network.

[0076] (3) Message passing

[0077] The purpose of message passing is to learn the information contained in the feature vectors of nodes and edges in the graph, allowing each edge and node to fully perceive the surrounding nodes and edges. Each message passing process consists of two updates: one from node to edge (n→e), and the other from edge to node (e→n). The number of iterations is fixed at L, which is set to 5. For each iteration l∈{0,...,L-1}, the update content is as shown in Formula 3:

[0078]

[0079]

[0080] In this context, [·] represents a splicing operation. and Represents a learnable network. Parameters are shared across different iteration layers in the diagram. They also share parameters. N iLet represent the neighboring nodes of the i-th node, and SUM denotes the aggregation operation that sums the surrounding information. After L iterations, each node contains information about all other nodes and edges at a distance of L in the graph. L is similar to the receptive field in a CNN, allowing each node to acquire contextual information about its location. The output edge feature vector of the L-th iteration is used as the edge feature for edge prediction.

[0081] (4) Side prediction

[0082] For each edge e i,j We use N cls Calculate whether the edges are connected. N cls The MLP and Sigmoid function are used to output a single value.

[0083]

[0084] Therefore, when the edge predicts the score P edge When the value is higher than c, it is assumed that no edge connection is needed, and the association state is set to 1, thus truncating the trajectory; otherwise, the trajectory is connected, and the association state is set to 0. c is an adjustable parameter.

[0085] When training the trajectory association model, the trajectory input to the model is obtained through the training dataset construction step, and the trajectory length is [10, 30]. Once trained, the trajectory association model does not require restrictions on the input trajectory when used.

[0086] When a trajectory association model is used to predict trajectory truncation, a CNN constructs each trajectory as a graph structure, where adjacent targets are connected by edges. Node and edge features are built based on the temporal, spatial, and appearance information of each target. This graph structure is fed into a GNN, which ultimately determines the specific truncation location by predicting all edges. If the network's predictions indicate that some edges are disconnected, the trajectory will naturally be truncated into multiple shorter trajectories. However, sometimes the trajectories themselves are consistent and do not require truncation.

[0087] In trajectory stitching scenarios, once short trajectories are obtained, a pairwise pre-stitching method is used to re-associate trajectories of the same target. This means that any two short trajectories are stitched together end-to-end according to their chronological order to form a new trajectory, and the stitching position is recorded. When the trajectory association model is used for trajectory stitching prediction, a CNN is also used to construct a graph structure of the stitched trajectories, where adjacent targets are connected by edges. Node features and edge features are constructed using the temporal, spatial, and appearance information of each target. This graph structure is also fed into a GNN, which ultimately determines whether the two trajectories need to be stitched together by predicting the edges at the stitching point.

[0088] Third, the invention was trained on the MOT17 dataset, and the effectiveness of the invention was verified through experiments.

[0089] We utilize four advanced multi-target tracking methods as trackers. The specific experimental setup includes tracking targets in the MOT17 test dataset using these trackers, obtaining the tracked trajectories, and evaluating their performance metrics. Secondly, we use the tracking results of each tracker as input to the trajectory association model of this invention to further optimize the generated trajectories and improve performance. The comparative experimental results on the MOT17 test set are shown in Table 1. It can be seen that after introducing this method, the existing trackers DeepSORT, CSTrack, ByteTrack, and BOT-SORT all show improvements in multi-target tracking performance metrics HOTA, IDF1, MOTA, and IDs, particularly in improving trajectory association accuracy and identity recognition. This demonstrates the potential of this method in the field of multi-target tracking and its ability to improve the performance of existing tracking systems.

[0090]

[0091] Table 1.

Claims

1. A method for multi-target tracking trajectory association based on graph neural networks, characterized in that, Including the following steps: Training data construction steps: Construct training datasets for four trajectory types, including time-continuous identity-consistent trajectories, time-discontinuous identity-consistent trajectories, time-discontinuous identity-inconsistent trajectories, and time-continuous identity-inconsistent trajectories; Each frame of the target data for each trajectory in the training dataset contains the true association state between the target and the targets in the preceding and following frames; the association state includes truncation and continuity; The trajectory association model training steps are as follows: The trajectory association model receives and extracts image features of the target in each frame from the input trajectory. These image features, along with the target's position and time features, form initial node features. Each target in the trajectory is treated as a node, and edges are constructed between adjacent targets to form initial edge features. The input trajectory is then transformed into a graph structure. The initial node features and initial edge features in the graph structure are then encoded. Node feature encoding and initial edge feature encoding are obtained by iteratively executing a series of message passing through a graph convolutional network. This process yields node features that perceive information about connected edges and edge features that perceive information about the two endpoints. The edge features are then used to predict edges and obtain prediction scores. The prediction scores are used to determine the association state of each edge in the graph structure, thereby determining whether to truncate or splice the current edge. During training, the training process of the trajectory association model is constrained by using the real association state of the training dataset as the training objective. Tracking trajectory association steps: Input the trajectory data to be predicted into the trained trajectory association model. The trajectory association model outputs the prediction score of the edge in the graph structure corresponding to the trajectory. The association state of each edge in the graph structure is determined by the prediction score, thereby determining whether to truncate or splice the trajectory containing the edge.

2. The method as described in claim 1, characterized in that, The specific construction methods for the four trajectory types are as follows: 1) Time-continuous identity consistency trajectory: 1-1) Randomly select a trajectory from a video; 1-2) Randomly extract this trajectory within the first set frame length range; the first set frame length is the frame length corresponding to the trajectory in the training process. 1-3) In the graph structure corresponding to the trajectory, the associated state of each edge is set to continuous; 2) Consistent identity trajectories with discontinuous time: 2-1) Randomly select a trajectory from a video; 2-2) Randomly extract this trajectory within half the length of the first set frame; 2-3) Skip the second set frame length at the end of the intercepted trajectory, then select a trajectory that is half the length of the first set frame, and stitch the two trajectories together into one trajectory; the second set frame length is the empirical value of the set trajectory break length. 2-4) Set the association state of each edge in the graph structure corresponding to the spliced ​​trajectory to continuous; 3) Discontinuous time identity inconsistency trajectory: This is used to represent trajectory types where there is a clear spatiotemporal discontinuity and a huge difference in appearance before and after, indicating that an identity switch has occurred. The trajectory association model learns that this type of trajectory does not need to be truncated. The construction method is as follows: 3-1) Randomly select a video; 3-2) First, randomly select a trajectory from the video whose length is within half the length of the first set frame. Then, randomly select another trajectory whose length is within half the length of the first set frame. Modify the time information of the two trajectories to ensure that the beginning and end of the two trajectories do not exceed the length of the second set frame. Then, stitch the two trajectories together into one trajectory. 3-3) The trajectory of the splicing is set to truncated for the edge in the graph structure corresponding to the splicing point, and the association state of other edges is set to continuous. 4) Time-continuous identity inconsistency trajectories: 4-1) Filter out single targets in a single frame whose occlusion rate is greater than the occlusion ratio threshold based on the occlusion rate; 4-2) Using the selected target as a benchmark, find another target with the largest coverage area; 4-3) Randomly select the trajectory of the reference target and the coverage target within half the length of the first set frame, one within half the length of the first set frame, and the other within half the length of the first set frame. Then, stitch the two trajectories together to form a single trajectory. 4-4) The edge association state of the spliced ​​trajectory in the graph structure corresponding to the splicing point is set to truncated, and the association state of other edges is set to continuous.

3. The method as described in claim 2, characterized in that, Each trajectory is assigned a unique identifier ID; when constructing the four trajectory types, the training target set for the nodes in the graph structure is the ID corresponding to the current location before trajectory splicing.

4. The method as described in claim 2, characterized in that, The first set frame length range is [10, 30], and the second set frame length range is [2, 50].

5. The method as described in claim 2, characterized in that, The occlusion ratio threshold is 0.

5.

6. The method as described in claim 1, characterized in that, In the trajectory association step, when the trajectory association model is used to predict trajectory truncation, the input trajectory to be predicted is first constructed as a graph structure. The association state of each edge is obtained by predicting the score of each edge in the graph structure. Trajectory truncation is performed at the position corresponding to the association state of truncation.

7. The method as described in claim 1, characterized in that, In the trajectory association step, when the trajectory association model is used to predict trajectory splicing, the two trajectories to be predicted are first connected end to end in chronological order to form a spliced ​​trajectory and the splicing position is recorded before being input into the trajectory association model. The trajectory association model constructs the spliced ​​trajectory into a graph structure, obtains the association state of each edge by predicting the score of each edge in the graph structure, and finally determines whether the two trajectories need to be spliced ​​together by judging the association state of the splicing position.

8. The method as described in claim 1, characterized in that, The image features of the target in each frame are obtained through a convolutional neural network (CNN). The temporal feature is the frame number of the target in the video, and the positional feature is the horizontal and vertical coordinates of the target in the image.

9. The method as described in claim 1, characterized in that, The initial node features and initial edge features in the graph structure are encoded using a multilayer perceptron (MLP).

Images