A multi-target tracking method based on motion direction consistency interaction features
By constructing a trajectory-detection heterogeneous graph with consistent motion direction in a multi-target tracking algorithm and extracting interaction features using a two-layer graph neural network, the problem of insufficient description of the interaction relationship between targets in complex scenarios is solved, thereby improving tracking accuracy and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA NORTH VEHICLE RES INST
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-16
AI Technical Summary
Existing multi-target tracking methods cannot fully describe the interaction relationships between targets in complex dynamic scenarios, leading to problems such as identity switching and trajectory breakage.
By constructing a trajectory-detection heterogeneous graph based on motion direction consistency, extracting interaction features using a two-layer graph neural network, and fusing motion direction consistency information, the multi-target tracking algorithm is improved.
It improves the accuracy and stability of multi-target tracking and reduces the occurrence of identity switching and trajectory breakage.
Smart Images
Figure CN122223055A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and multi-target tracking technology, specifically relating to a target relationship perception and tracking method based on motion direction consistency interaction feature modeling, which is particularly suitable for dynamic tracking tasks in complex scenes where multiple targets have cooperative behaviors or interactive relationships. Background Technology
[0002] Multi-object tracking is an important research area in computer vision, and it is widely used in fields such as intelligent surveillance and autonomous driving. Common multi-object tracking methods associate trajectories based on the similarity of the appearance features and motion features (i.e., the similarity of positional information) of the targets to complete the tracking task. However, in practical applications, targets are prone to problems such as occlusion, intersection, dense distribution, and similar appearance, which makes traditional appearance- or position-based matching strategies susceptible to errors such as identity switching.
[0003] In recent years, with the successful application of Graph Neural Networks (GNNs) in relation modeling, Papakis et al. proposed the Graph Convolutional Neural Networks Match (GCNNMatch) algorithm. This algorithm utilizes trajectory and distance information between detected targets to construct a graph and uses a Graph Convolutional Network (GCN) to extract the appearance interaction features of the targets. Liang et al. proposed a multi-object tracking algorithm based on a neighbor graph (MOTNG), which constructs a graph of the target and its neighbors that are close to the target in the current frame and extracts interaction features containing the features of neighbors in the same frame. Zhang et al. proposed a multi-object tracking method based on directed reliable neighbor graphs (Improving the Second Association for Multi-Object Tracking by Directed Reliable Neighbor Graphs). This method uses two types of edges to represent neighbor relationships and temporal relationships respectively to construct a directed reliable neighbor graph, and uses a two-layer Graph Attention Network (GAT) to update the graph to obtain neighbor interaction features containing temporal features.
[0004] However, the aforementioned methods only consider distance between targets during mapping, neglecting differences in their motion directions, making it difficult to effectively represent complex dynamic interactions between targets. When the nearest neighbor of a target moves in the opposite direction or shows a clear separation trend, relying solely on spatial distance as the association criterion can easily lead to incorrect adjacency relationships, distorting subsequent feature propagation and matching results and affecting the stability and accuracy of multi-target tracking. Especially in dense or intersecting scenes, this mapping method that ignores directional relationships is more prone to problems such as identity switching and trajectory breakage, reducing overall tracking performance. Summary of the Invention
[0005] The technical problem this invention aims to solve is the inability to fully describe the interaction relationships between targets in complex dynamic scenarios.
[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows: A multi-target tracking method based on motion direction consistency interaction features includes the following steps: Step 1: Train the interaction feature extractor: Step 1.1: Input the video image and its YOLOX target detection results, wherein the target detection results are derived from ( t , x , y , w , h ) indicates that among them x , y These are the x and y coordinates of the top left corner of the target bounding box, respectively. w , h These are the width and height of the target bounding box, respectively. t For timestamps; Step 1.2: Based on the YOLOX target detection results, crop the target image patch in the video, input the image patch into the pre-trained ReID module, extract the 1024-dimensional appearance features of each target, and use the average appearance features of the objects in the trajectory as the trajectory appearance features; Step 1.3: For t Arbitrary trajectory object in frame -1 i and t Any detected object in the frame j If the distance between the two is less than a preset distance threshold, then the motion vectors of the two are calculated and normalized, and then the cosine value of the angle between the normalized motion vectors is calculated. Based on the sign of the cosine value, the trajectory object is... i With the test object j The edges between them are classified into sets of edges moving in the same direction. Or non-unidirectional motion edge set The cosine value of the included angle is used as the weight of the corresponding side; Step 1.4, repeat step 1.3, until all trajectory-detection pairs with distances less than the threshold have been partitioned into edge weights and edge types, constructing a trajectory-detection heterogeneous graph. G =( H , E ),in H The node features are represented by 2048-dimensional features and 4-dimensional motion features of the trajectory or detected object. x , y , w , h It is pieced together. ; Step 1.5, generate the trajectory-detection heterogeneous graph. G A two-layer graph neural network is input to encode node features layer by layer, aggregate messages, and fuse features to obtain node interaction features that integrate multi-layer interaction information. The two-layer graph neural network is trained using the Adam optimizer and the weighted binary cross-entropy loss function to complete the training of the interaction feature extractor. Step 2: Perform multi-target tracking: Step 2.1: Input the video to be tracked frame by frame, and use the YOLOX detector to obtain the target detection results for each frame; Step 2.2: Use the ReID module to extract the appearance features of each trajectory and the detected object, where the appearance feature of the trajectory is the mean of the appearance features of each object in the trajectory; Step 2.3: Using the interaction feature extractor trained in Step 1, extract the interaction features of each trajectory and the detected object; Step 2.4: Calculate the Intersection over Union (IoU) score, appearance feature cosine similarity, and interaction feature similarity between each trajectory and each detected object. Use a weighted fusion method to fuse the three to obtain a similarity score, and then construct a similarity matrix. Step 2.5: Input the similarity matrix into the Hungarian algorithm to obtain the matching results between the trajectory and the detected object; Step 2.6: Repeat steps 2.1 to 2.5 until the last frame of the video to be tracked is processed to obtain the complete multi-object tracking result.
[0007] Furthermore, in step 1.3, the motion vector is calculated as follows: trajectory object i motion vector ; Detection object j motion vector ； The normalized motion vectors are as follows: , In the formula Represents the magnitude of a vector. A very small constant to prevent division by zero errors; Cosine value of the angle between the motion vectors ; The rules for classifying edges and determining their weights are as follows: ; This represents the edge weight.
[0008] Furthermore, in step 1.5, the processing of node features by the two-layer graph neural network includes: Step 1.5.1: Using the formula Encode node features: In the formula, This represents the encoded node features. Indicates the initial node characteristics. and The weights that need to be trained; Step 1.5.2: For each node i Calculate aggregated messages of motion in the same direction : ; In the formula, Represents a set of neighbors. j Indicates the neighbor node index. Represents a node j In the l The feature vector of the layer, For trainable parameters, Attention weight ; In the formula, For the weights to be trained, exp(.) represents the exponential function, and LeakyReLU(.) is the linear rectified function with leakage; Step 1.5.3: Using the same method as in Step 1.5.2, aggregate messages from the non-co-directional neighbors of each node to obtain the non-co-directional aggregated message. The attention weights are calculated in the same way as those for edges in the same direction, but using independent trainable parameters. and ; Step 1.5.4, using the formula The messages from heterogeneous edges and the node's own characteristics are fused. and For the weights and biases that need to be trained, It is a non-linear activation function.
[0009] Step 1.5.5: Repeat steps 1.5.1 to 1.5.4 to complete the feature update of the two-layer graph neural network and obtain the final interaction features of the nodes. Each layer of the graph neural network has an input and output dimension of 512 and uses a single-head mode.
[0010] Furthermore, in step 2.4, the formula for fusion of similarity scores is: ; in, For trajectory and detection Similarity score, The IoU scores of the two The cosine similarity of the appearance features of the two is... The similarity of their interaction features. a and b All are manually adjusted weighting coefficients, and satisfy the following conditions: .
[0011] The present invention has the following advantages: by fusing motion direction consistency to obtain more reliable interaction features, it can effectively improve the accuracy and stability of multi-target tracking and reduce identity switching. Attached Figure Description
[0012] Figure 1 This invention relates to a multi-target tracking method based on motion direction consistency interaction features. Detailed Implementation
[0013] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.
[0014] This embodiment designs a multi-target tracking technique based on motion direction consistency interaction features, using the motion direction consistency between targets in the tracking process. The approach is as follows: First, a detector and a ReID module are used to detect targets and extract their appearance features. Then, based on the trajectory and the motion direction consistency between detected targets, co-directional and non-co-directional motion edges are constructed, thus building a trajectory-detection heterogeneous graph. Next, a graph neural network is used to update the trajectory-detection heterogeneous graph to obtain interaction features. Finally, the Intersection over Union (IoU) score, interaction features, and appearance feature similarity are fused to obtain a similarity matrix, and the Hungarian algorithm is used to obtain the matching results.
[0015] The multi-target tracking method based on motion direction consistency interaction features in this embodiment has the following specific steps: Step 1, Interactive Feature Extractor Training: Let the input be a video image and its YOLOX target detection results. Let the detection result for each target be derived from (…). t , x , y , w , h ) indicates that among them x , y These represent the x and y coordinates of the top-left corner of the target bounding box, respectively. w , h These represent the width and height of the target bounding box, respectively. t Represents a timestamp.
[0016] Step 1.1: Using the object detection results from YOLOX, image patches corresponding to each object are cropped from the input video. Each image patch is input into the ReID module trained by Aharon et al. to obtain the 1024-dimensional appearance features of each object, and the average features of the objects in the trajectory are used as the trajectory appearance features.
[0017] Step 1.2, construct edges and calculate edge weights: For t Any trajectory object in frame -1 i and t Any one detected object in the frame j If the distance between them is less than the distance threshold, then: Step 1.2.1, calculate the object using formula (1) and formula (2). i and j motion vector : (1) (2) Step 1.2.2: Normalize the motion vector of each object using formula (3): (3) In the formula Represents the magnitude of a vector. A very small constant to prevent division by zero errors.
[0018] Step 1.2.3, use formula (4) to calculate the angle between the two vectors: (4) Step 1.2.4: Update the set of edges moving in the same direction using formula (5). Non-co-directional motion edge set : (5) In the formula, the edges are divided into two categories by the cosine of the angle between the vectors, and the set of edges moving in the same direction. Non-co-directional motion edge set and will As the value of the edge weight.
[0019] Step 1.3, repeat step 1.2 until all trajectories less than the distance threshold have been calculated - detect the edge weights and edge types between pairs.
[0020] Step 1.4, Constructing a trajectory-detection heterogeneous graph G =( H , E ),in H The node features are represented by 2048-dimensional features and 4-dimensional motion features of the trajectory or detected object. x , y , w , h It is pieced together. .
[0021] Step 1.5, draw the diagram G Updating interaction features in an input graph neural network: Step 1.5.1: Encode the node features using formula (6): (6) In the formula, This represents the encoded node features. Indicates the initial node characteristics. and These are the weights that need to be trained.
[0022] Step 1.5.2, for each node i Formula (7) is used to calculate the aggregated messages of the same direction of motion. : (7) In the formula, Represents a set of neighbors. j Indicates the neighbor node index. Represents a node j In the l The feature vector of the layer, For trainable parameters, attention weights Calculated using equation (8): (8) In the formula, Here are the weights that need to be trained, exp(.) represents the exponential function, and LeakyReLU(.) is the linear rectified function with leakage.
[0023] Step 1.5.3: Using the same method as in Step 1.5.2, aggregate messages from the non-co-directional neighbors of each node to obtain the non-co-directional aggregated message. The attention weights are calculated in the same way as for edges in the same direction, but use independent trainable parameters. and This is to distinguish the feature interaction relationships under different motion directions.
[0024] Step 1.5.4: Use formula (9) to fuse the messages of heterogeneous edges and the characteristics of the nodes themselves.
[0025] (9) In the formula, and For the weights and biases that need to be trained, It is a non-linear activation function.
[0026] Step 1.5.5: Repeat steps 1.5.1-1.5.4 to construct a two-layer graph neural network, update the node features layer by layer, and finally obtain a node representation that integrates multi-layer interaction information. Each layer of the graph neural network has an input and output dimension of 512 and uses a single-head mode.
[0027] Step 1.5.6: The Adam optimizer is used to train the above two-layer graph neural network. The loss function is weighted binary cross-entropy to balance the ratio of positive and negative samples. The training objective is to make the interaction features of the same object in adjacent frames as similar as possible in the feature space, thereby improving the discrimination ability and matching accuracy of interaction features.
[0028] Step 2, Multi-target tracking: Step 2.2.1: Input the video to be tracked frame by frame and obtain the detection results using the YOLOX detector.
[0029] Step 2.2.2: Extract the appearance features of each trajectory and detected object using the ReID module trained by Aharon et al. The appearance features of the trajectory object are the mean of the appearance features of each object in the trajectory.
[0030] Step 2.2.3: Using the interaction feature extractor trained in Step 1, extract the interaction features of each trajectory and the detected object.
[0031] Step 2.2.4, calculate each trajectory and detection IoU score between pairs Cosine similarity of appearance features Similarity with interaction features .
[0032] Step 2.2.5: Use formula (10) to fuse the IoU score, appearance, and interaction feature similarity to obtain the similarity score. .
[0033] (10) In the formula, a and b All weighting coefficients are manually adjusted and satisfy the constraints. .
[0034] Step 2.2.5: Repeat steps 2.2.4-2.2.5 to obtain the similarity matrix S=[ ].in, and N These are the trajectory and the number of detections, respectively.
[0035] Step 2.2.6, convert the similarity matrix S The input is fed into the Hungarian algorithm to obtain the matching results.
[0036] Step 2.2.7: Repeat steps 2.2.1-2.2.6 until the last frame of the video to obtain the tracking result of the video.
[0037] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art will be able to make various modifications and improvements without departing from the principles of the present invention, and these modifications and improvements should also be considered to fall within the scope of protection of the present invention.
Claims
1. A multi-target tracking method based on motion direction consistency interaction features, characterized in that, Includes the following steps: Step 1: Train the interaction feature extractor: Step 1.1: Input the video image and its YOLOX target detection results, wherein the target detection results are derived from ( t , x , y , w , h ) indicates that among them x , y These are the x and y coordinates of the top left corner of the target bounding box, respectively. w , h These are the width and height of the target bounding box, respectively. t For timestamps; Step 1.2: Based on the YOLOX target detection results, crop the target image patch in the video, input the image patch into the pre-trained ReID module, extract the 1024-dimensional appearance features of each target, and use the average appearance features of the objects in the trajectory as the trajectory appearance features; Step 1.3: For t Arbitrary trajectory object in frame -1 i and t Any detected object in the frame j If the distance between the two is less than a preset distance threshold, then the motion vectors of the two are calculated and normalized, and then the cosine value of the angle between the normalized motion vectors is calculated. Based on the sign of the cosine value, the trajectory object is... i With the test object j The edges between them are classified into sets of edges moving in the same direction. Or sets of edges moving in the same direction The cosine value of the included angle is used as the weight of the corresponding side; Step 1.4, repeat step 1.3, until all trajectory-detection pairs with distances less than the threshold have been partitioned into edge weights and edge types, constructing a trajectory-detection heterogeneous graph. G =( H , E ),in H The node features are represented by 2048-dimensional features and 4-dimensional motion features of the trajectory or detected object. x , y , w , h It is pieced together. ; Step 1.5, generate the trajectory-detection heterogeneous graph. G A two-layer graph neural network is input to encode node features layer by layer, aggregate messages, and fuse features to obtain node interaction features that integrate multi-layer interaction information. The two-layer graph neural network is trained with the Adam optimizer and the weighted binary cross-entropy loss function to complete the training of the interaction feature extractor. Step 2: Perform multi-target tracking: Step 2.1: Input the video to be tracked frame by frame, and use the YOLOX detector to obtain the target detection results for each frame; Step 2.2: Use the ReID module to extract the appearance features of each trajectory and the detected object, where the appearance feature of the trajectory is the mean of the appearance features of each object in the trajectory; Step 2.3: Using the interaction feature extractor trained in Step 1, extract the interaction features of each trajectory and the detected object; Step 2.4: Calculate the Intersection over Union (IoU) score, appearance feature cosine similarity, and interaction feature similarity between each trajectory and each detected object. Use a weighted fusion method to fuse the three to obtain a similarity score, and then construct a similarity matrix. Step 2.5: Input the similarity matrix into the Hungarian algorithm to obtain the matching results between the trajectory and the detected object; Step 2.6: Repeat steps 2.1 to 2.5 until the last frame of the video to be tracked is processed to obtain the complete multi-object tracking result.
2. The multi-target tracking method based on motion direction consistency interaction features according to claim 1, characterized in that, In step 1.3, the motion vector is calculated as follows: trajectory object i motion vector ; Detection object j motion vector ; The normalized motion vectors are as follows: , In the formula Represents the magnitude of a vector. A very small constant to prevent division by zero errors; Cosine value of the angle between the motion vectors ; The rules for classifying edges and determining their weights are as follows: ； For the sake of weight values.
3. The multi-target tracking method based on motion direction consistency interaction features according to claim 1, characterized in that, In step 1.5, the processing of node features by the two-layer graph neural network includes: Step 1.5.1: Using the formula Encode node features: In the formula, This represents the encoded node features. Indicates the initial node characteristics. and The weights that need to be trained; Step 1.5.2: For each node i Calculate aggregated messages of motion in the same direction : ； In the formula, Represents a set of neighbors. j Indicates the neighbor node index. Represents a node j In the l The feature vector of the layer, For trainable parameters, Attention weight ; In the formula, For the weights to be trained, exp(.) represents the exponential function, and LeakyReLU(.) is the linear rectified function with leakage; Step 1.5.3: Using the same method as in Step 1.5.2, aggregate messages from the non-same-direction neighbors of each node to obtain the non-same-direction aggregated message. The attention weights are calculated in the same way as those for edges in the same direction, but use independent trainable parameters. and ; Step 1.5.4, using the formula The messages from heterogeneous edges and the node's own characteristics are fused. and For the weights and biases that need to be trained, It is a non-linear activation function. Step 1.5.5: Repeat steps 1.5.1 to 1.5.4 to complete the feature update of the two-layer graph neural network and obtain the final interaction features of the nodes. Each layer of the graph neural network has an input and output dimension of 512 and uses a single-head mode.
4. The multi-target tracking method based on motion direction consistency interaction features according to claim 1, characterized in that, In step 2.4, the formula for fusion of similarity scores is: ; in, For trajectory and detection Similarity score, The IoU scores of the two The cosine similarity of the appearance features of the two is... The similarity of the interaction features between the two is... a and b All are manually adjusted weighting coefficients, and satisfy the following conditions: .