Multi-target tracking method with joint detection and feature embedding

Abstract

The application discloses a multi-target tracking method combining detection and feature embedding, and comprises the following steps: obtaining different features of input detection branches and identity branches through a feature separation module; obtaining more fine feature expression of the features of the input identity branches through a multi-scale feature fusion module; obtaining detection results of t frames of the features of the input detection branches through a target detector; a data association module firstly performs first matching on high-score detection boxes and all existing tracks, and then performs second matching between the tracks which are not matched in the first stage and low-score detection boxes, so that the targets in the low-score detection boxes are correctly matched to the previous tracks and background interference is removed; if both the first matching and the second matching are unsuccessful, continuous hypothesis tracking is performed on the targets, target track information and feature x frames are reserved, and finally unified track management is performed. The method can still maintain tracking accuracy and speed under the conditions of camera shaking and local occlusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a multi-target tracking method of joint detection and feature embedding. Background Art

[0002] Intelligent video analysis technology can process video data captured by surveillance cameras and perform structured analysis and organization of physical targets and their behaviors for different application requirements, generating structured data that is easily queried. In the field of public safety, human targets and their behaviors are of particular concern, and obtaining state information such as their position and scale is the basis for behavioral analysis. Therefore, the multi-target tracking task has emerged, aiming to identify multiple moving objects in an image sequence and assign specific numbers to the moving objects in different frames, thereby determining the motion trajectories of the different objects. As a mid-level task in computer vision, multi-target tracking provides a foundation for higher-level tasks such as pose estimation, action recognition, and behavior analysis, and has important theoretical research significance and practical application value.

[0003] Multi-target tracking is generally divided into two frameworks: SDE (Separate Detection and Embedding) and JDE (Joint Detection and Embedding). SDE separates Reid feature extraction and target detection into two independent networks. This approach has the advantage of achieving relatively high accuracy by independently optimizing each component, but the disadvantage is that the computational complexity increases. JDE integrates target detection and Reid feature extraction into a single network, effectively reducing the algorithm's computational complexity.

[0004] At the same time, due to the complexity and size of the dataset, the diverse pedestrian trajectories, and the high density of pedestrians, data labeling and training are difficult. Real-world scenarios present challenges such as trajectory interruptions and ID switching caused by detection errors, prediction errors caused by the frequent disappearance and reappearance of target individuals due to partial occlusion, and unstable extracted features. To address these challenges, a multi-target tracking method that combines detection and feature embedding is urgently needed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to provide a multi-target tracking method that can effectively maintain the accuracy and real-time rate of multi-target tracking in complex scenarios by joint detection and feature embedding.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a multi-target tracking method with joint detection and feature embedding, comprising the following steps:

[0007] S1: Input the video frame by frame into the backbone network to obtain the features of the current frame t, then obtain the different features of the input detection branch and identity branch through the feature separation module, and finally output the features required by the detection branch and identity branch at the same time;

[0008] S2. The features of the input identity branch are used to obtain a more refined feature expression through the multi-scale feature fusion module. The features of the input detection branch are used to obtain the detection results of t frames through the object detector, while retaining both high-score detection frames and low-score detection frames.

[0009] S3, the data association module first performs graph matching between the high-scoring detection boxes and all existing trajectories, and then performs a second matching between the unmatched trajectories in the first stage and the low-scoring detection boxes, so that the targets in the low-scoring detection boxes are correctly matched to the previous trajectories and background interference is removed;

[0010] S4. If no matching is successful twice, continuous hypothesis tracking is performed, target trajectory information and feature x frames are retained, and finally unified trajectory management is performed to achieve multi-target tracking of pedestrians;

[0011] The beneficial effects of adopting the above technical solution are: the method of the present invention adopts the JDE paradigm to output the features and detection results at the same time, reducing the time of calculating the identity features by re-identification neural network alone under the SDE paradigm, which can effectively reduce the amount of calculation. The real target person in the low-score detection frame is mined and the background is removed through a secondary data association matching strategy. Before the NSA Kalman filter predicts the detection frame, camera motion compensation is performed to alleviate the drift of the detection frame caused by camera shake, which ultimately leads to data association matching errors. The first data association uses graph matching, and the second data association uses the Hungarian matching algorithm, in which the cost matrix is ​​calculated for motion feature information, making the target person data association matching more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] Figure 1 is a flow chart of the method described in an example embodiment of the present invention;

[0014] Figure 2 This is the basic JDE model used in the embodiments of the present invention;

[0015] Figure 3 This is a flowchart of multi-scale feature fusion according to an embodiment of the present invention;

[0016] Figure 4 This is the process of data association in an embodiment of the present invention;

[0017] Figure 5This is a graph matching method for data association according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] like Figure 1 As shown, an embodiment of the present invention discloses a multi-target tracking method of joint detection and feature embedding, the method comprising the following steps:

[0021] S1: Input the video frame by frame into the backbone network to obtain the features of the current frame t, then obtain the different features of the input detection branch and identity branch through the feature separation module, and finally output the features required by the detection branch and identity branch at the same time;

[0022] S2. The features of the input identity branch are used to obtain a more refined feature expression through the multi-scale feature fusion module. The features of the input detection branch are used to obtain the detection results of t frames through the object detector, while retaining both high-score detection frames and low-score detection frames.

[0023] S3, the data association module first performs graph matching between the high-scoring detection boxes and all existing trajectories, and then performs a second matching between the unmatched trajectories in the first stage and the low-scoring detection boxes, so that the targets in the low-scoring detection boxes are correctly matched to the previous trajectories and background interference is removed;

[0024] S4. If no matching is successful twice, continuous hypothesis tracking is performed, target trajectory information and feature x frames are retained, and finally unified trajectory management is performed to achieve multi-target tracking of pedestrians;

[0025] like Figure 2As shown, this embodiment adopts the classic JDE paradigm, simultaneously outputting detection results and features. This eliminates the need to recalculate feature vectors through the re-identification neural network, reducing the computational complexity of the algorithm. Video is detected frame by frame. The features output by the backbone network first pass through the feature separation module before entering the detection branch and the identity branch, respectively. They then enter the data association module, where, through data association matching, the target person's trajectory information is ultimately output, enabling multi-target tracking of pedestrians.

[0026] Furthermore, in step S1: a Reid branch is added to the YOLOv5 target detector, and ResNet-50 is used as the backbone network to extract high-level semantic features; the extracted features are divided into two branches, an identity branch and a detection branch, for processing by a feature separation module. The identity branch obtains local features of the target pedestrian, and the detection branch obtains global features for the target detector.

[0027] The feature separation module specifically involves the following steps: The identity branch passes feature F, which has dimensions C×H×W, through three convolutional layers, generating feature maps F1, F2, and F3 with the same dimensions as feature F. Feature map F is resized to C×K, where K=H×W, and then transposed and multiplied to K×C. Feature map F2 is resized to C×K, and the two matrices are multiplied before passing through the SoftMax function to generate a new feature map. This feature map is then multiplied by the transformed matrix F3 and resized to restore it to the same dimension as feature F. Finally, it is summed with feature F to generate the output. The detection branch uses a self-attention module to strengthen the association between each feature point and other features, enhancing the representation of global features and achieving more accurate object detection results.

[0028] Furthermore, in step S2, the target detector detects the video frame by frame, retains the high-score detection frames and the low-score detection frames, and predicts and updates the target trajectory through the NSA Kalman filter; the multi-scale feature fusion module in the identity branch mainly aggregates features of different resolutions to align features from objects of different sizes.

[0029] like Figure 3 Figure 2 shows the process of the multi-scale feature fusion module in the feature branch. Specifically, features at scales of 1 / 16 and 1 / 32 (compared to the input image size) are first upsampled to 1 / 8. A 3×3 convolutional layer is then used to encode the upsampled feature maps. To aggregate useful information about each object at different resolutions, the present invention first uses a spatial attention module (SAM) to enhance target-related features and suppress background noise.

[0030] Average pooling and maximum pooling are performed on the channel dimension to obtain two feature maps of size 1×H×W, which are sequentially connected and processed by a 7×7 convolutional layer and a sigmoid layer to generate a spatial attention feature map. The resulting spatial attention map is learned separately at each resolution and fused with the original feature through element-wise multiplication and residual attention.

[0031] Feature maps from different scales are concatenated and passed through a channel attention module (CAM). This module includes a global average pooling layer and a max pooling layer, which respectively learn different feature information from features at different resolutions. The output of the pooling layer is processed by a shared network consisting of a one-dimensional convolutional layer and a fully connected layer. The feature maps are fused through element-wise addition and normalized through a sigmoid layer to generate an attention map with a single channel.

[0032] Furthermore, in step S3, if the matching value between the current t-frame high-scoring detection frame and the trajectory prediction frame is greater than a preset threshold, the association is successful, the identity features and trajectory information of the current frame trajectory are updated, and the position of the next frame detection frame is predicted through the NSA Kalman filter; otherwise, the first unmatched trajectory is matched with the low-scoring detection frame. If the matching value is greater than the preset threshold, the association is successful. Otherwise, it is marked as a lost trajectory and is continuously hypothesized to be tracked for x frames.

[0033] like Figure 4 The following is a flowchart of data association in this example. First, the video is input frame by frame, the trajectory is initialized, all the target people in the first frame t are assigned ID information, and the NSA Kalman filter is used to predict the position of the candidate box in the t+1 frame.

[0034] A further technical solution is that the NSA Kalman filter can overcome the sensitivity of the linear Kalman filter to noise and proposes a formula that can adaptively calculate the noise covariance:

[0035]

[0036] In the formula is the preset constant measurement noise covariance, is the detection confidence score in state k.

[0037] The target detector is used to obtain t-frame detection results, and the detection frames with detection scores greater than 0.3 and less than 0.6 are marked as low-score detection frames, and the detection frames with detection scores greater than 0.6 are marked as high-score detection frames. Then, the first match is performed between the high-score detection frame and the predicted frame. If it is greater than the preset threshold, the match is successful, and the position information and identity feature information are updated; the trajectory that was not matched the first time is matched for the second time with the low-score detection frame. If the match is successful, the position information and identity feature information are updated, otherwise it is marked as a lost trajectory. The first matching method is graph matching, and the second matching method is the Hungarian matching algorithm. If the trajectory is not successfully matched twice, it is hypothesized to be tracked for x frames. If the re-association is successful within x frames, the trajectory is reactivated, otherwise the trajectory is deleted. Finally, unified trajectory management is performed to ensure the uniqueness of the target identity and realize multi-target tracking of pedestrians.

[0038] The relationship between trajectories and detection results is modeled as an undirected graph, transforming the data association problem into a graph matching problem. The undirected graph construction first extracts features from the detections and uses these features to construct a detection graph. The trajectory graph follows a similar approach, but with the appearance features averaged. Next, a cross-graph GCN is used to enhance the features. The cross-graph GCN is used to enhance the features, with weights derived from a weighted fusion of appearance feature similarity and motion feature similarity.

[0039] The appearance feature similarity is calculated using the cosine distance, and the motion feature similarity is calculated using the intersection-over-union ratio of the current frame detection frame and the prediction frame. The specific formula is as follows:

[0040]

[0041] The cosine value range is [-1, 1]. The angle between two vectors A and B is calculated, and the corresponding cosine value of the angle is obtained. This cosine value can represent the similarity of the two vectors.

[0042]

[0043] The formula means the area of ​​the intersection of the predicted bounding box and the actual bounding box divided by the area of ​​the union.

[0044] like Figure 5 As shown in the figure, this is the graph matching method used in the data association of this example. First, features are extracted from the detection results and these features are used to construct the detection graph. The trajectory graph is similar, but the appearance features need to be averaged. The cross-graph GCN module is used to enhance the features and weights. Derived from feature similarity and geometric information, where and B represent the edge similarity matrix and vertex similarity matrix, respectively. The elements of the edge similarity matrix are the cosine distances between edge features in the two graphs, and the elements of the vertex similarity matrix are the cosine similarities between vertex features. We then derive the optimal matching score vector and reshape it to obtain the optimal matching score mapping.

[0045] A further technical solution is that the t frame detection set is . Get the trajectory from the detection box of the past frame . and The number of detected targets and the number of tracking candidates. Indicates that The image pixels containing the detection area, Is a geometric vector containing the center position and size of the detection bounding box. Each track contains a series of detection objects with the same ID. Add track ,but It can be expressed as .

[0046] A further technical solution is to define the t frame detection map , trajectory diagram Each vertex and vertices Representative detection and tracking The data association problem can be expressed as and Graph matching problem between . Using a pre-trained re-identification network and a multi-layer perceptron to generate detection Appearance characteristics . Trajectory Appearance characteristics Obtained by averaging all previous detection appearance features.

[0047] A further technical solution is to detect the and tracking graph , the graph matching problem is to maximize the similarity between matching vertices and the corresponding edges connected by these vertices. Each element of the vertex similarity matrix B is Vertex features and Vertex features The cosine similarity between:

[0048]

[0049] Second, calculate and The similarity of the edges in the matrix is ​​used to construct the . The elements in are the edge features of the two graphs and The cosine similarity between:

[0050] in yes The edge of yes The edge in.

[0051] A further technical solution is to and trajectory diagram The GCN module is used to enhance the features. The initial vertex features on the detection graph and tracking graph are the appearance features on the vertices, that is, , . and is the vertex and vertices The lth propagation feature of GCN is defined as As the appearance and geometric similarity between vertex i and vertex j:

[0052]

[0053] in represents the cosine similarity of the input features, Represents the IoU (intersection over union) of two bounding boxes, i.e. the similarity of motion features. For the detection vertex i, Is its corresponding detection bounding box. For tracking vertex j, the bounding box of frame t is predicted by NSA Kalman filter. .

[0054] A further technical solution is to use the edge similarity matrix Me and the vertex similarity matrix B to optimize the quadratic assignment problem, and derive the optimal matching score vector x and reshape it into Size, through which the optimal match is obtained.

[0055] A further technical solution is to use binary weighted cross entropy loss for training throughout the process:

[0056]

[0057] Among them, it means detection With tracking The matching score, Indicates whether the target is the true value in the trajectory. is the weight to balance the loss between positive and negative samples.

[0058] In summary, the method of the present invention adopts the JDE paradigm to output features and detection results at the same time, reducing the time for calculating identity features through the re-identification neural network alone under the SDE paradigm, which can effectively reduce the amount of calculation. A secondary data association matching strategy is used to mine the real target person in the low-scoring detection frame and remove the background. Before the NSA Kalman filter predicts the detection frame, camera motion compensation is performed to alleviate the drift of the detection frame caused by camera shake, which ultimately leads to data association matching errors. The first data association uses graph matching, and the second data association uses the Hungarian matching algorithm, where the cost matrix is ​​calculated for motion feature information, making the target person data association matching more accurate.

Claims

1. A multi-target tracking method based on joint detection and feature embedding, characterized in that The steps include: S1: Input the video frame by frame into the backbone network to obtain the features of the current frame t, and then obtain the different features of the input detection branch and the identity branch through the feature separation module; S2: The features of the input identity branch are used to obtain a more refined feature expression through the multi-scale feature fusion module. The features of the input detection branch are used to obtain the detection results of t frames through the object detector, and both high-scoring detection frames and low-scoring detection frames are retained for subsequent data association. S3: The data association module first performs a first match between the high-scoring detection boxes and all existing tracks. Then, it performs a second match between the unmatched tracks in the first stage and the low-scoring detection boxes, so that the targets in the low-scoring detection boxes are correctly matched to the previous tracks and background interference is removed. S4: If both matches are unsuccessful, continuous hypothesis tracking is performed, the target trajectory information and feature x frames are retained, and finally unified trajectory management is performed to ensure the uniqueness of the target identity and realize multi-target tracking of pedestrians; The step S3 includes: if the matching value between the high-scoring detection frame and the trajectory prediction frame of the current frame t is greater than a preset threshold, the first matching is successful, the identity features and trajectory information of the current frame trajectory are updated, and the position of the next frame detection frame is predicted through the NSA Kalman filter; otherwise, the trajectory that was not matched the first time is matched with the low-scoring detection frame. If the matching value is greater than the preset threshold, the second matching is successful. Otherwise, it is marked as a lost trajectory and is continuously hypothesized and tracked for x frames; The first matching algorithm of data association is graph matching, and the second matching algorithm of data association is Hungarian matching algorithm; The graph matching method models the relationship between trajectories and detection results as an undirected graph, transforming the data association problem into a graph matching problem; The undirected graph construction first extracts features from the detection results and uses these features to construct the detection graph. The trajectory graph is similar, but the appearance features need to be averaged. Then, the cross-graph GCN is used to enhance the features. Cross-graph GCN is used to enhance features, and the weights come from the fusion weighted calculation of appearance feature similarity and motion feature similarity.

2. The multi-target tracking method of joint detection and feature embedding according to claim 1, characterized in that In step S1: the classic JDE paradigm is adopted to output the detection results and features at the same time.

3. The multi-target tracking method of joint detection and feature embedding according to claim 1, characterized in that In the step S1: A Reid branch is added to the YOLOv5 target detector, and ResNet-50 is used as the backbone network to extract high-level semantic features. The extracted features are divided into two branches, the identity branch and the detection branch, for processing. The identity branch obtains the local features of the target pedestrian, and the detection branch obtains the global features for the target detector.

4. The multi-target tracking method of joint detection and feature embedding according to claim 3, characterized in that The specific processing steps of the feature separation module are as follows: the identity branch passes the feature F of dimension C×H×W through three convolutional layers respectively to obtain feature maps F1, F2, and F3 with the same dimension as feature F; the feature map F is resized to C×K, where K=H×W, and then transposed and multiplied to K×C. The feature map F2 is resized to C×K with the same dimension, and the two matrices are multiplied and then passed through the SoftMax function to obtain a new feature map. The feature map is multiplied by the matrix converted from the F3 dimension, and then the dimension is adjusted to finally restore it to the feature map with the same dimension as feature F, and finally added to feature F to obtain the output; the detection branch uses the self-attention module to strengthen the association between each feature point and other features to enhance the expression of global features.

5. The multi-target tracking method of joint detection and feature embedding according to claim 1, characterized in that The processing process of the multi-scale feature fusion module in step S2 is as follows: First, the features at scales of 1 / 16 and 1 / 32 are upsampled to 1 / 8. Then, a 3×3 convolutional layer is used to encode the upsampled feature maps, and a spatial attention module (SAM) is used to enhance target-related features and suppress background noise. Average pooling and max pooling are performed on the channel dimension to obtain two feature maps of size 1×H×W. The above feature maps are sequentially connected and processed by a 7×7 convolutional layer and a sigmoid layer to generate a spatial attention feature map; The resulting spatial attention maps are learned separately at each resolution and fused with the original features via element-wise multiplication and residual attention; Concatenate feature maps from different scales and pass them through a channel attention module; The channel attention module (CAM) consists of a global average pooling layer and a maximum pooling layer, which respectively learn different statistical information of features at different resolutions. The output of the pooling layer is processed by a shared network consisting of a one-dimensional convolutional layer and a fully connected layer. The feature maps are fused by element-wise addition and normalized by a Sigmoid layer to generate an attention map with only one channel.

6. The multi-target tracking method of joint detection and feature embedding according to claim 1, characterized in that In the step S2: The object detector detects the video frame by frame, retains high-score detection frames and low-score detection frames, and predicts and updates the target trajectory through the NSA Kalman filter; the multi-scale feature fusion module in the identity branch mainly aggregates features of different resolutions to align features from objects of different sizes.

7. The multi-target tracking method of joint detection and feature embedding according to claim 1, characterized in that The graph matching method comprises the following steps: The t-frame detection set is Get the trajectory from the detection box of the past frame n d and n I is the number of detection targets and the number of tracking candidates; detection Indicates that The image pixels containing the detection area, Is a geometric vector containing the center position and size of the detection bounding box; each track contains a series of detection objects with the same ID; Add track but Expressed as Define t frame detection map Trajectory Map Each vertex and vertices Representative detection and tracking The data association problem is expressed as and Graph matching problem between; using a pre-trained re-identification network and a multi-layer perceptron to generate detection D t Appearance characteristics Track T j Appearance characteristics Obtained by averaging all previous detection appearance features; Detection graph G D and tracking graph G T The graph matching problem is to maximize the similarity between matching vertices and the corresponding edges connected by these vertices; each element of the vertex similarity matrix B is i∈v D Vertex feature h i With j∈v T Vertex feature h j The cosine similarity between: Next, calculate the detection graph G D and tracking graph G T The similarity of the edges in the matrix is ​​used to construct the M e The elements in are the edge features h in the two graphs i,i' and h j,j' The cosine similarity between: where e u =(i,i') is G D The edge in e v =(j,j') is G T The edge of In the detection graph G D and trajectory graph G T The GCN module is used to enhance the features. The initial vertex features on the detection graph and tracking graph are the appearance features on the vertices, that is, and is a vertex i∈G D and vertex j∈G T The l-th propagation feature; define the aggregation weight coefficient in GCN As the appearance and geometric similarity between vertex i and vertex j: Where cos(·,·) represents the cosine similarity of the input features, IoU(·,·) represents the similarity of the motion features of the two bounding boxes; for the detection vertex i, g i is its corresponding detection bounding box; for tracking vertex j, the bounding box g of frame t is predicted by NSA Kalman filtering i ; Use the edge similarity matrix Me and the vertex similarity matrix B to optimize the quadratic assignment problem, derive the optimal matching score vector x, and reshape it into n d ×n t Size, through which the best match is obtained; Binary weighted cross entropy loss is used for training throughout: Where, it represents the detection D i With Tracking T j The matching score, y i,j Indicates whether the target is the true value in the trajectory, k = (n t -1 ) is the weight to balance the loss between positive and negative samples.

Images