Multi-target tracking method based on joint detection and feature embedding

Abstract

The invention discloses a multi-target tracking method based on joint detection and feature embedding. The method comprises the following steps: acquiring different features of an input detection branch and an identity branch through a feature separation module; inputting the features of the identity branches to obtain finer feature expression through a multi-scale feature fusion module, and inputting the features of the detection branches to obtain a detection result of a t frame through a target detector; the data association module firstly matches a high-score detection frame with all existing tracks for the first time, and then matches unmatched tracks in the first stage with a low-score detection frame for the second time, so that a target in the low-score detection frame is correctly matched with the previous track and background interference is removed; and if the two times of matching are not successful, carrying out continuous hypothesis tracking, reserving target trajectory information and a feature x frame, and finally executing unified trajectory management. According to the method, the tracking precision and speed can still be kept under the conditions of camera shaking, local shielding and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a multi-target tracking method of joint detection and feature embedding. Background Art

[0002] Intelligent video analysis technology can process video data obtained by surveillance cameras, conduct structured analysis and organization of physical targets and their behaviors for different application requirements, and form structured data that is easy to query. In the field of public safety, human targets and their behaviors are the focus of attention, and obtaining state information such as the position and scale of human targets is the basis of behavior analysis. Therefore, the multi-target tracking task came into being, aiming to find multiple moving objects from an image sequence and assign specific numbers to moving objects in different frames, thereby giving the motion trajectories of different objects. As a mid-level task in computer vision, multi-target tracking provides a foundation for advanced tasks such as posture estimation, action recognition, and behavior analysis, and has important theoretical research significance and practical application value.

[0003] Multi-target tracking is generally divided into two frameworks: SDE (Separate Detecting and Embedding) and JDE (Joint Detecting and Embedding). SDE separates Reid feature extraction and target detection into two independent networks. The advantage of this is that independent optimization of each part can achieve relatively high accuracy, but the disadvantage is that the amount of calculation will increase; JDE puts target detection and Reid feature extraction in one network, which can effectively reduce the amount of calculation of the algorithm.

[0004] At the same time, due to the complexity and size of the data set, the different trajectories of pedestrians, and the high density of pedestrians, which make data annotation difficult and training difficult, the real scene mainly faces problems such as trajectory interruption and ID switching caused by detection errors, prediction errors caused by frequent disappearance and reappearance of target characters due to local occlusion, and unstable extracted features. In order to solve the current thorny problems, it is urgent to develop a multi-target tracking method that combines detection and feature embedding. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to provide a multi-target tracking method of joint detection and feature embedding that can effectively maintain the accuracy and real-time rate of multi-target tracking in complex scenarios.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-target tracking method of joint detection and feature embedding, comprising the following steps: S1, input the video frame by frame into the backbone network to obtain the features of the current frame t, then obtain the different features of the input detection branch and identity branch through the feature separation module, and finally output the features required by the detection branch and identity branch at the same time; S2. The features of the input identity branch are expressed more finely through the multi-scale feature fusion module, and the features of the input detection branch are used to obtain the detection results of t frames through the object detector, while retaining the high-score detection frames and the low-score detection frames; S3, the data association module first performs graph matching between the high-score detection frame and all existing trajectories, and then performs a second match between the unmatched trajectories in the first stage and the low-score detection frame, so that the target in the low-score detection frame is correctly matched to the previous trajectory and background interference is removed; S4, if the matching fails twice, then continue to hypothesize and track the target, retain the target trajectory information and feature x frames, and finally perform unified trajectory management to achieve tracking of multiple pedestrian targets; The beneficial effects of adopting the above technical solution are: the method described in the present invention adopts the JDE paradigm to output the features and detection results at the same time, reducing the time of calculating the identity features by re-identification neural network alone under the SDE paradigm, and can effectively reduce the amount of calculation. The real target person in the low-score detection frame is mined and the background is removed through a secondary data association matching strategy. Before the NSA Kalman filter predicts the detection frame, camera motion compensation is performed to alleviate the drift of the detection frame caused by camera shake, which ultimately leads to data association matching errors. The first data association uses graph matching, and the second data association uses the Hungarian matching algorithm, in which the cost matrix is ​​calculated for motion feature information, making the target person data association matching more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0008] Figure 1 is a flow chart of the method described in the example of the embodiment of the present invention; Figure 2 It is the basic JDE model adopted by the embodiment of the present invention; Figure 3 is a flow chart of multi-scale feature fusion according to an embodiment of the present invention; Figure 4 This is the process of data association in an embodiment of the present invention; Figure 5 This is a graph matching method for data association in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0011] like Figure 1 As shown, an embodiment of the present invention discloses a multi-target tracking method of joint detection and feature embedding, the method comprising the following steps: S1, input the video frame by frame into the backbone network to obtain the features of the current frame t, then obtain the different features of the input detection branch and identity branch through the feature separation module, and finally output the features required by the detection branch and identity branch at the same time; S2. The features of the input identity branch are expressed more finely through the multi-scale feature fusion module, and the features of the input detection branch are used to obtain the detection results of t frames through the object detector, while retaining the high-score detection frames and the low-score detection frames; S3, the data association module first performs graph matching between the high-score detection frame and all existing trajectories, and then performs a second match between the unmatched trajectories in the first stage and the low-score detection frame, so that the target in the low-score detection frame is correctly matched to the previous trajectory and background interference is removed; S4, if the matching fails twice, then continue to hypothesize and track the target, retain the target trajectory information and feature x frames, and finally perform unified trajectory management to achieve tracking of multiple pedestrian targets; like Figure 2 As shown, the classic JDE paradigm is adopted in this embodiment, and the detection results and features are output at the same time. It is no longer necessary to recalculate the feature vector through the re-identification neural network, which reduces the algorithm calculation amount. The video is detected frame by frame, and the features output by the backbone network first pass through the feature separation module and then enter the detection branch and identity branch respectively. Then it enters the data association module, and finally outputs the trajectory information of the target person through data association matching, realizing multi-target tracking of pedestrians.

[0012] Furthermore, in step S1: a Reid branch is added to the YOLOv5 target detector, and ResNet-50 is used as the backbone network to extract high-level semantic features; the extracted features are divided into two branches, an identity branch and a detection branch, for processing by a feature separation module, the identity branch obtains local features of the target pedestrian, and the global features obtained by the detection branch are used for the target detector.

[0013] The specific steps of the feature separation module are as follows: the identity branch passes the feature F of dimension C×H×W through three convolutional layers respectively to obtain feature maps F1, F2, and F3 with the same dimension as feature F. The feature map F is adjusted to C×K, where K=H×W, and then the dimension is changed to K×C through transposition multiplication. The feature map F2 is adjusted to C×K with the same dimension, and the two matrices are multiplied, and then a new feature map is obtained through the SoftMax function. The feature map is multiplied by the matrix transformed by the F3 dimension, and the dimension is adjusted to restore it to the feature map with the same dimension as feature F, and finally it is added to feature F to get the output; the detection branch uses the self-attention module to strengthen the association between each feature point and other features to enhance the expression of global features and obtain more accurate target detection results.

[0014] Furthermore, in step S2, the target detector detects the video frame by frame, retains the high-score detection frame and the low-score detection frame, and predicts and updates the target trajectory through the NSA Kalman filter; the multi-scale feature fusion module in the identity branch mainly aggregates features of different resolutions to align features from objects of different sizes.

[0015] like Figure 3 As shown, the process of the multi-scale feature fusion module of the feature branch, in particular, the features of the 1 / 16 and 1 / 32 scales (compared to the size of the input image) are first upsampled to 1 / 8. Then, a 3×3 convolutional layer is used to encode the upsampled feature map. In order to aggregate the useful information of each target at different resolutions, the present invention first uses the spatial attention module SAM to enhance the target-related features and suppress background noise.

[0016] Average pooling and max pooling are performed on the channel dimension to obtain two feature maps of size 1×H×W, which are sequentially connected and processed by a 7×7 convolutional layer and a sigmoid layer to generate a spatial attention feature map. The resulting spatial attention map is learned separately at each resolution and fused with the original features through element-wise multiplication and residual attention.

[0017] The feature maps from different scales are concatenated and passed through the channel attention module CAM. The channel attention module CAM includes a global average pooling layer and a maximum pooling layer to learn different feature information of features at different resolutions. The output of the pooling layer is processed by a shared network consisting of a one-dimensional convolutional layer and a fully connected layer. The feature maps are fused by element-wise addition and normalized by a sigmoid layer to generate an attention map with only one channel.

[0018] Furthermore, in step S3, if the matching value between the current t-frame high-score detection frame and the trajectory prediction frame is greater than a preset threshold, the association is successful, the identity features and trajectory information of the current frame trajectory are updated, and the position of the next frame detection frame is predicted through the NSA Kalman filter; otherwise, the first unmatched trajectory is matched with the low-score detection frame, and if the matching value is greater than the preset threshold, the association is successful, otherwise it is marked as a lost trajectory and is continuously hypothesized to be tracked for x frames.

[0019] like Figure 4 The figure shows the flowchart of data association in this example. First, the video is input frame by frame, the trajectory is initialized, all the target characters in the first frame t are assigned ID identity information, and the NSA Kalman filter is used to predict the position of the candidate frame in the t+1 frame.

[0020] A further technical solution is that the NSA Kalman filter can overcome the sensitivity of the linear Kalman filter to noise, and proposes a formula that can adaptively calculate the noise covariance:

[0021] In the formula is the preset constant measurement noise covariance, is the detection confidence score in state k.

[0022] The detection results of t frames are obtained through the target detector, and the detection frames with detection scores greater than 0.3 and less than 0.6 are marked as low-score detection frames, and the detection frames with detection scores greater than 0.6 are marked as high-score detection frames. Then, the first match is performed between the high-score detection frame and the predicted frame. If it is greater than the preset threshold, the match is successful, and the position information and identity feature information are updated; the trajectory that was not matched for the first time is matched for the second time with the low-score detection frame. If the match is successful, the position information and identity feature information are updated, otherwise it is marked as a lost trajectory. The first matching method is graph matching, and the second matching method is the Hungarian matching algorithm. If the trajectory is not successfully matched twice, it is hypothesized to be tracked for x frames. If the re-association is successful within x frames, the trajectory is reactivated, otherwise the trajectory is deleted. Finally, unified trajectory management is performed to ensure the uniqueness of the target identity and realize multi-target tracking of pedestrians. The relationship between the trajectory and the detection results is modeled as an undirected graph, and the data association problem is transformed into a graph matching problem. The construction of the undirected graph first extracts features from the detection and uses these features to construct the detection graph. The trajectory graph is similar, but the appearance features need to be averaged. Then, the cross-graph GCN is used to enhance the features. The cross-graph GCN is used to enhance the features, and the weights come from the fusion weighted calculation of the appearance feature similarity and the motion feature similarity.

[0023] The appearance feature similarity is calculated using the cosine distance, and the motion feature similarity is calculated using the intersection-over-union ratio of the current frame detection frame and the prediction frame. The specific formula is as follows:

[0024] The cosine value range is [-1, 1]. The angle between two vectors A and B is calculated, and the corresponding cosine value of the angle is obtained. This cosine value can represent the similarity of the two vectors.

[0025]

[0026] The formula means the area of ​​the intersection of the predicted bounding box and the actual bounding box divided by the area of ​​the union.

[0027] like Figure 5 As shown in the figure, this is the graph matching method used in the data association of this example. First, features are extracted from the detection results and these features are used to construct the detection graph. The trajectory graph is similar, but the appearance features need to be averaged. The cross-graph GCN module is used to enhance the features and weights Derived from feature similarity and geometric information, where and B represent the edge similarity matrix and vertex similarity matrix respectively. The elements of the edge similarity matrix are the cosine distances between the edge features in the two graphs, and the elements of the vertex similarity matrix are the cosine similarities between the vertex features. Then the optimal matching score vector is derived and its shape is reshaped to obtain the optimal matching score mapping.

[0028] A further technical solution is that the t frame detection set is . Get the trajectory from the detection box of the past frame . and The number of detected targets and the number of tracking candidates. Indicates that The image pixels that contain the detection area, is a geometric vector containing the center position and size of the detection bounding box. Each track contains a series of detection objects with the same ID. Add Track ,but It can be expressed as .

[0029] A further technical solution is to define t frame detection graph , trajectory diagram Each vertex and vertices Representative detection and tracking The data association problem can be expressed as and A graph matching problem between . Using a pre-trained re-identification network and a multi-layer perceptron to generate detection Appearance characteristics . Trajectory Appearance characteristics Obtained by averaging all previous detected appearance features.

[0030] A further technical solution is to detect the and tracking graph , the graph matching problem is to maximize the similarity between matching vertices and the corresponding edges connected by these vertices. Each element of the vertex similarity matrix B is Vertex features and Vertex features The cosine similarity between:

[0031] Second, calculate and The similarity of the edges in the matrix . The elements in are the edge features of the two graphs and The cosine similarity between:

[0032] in yes The edge of yes The edge in .

[0033] A further technical solution is to detect the and trajectory diagram The GCN module is used to enhance the features. The initial vertex features on the detection graph and the tracking graph are the appearance features on the vertices, that is, , . and is the vertex and vertices The lth propagation feature of GCN. As the appearance and geometric similarity between vertex i and vertex j:

[0034] in represents the cosine similarity of the input features, Represents the IoU (intersection over union) of two bounding boxes, i.e. the similarity of motion features. For the detection vertex i, is its corresponding detection bounding box. For tracking vertex j, the bounding box of frame t is predicted by NSA Kalman filtering. .

[0035] A further technical solution is to use the edge similarity matrix Me and the vertex similarity matrix B to optimize the quadratic assignment problem, and derive the optimal matching score vector x and reshape it into Size, through which the best match is obtained.

[0036] A further technical solution is to use binary weighted cross entropy loss for training throughout the process:

[0037] Among them, it means detection With tracking The matching score, Indicates whether the target is the true value in the trajectory. . is the weight to balance the loss between positive and negative samples.

[0038] In summary, the method of the present invention adopts the JDE paradigm to output the features and detection results at the same time, reducing the time for calculating the identity features by the re-identification neural network alone under the SDE paradigm, and can effectively reduce the amount of calculation. The real target person in the low-score detection frame is mined and the background is removed through a secondary data association matching strategy. Before the NSA Kalman filter predicts the detection frame, camera motion compensation is performed to alleviate the drift of the detection frame caused by camera shake, which ultimately leads to data association matching errors. The first data association uses graph matching, and the second data association uses the Hungarian matching algorithm, in which the cost matrix is ​​calculated for motion feature information, making the target person data association matching more accurate.

Claims

1. A multi-target tracking method based on joint detection and feature embedding, characterized in that The steps include: S1: Input the video frame by frame into the backbone network to obtain the features of the current frame t, and then obtain the different features of the input detection branch and the identity branch through the feature separation module; S2: The features of the input identity branch are expressed more finely through the multi-scale feature fusion module, and the features of the input detection branch are used to obtain the detection results of t frames through the object detector, and both the high-score detection frames and the low-score detection frames are retained for subsequent data association; S3: The data association module first matches the high-score detection frame with all existing trajectories, and then performs a second match between the unmatched trajectories in the first stage and the low-score detection frame, so that the target in the low-score detection frame is correctly matched to the previous trajectory and background interference is removed; S4: If both matches are unsuccessful, continuous hypothesis tracking is performed to retain the target trajectory information and feature x frames, and finally unified trajectory management is performed to ensure the uniqueness of the target identity and realize multi-target tracking of pedestrians.

2. The multi-target tracking method of joint detection and feature embedding as claimed in claim 1, characterized in that: In the step S1: the classic JDE paradigm is adopted to output the detection results and features at the same time.

3. The multi-target tracking method of joint detection and feature embedding as claimed in claim 1, characterized in that: In step S1: A Reid branch is added to the YOLOv5 target detector, and ResNet-50 is used as the backbone network to extract high-level semantic features. The extracted features are divided into two branches, the identity branch and the detection branch, for processing through the feature separation module. The identity branch obtains the local features of the target pedestrian, and the global features obtained by the detection branch are used for the target detector.

4. The multi-target tracking method of joint detection and feature embedding as claimed in claim 3, characterized in that: The specific processing steps of the feature separation module are as follows: the identity branch passes the feature F with a dimension of C×H×W through three convolutional layers respectively to obtain feature maps F1, F2, and F3 with the same dimension as feature F; the feature map F is adjusted to a dimension of C×K, where K=H×W, and then transposed and multiplied to a dimension of K×C; The feature map F2 is adjusted to C×K with the same dimension, and the two matrices are multiplied and then passed through the SoftMax function to obtain a new feature map; The feature map is multiplied by the matrix transformed by the F3 dimension, and then the dimension is adjusted to finally restore it to the feature map with the same dimension as feature F, and finally added to feature F to get the output; the detection branch strengthens the association between each feature point and other features through the self-attention module to enhance the expression of global features.

5. The multi-target tracking method of joint detection and feature embedding as claimed in claim 1, characterized in that: The processing process of the multi-scale feature fusion module in step S2 is as follows: First, the features of 1 / 16 and 1 / 32 scales are upsampled to 1 / 8. Then, a 3×3 convolutional layer is used to encode the upsampled feature map, and a spatial attention module SAM is used to enhance target-related features and suppress background noise. Average pooling and maximum pooling are performed on the channel dimension to obtain two feature maps of size 1×H×W. The above feature maps are connected and processed in sequence by a 7×7 convolutional layer and a sigmoid layer to generate a spatial attention feature map; The resulting spatial attention maps are learned separately at each resolution and fused with the original features via element-wise multiplication and residual attention; Concatenate feature maps from different scales and pass them through a channel attention module; The channel attention module CAM includes a global average pooling layer and a maximum pooling layer, which respectively learn different statistical information of features with different resolutions; the output of the pooling layer is processed by a shared network consisting of a one-dimensional convolutional layer and a fully connected layer; The feature maps are fused by element-wise addition and normalized by a Sigmoid layer to generate an attention map with only one channel.

6. The multi-target tracking method of joint detection and feature embedding as claimed in claim 1, characterized in that: In step S2: The object detector detects the video frame by frame, retains the high-score detection frames and the low-score detection frames, and predicts and updates the target trajectory through the NSA Kalman filter; the multi-scale feature fusion module in the identity branch mainly aggregates features of different resolutions to align features from objects of different sizes.

7. The multi-target tracking method of joint detection and feature embedding as claimed in claim 1, characterized in that: The S3 step includes: If the matching value between the high-score detection frame and the trajectory prediction frame of the current t frame is greater than the preset threshold, the first matching is successful, the identity features and trajectory information of the current frame trajectory are updated, and the position of the next frame detection frame is predicted through the NSA Kalman filter; otherwise, the trajectory that was not matched for the first time is matched with the low-score detection frame. If the matching value is greater than the preset threshold, the second matching is successful, otherwise it is marked as a lost trajectory and is continuously hypothesized to be tracked for x frames.

8. The multi-target tracking method of joint detection and feature embedding as claimed in claim 7, characterized in that: The first matching algorithm of data association is graph matching, and the second matching algorithm of data association is Hungarian matching algorithm; The graph matching method models the relationship between trajectories and detection results as an undirected graph, transforming the data association problem into a graph matching problem; The construction of the undirected graph first extracts features from the detection results and uses these features to construct the detection graph. The trajectory graph is similar, but the appearance features need to be averaged. Secondly, the cross-graph GCN is used to enhance the features. Cross-graph GCN is used to enhance features, and the weights come from the fusion weighted calculation of appearance feature similarity and motion feature similarity.

9. The multi-target tracking method of joint detection and feature embedding as claimed in claim 8, characterized in that: The graph matching method comprises the following steps: The t frame detection set is ; Get the trajectory from the detection box of the past frame ; and is the number of detection targets and the number of tracking candidates; Indicates that The image pixels that contain the detection area, is a geometric vector containing the center position and size of the detection bounding box; each track contains a series of detection objects with the same ID; Add track ,but It can be expressed as ; Define t frame detection map , trajectory diagram ; Each vertex and vertices Representative detection and tracking ; The data association problem can be expressed as and Graph matching problem between ; using a pre-trained re-identification network and a multi-layer perceptron to generate detection Appearance characteristics ; Track Appearance characteristics Obtained by averaging all previous detection appearance features; Detection diagram and tracking graph The graph matching problem is to maximize the similarity between matching vertices and the corresponding edges connected by these vertices; each element of the vertex similarity matrix B is Vertex features and Vertex features The cosine similarity between: ； Next, calculate the detection map and tracking graph The similarity of the edges in the matrix ; The elements in are the edge features of the two graphs and The cosine similarity between: ； in yes The edge of yes The edge in In the detection diagram and trajectory diagram The GCN module is used to enhance the features. The initial vertex features on the detection graph and the tracking graph are the appearance features on the vertices, that is, , ; and is the vertex and vertices The lth propagation feature of GCN; define the aggregation weight coefficient in GCN As the appearance and geometric similarity between vertex i and vertex j: ； in represents the cosine similarity of the input features, Represents the similarity of motion features of two bounding boxes; for the detection vertex i, is the corresponding detection bounding box; for the tracked vertex j, the bounding box of frame t is predicted by NSA Kalman filtering ; Use the edge similarity matrix Me and the vertex similarity matrix B to optimize the quadratic assignment problem, derive the optimal matching score vector x, and reshape it into Size, through which the best match is obtained; A binary weighted cross entropy loss is used for training throughout the process: ； Among them, it means detection With tracking The matching score, Indicates whether the target is the true value in the trajectory, is the weight to balance the loss between positive and negative samples.

Images