Multi-target tracking method based on trajectory guidance

Abstract

The invention discloses a multi-target tracking method based on track guidance, and the method comprises the following steps: carrying out the detection of a video frame through a target detector, obtaining the appearance information and position information of a target person, keeping a high-confidence detection frame and a low-confidence detection frame, and initializing a track; predicting the position of the target person in the next frame according to the track position information, wherein the corrected state vector and covariance matrix are used in the Kalman filter prediction stage; performing first matching on the high-confidence detection frame and all target candidate frames according to the motion similarity, the appearance similarity and the speed direction calculation cost matrix, and then performing second matching between the unmatched target candidate frames and the low-confidence detection frame by using the motion similarity and the speed direction calculation cost matrix; and performing unified management on matching results, and outputting target trajectory information. According to the invention, an end-to-end multi-target tracking algorithm can be realized, and the real-time rate and accuracy of tracking are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a multi-target tracking method based on trajectory guidance. Background Art

[0002] The goal of multiple object tracking is to find multiple moving objects in a given image sequence, give a specific number to the moving objects in different frames, and then give the motion trajectories of different objects. Multiple object tracking has a wide range of application scenarios and has been widely used in video surveillance, human-computer interaction, virtual reality, and autonomous driving. It has very important theoretical research significance and practical application value.

[0003] The key to multi-target tracking tasks is how to obtain accurate target detection and discriminative feature representation in dense scenes. From a technical perspective, it is necessary to accurately obtain the trajectory of each moving target from a given surveillance video. First, it is necessary to accurately obtain the bounding box information of the target in each frame, and then extract the feature information that can represent the identity of the target, and it is necessary to ensure that the feature similarity between the same target is greater than the feature similarity between different targets. Finally, explore how to build an effective data association algorithm to obtain the optimal matching result. The complexity of multi-target tracking technology lies in the integration of different subtasks, and the detection results and corresponding features output by the target detection and feature extraction modules are used for data association. Multi-target tracking also needs to solve problems such as frequent occlusion of targets, unknown start and end times of trajectories, targets that are too small, similar appearances, and overlaps between targets, and the processing method is more complicated.

[0004] Due to the complexity and diversity of real scenes, multi-target tracking still faces problems such as trajectory interruption and ID switching caused by detection errors, prediction errors caused by frequent disappearance and reappearance of target characters due to local occlusion, unstable extracted features and camera shake. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to provide a multi-target tracking method based on trajectory guidance that can effectively maintain the accuracy and real-time rate of multi-target tracking in complex scenarios.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-target tracking method based on trajectory guidance, comprising the following steps:

[0007] S1, detect the video frame through the target detector, obtain the appearance information and position information of the target person, retain the high confidence detection frame and the low confidence detection frame, and initialize the trajectory;

[0008] S2, predict the position of the target person in the next frame based on the trajectory position information, obtain the target candidate frame, perform camera motion compensation before completing the position prediction, and use the corrected state vector and covariance matrix in the Kalman filter prediction stage;

[0009] S3, according to the motion similarity, appearance similarity and speed direction, the cost matrix is ​​calculated to match the high-confidence detection frame with all target candidate frames for the first time, and then the motion similarity and speed direction are used to calculate the cost matrix to perform a second match between the unmatched target candidate frame and the low-confidence detection frame, so that the low-confidence detection target is correctly matched to the previous trajectory and the background interference is removed to obtain the final matching result;

[0010] S4, uniformly manages the matching results, outputs the target trajectory information, and uses the output target trajectory information to enhance the underlying features of the target person extracted by the target detector to achieve multi-target tracking.

[0011] The beneficial effects of adopting the above technical solution are: the method described in the present invention adds the Reid branch to the target detector, eliminating the computational cost of calculating Reid separately and improving the real-time rate of the algorithm; an adaptive exponential moving average is used to update the appearance information of the target, based on the detection confidence, thereby improving the reliability of the appearance features; the cost matrix is ​​calculated based on information such as appearance similarity, speed similarity and motion similarity, thereby improving the accuracy of data association. Camera motion compensation is used to correct the error of the Kalman filter, so that the prediction of the Kalman filter is more accurate in dynamic video sequences. By using the feature information in the historical trajectory of the target, more robust detection results and features are generated, and higher quality trajectories can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0013] Figure 1 is a flow chart of the method described in the example of the embodiment of the present invention;

[0014] Figure 2 is a flow chart of data association matching according to an embodiment of the present invention;

[0015] Figure 3 is a flow chart of secondary association matching of data in an embodiment of the present invention;

[0016] Figure 4 This is a comparison diagram of tracking effects before and after camera motion compensation correction according to an embodiment of the present invention;

[0017] Figure 5 This is the detailed architecture of the backbone network of the embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] like Figure 1 As shown, an embodiment of the present invention discloses a multi-target tracking algorithm based on trajectory guidance, and the method comprises the following steps:

[0021] S1, detect the video frame through the target detector, obtain the appearance information and position information of the target person, retain the high confidence detection frame and the low confidence detection frame, and initialize the trajectory;

[0022] S2, predict the position of the target person in the next frame based on the trajectory position information, obtain the target candidate frame, perform camera motion compensation before completing the position prediction, and use the corrected state vector and covariance matrix in the Kalman filter prediction stage;

[0023] S3, calculate the cost matrix based on motion similarity, appearance similarity and speed direction, perform the first match between the high-confidence detection frame and all target candidate frames, and then use the motion similarity and speed direction to calculate the cost matrix to perform a second match between the unmatched target candidate frame and the low-confidence detection frame, so that the low-confidence detection target is correctly matched to the previous trajectory and the background interference is removed to obtain the final matching result;

[0024] S4, uniformly manage the matching results and output the target trajectory information. The output target trajectory information is used to enhance the underlying features of the target person extracted by the target detector to achieve multi-target tracking.

[0025] The step S1 includes: adding a Reid branch to the YOLOv5 target detector and training based on the training set to obtain the appearance information and location information of the target person; the network uses ResNet-50 as the backbone to extract high-level semantic features, and then integrates the feature pyramid network to generate a multi-scale feature map for subsequent prediction; inputting adjacent frames of the video into the detector to obtain the detection results. The detection results are screened by the non-maximum suppression method to obtain the final detection results including the appearance information and location information of the target person.

[0026] The step S2 includes: predicting the position of the target person in the next frame according to the trajectory position information, estimating the motion state of the existing trajectory in the current t-frame image based on the Kalman filter uniform velocity model to obtain the state vector and the corresponding covariance matrix; the camera motion compensation uses the image registration method to calculate the affine transformation matrix of the key points in the background area of ​​the previous and next two frames of images; using the affine transformation matrix to correct the estimated state vector and the corresponding covariance matrix of the target in the current t-frame image to obtain the corrected Kalman filter state vector and its corresponding covariance matrix.

[0027] The Kalman prediction phase includes:

[0028]

[0029] Where x k represents the estimated state vector at time k, F k ∈R n×n is the driving matrix, Q k Represents the covariance matrix of the process noise.

[0030] The Kalman update phase includes:

[0031]

[0032] p k / k =(Ik k H k ) k / k-1

[0033] In the formula is the measurement matrix, R k represents the covariance matrix of the measurement noise, z k is the measured value.

[0034] The step S3 includes: setting a first confidence threshold and a second confidence threshold, defining a detection box with a confidence greater than the second confidence as a high confidence detection, and performing a first match; defining a detection box with a confidence less than the second confidence and greater than the first confidence as a low confidence detection box, and performing a second match. Preferably, the first confidence threshold is set to 0.2, and the second confidence threshold is set to 0.6.

[0035] The motion similarity in step S3 is represented by calculating the intersection-over-union ratio of the target candidate frame and the detection frame output by the target detector; the appearance similarity is represented by calculating the cosine similarity of the appearance information of the existing trajectory and the appearance information of the target candidate frame; the speed direction is represented by calculating the cosine value of the angle between the motion direction of the existing trajectory and the motion direction of the target candidate frame.

[0036] Step S4 includes: using the output target trajectory information to enhance the consistency area between the current t frame feature and the trajectory feature through the feature enhancement module, and suppressing the inconsistent area through the feature suppression module, thereby improving the accuracy of the underlying features of the target person who is partially occluded and realizing the tracking of multiple targets.

[0037] like Figure 2 and Figure 3 As shown, an embodiment of the present invention also discloses a data secondary association matching method. For each frame image in a video sequence, it is divided into a high-confidence detection frame and a low-confidence detection frame. First, the high-confidence detection frame is matched with all target candidate frames for the first time according to the cost matrix of motion similarity, appearance feature similarity and direction consistency, and the Hungarian algorithm is used to complete the similarity-based matching. Then, a second match is performed between the unmatched target candidate frame and the low-confidence detection frame, and the cost matrix is ​​calculated by weighted fusion of motion similarity and speed consistency. Through the secondary association matching of the data, the background can be removed in the low-confidence detection and the real target person can be excavated.

[0038] Furthermore, the trajectories are managed: the Kalman filter and appearance features are updated for the successfully matched trajectories; the trajectories that are not matched twice are tracked continuously for 30 frames; the unmatched high-confidence detection boxes initialize new trajectories; and the inactive trajectories are deleted to achieve tracking of multiple targets.

[0039] Specifically, the appearance information in the detection result is adaptively adjusted based on the detection confidence, and is fused with the appearance information of the trajectory through an exponential moving average method according to the following formula to update the appearance information of the existing trajectory:

[0040]

[0041] in, is the appearance feature vector of the ith trajectory corresponding to the t-frame image, αt is the smoothing index, S represents the appearance feature vector in the detection result of the t-th frame matching the i-th existing track. det is the detector confidence, and β is the detection confidence threshold.

[0042] like Figure 4 As shown, in this embodiment, the motion state estimation of the current t-frame image is performed by correcting the Kalman filter through the affine transformation matrix, and the process is as follows:

[0043] Correct the position of the detection frame and transform the predicted detection frame from the coordinate system of the k-1th frame to the coordinate system of the kth frame. The transformation step includes rotating the scaling matrix and translation matrix in, The translation matrix transforms the center point position of the detection frame, the rotation and scaling matrix transforms the state vector and noise matrix of the detection target KF, and the camera motion correction:

[0044]

[0045] Among them, C k|k-1 and They represent the Kalman filter state vectors corresponding to the detected target states before and after camera motion compensation in the Kth frame of the video sequence; P k|k-1 and They respectively represent the covariance matrix of the Kalman filter state vector corresponding to the detected target state before and after camera motion compensation in the Kth frame.

[0046] A further technical solution is that in this embodiment, the output target trajectory information is used to enhance the underlying features of the target person extracted by the target detector. The process is as follows: Due to the complex scenes caused by camera shake and partial occlusion in the video, it is difficult to detect the occluded target person using only the current t frame features. The historical trajectory contains the spatiotemporal information of each target person, which can assist the detector in obtaining more accurate detection results. The process is defined as follows:

[0047] Given the tth frame image I t The underlying features F t , the corresponding test result Β t Generated by detector D.

[0048] Β t =D(F t )

[0049] To enhance the feature consistency region, the process is defined as:

[0050]

[0051] Where F t represents the underlying features given by frame t, represents the feature enhancement module, T t-1 Represents the historical trajectory feature map to generate more powerful discriminant features This can generate more accurate detection results.

[0052] Furthermore, the enhanced features can improve the accuracy of the target detector, and the detection formula can be reconstructed as:

[0053]

[0054] The detection result Β of the current t frame obtained based on the detection branch prediction t , the feature branch aims to generate corresponding visual feature descriptions. Since trajectory features contain rich temporal complementary information, they can be used to enhance the description of occluded targets and reduce identity jumps. Therefore, we use trajectory features to enhance the feature description of detection results. Given the underlying feature F t With detection box B t，i , B t，i The corresponding feature map F t,i is defined as:

[0055]

[0056] Where θ represents the convolutional layer, δ represents the trajectory-guided feature enhancement module, and T t-1 Represents the historical trajectory feature map. B t，i The corresponding features can be obtained by global average pooling operation f t,i = GAP(F' t,i ) is obtained. The feature set corresponding to the current t frame detection set is F t ={f t,1 ,…,f t,i ,…,f t,n}.

[0057] Furthermore, the process of generating enhanced features is defined as follows:

[0058]

[0059] In the formula, given the underlying feature F extracted by the backbone network t , the corresponding enhanced features Ε t-1 represents the average characteristics of historical trajectories, f(F t ,E t-1 ) and h(F t ,E t-1 ) represents the underlying feature F t The average characteristic of the historical trajectoryt-1 between enhancing attention and erasing attention, Indicates that elements are added level by level.

[0060] Furthermore, the historical trajectory features are used to erase the underlying interference features F t,* The interference region in can be defined as:

[0061]

[0062] where e t-1,k It is the feature obtained by global average pooling of the historical trajectory feature map. represents the multiplication operation between elements, and γ and κ use binary masks to search for relevant features.

[0063] Furthermore, the specific process of erasing interference features is as follows: a block binarization layer is used to generate a mask to erase the interference term area of ​​feature F. A sliding block of size 2×2 and stride 1 is used to search for the most prominent continuous area in the relevant features, and the relevant value of each block is defined as the sum of the values ​​in the block. Then, the candidate block with the highest value is selected as the block to be retained, that is, the binary mask of the relevant feature is obtained by setting the value of the selected block to 0 and the other values ​​to 1.

[0064] like Figure 5 As shown, the backbone network architecture of an embodiment of the present invention is shown. Through a given image sequence, the image is input, and ResNet-50 is used as the backbone network to extract high-level semantic features, and then a multi-scale feature map is generated through a feature pyramid.

[0065] In summary, the method described in this invention improves the false detection caused by detection error by secondary associating low confidence boxes in dealing with multi-target tracking in complex scenes; obtains more discriminative appearance information by adaptive exponential moving average; improves the real-time rate of the algorithm by adding Reid branches to target detection, reduces the computational cost, and calculates the cost matrix by fusing three characteristics to reduce data association failure caused by noise interference; enhances the consistency area between the current t-frame feature and the trajectory feature through the enhanced attention guided by the output target trajectory information and the erased attention guided by the output target trajectory information, erases the interference item area in the current t-frame feature, and finally enhances the underlying features. The enhanced underlying features can assist the feature extraction module in generating more discriminative features to reduce identity jumps and generate higher quality trajectories.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-target tracking method based on trajectory guidance, characterized in that The steps include: S1, detect the video frame through the target detector, obtain the appearance information and position information of the target person, retain the high confidence detection frame and the low confidence detection frame, and initialize the trajectory; S2, predict the position of the target person in the next frame based on the trajectory position information, obtain the target candidate frame, perform camera motion compensation before completing the position prediction, and use the corrected state vector and covariance matrix in the Kalman filter prediction stage; S3, calculate the cost matrix based on motion similarity, appearance similarity and speed direction, perform the first match between the high-confidence detection frame and all target candidate frames, and then use the motion similarity and speed direction to calculate the cost matrix to perform a second match between the unmatched target candidate frame and the low-confidence detection frame, so that the low-confidence detection target is correctly matched to the previous trajectory and the background interference is removed to obtain the final matching result; S4, uniformly manages the matching results, outputs the target trajectory information, and uses the output target trajectory information to enhance the underlying features of the target person extracted by the target detector to achieve multi-target tracking.

2. The multi-target tracking method based on trajectory guidance as claimed in claim 1, characterized in that In step S1: A Reid branch is added to the YOLOv5 target detector, and training is performed based on the training set to obtain the appearance information and location information of the target person; the network uses ResNet-50 as the backbone to extract high-level semantic features, and then integrates the feature pyramid network to generate multi-scale feature maps for subsequent predictions; adjacent frames of the video are input into the detector to obtain detection results; the detection results are filtered by the non-maximum suppression method to obtain the final detection results including the appearance information and location information of the target person.

3. The multi-target tracking method based on trajectory guidance as claimed in claim 1, characterized in that In the step S2: the motion state of the existing trajectory in the current t-frame image is estimated based on the Kalman filter uniform velocity model to obtain a state vector and a corresponding covariance matrix.

4. The multi-target tracking method based on trajectory guidance as claimed in claim 3, characterized in that: Camera motion compensation uses image registration method to calculate the affine transformation matrix of key points in the background area of ​​the previous and next two frames of images; The estimated state vector and the corresponding covariance matrix of the target in the current t-frame image are corrected using the affine transformation matrix to obtain the corrected Kalman filter state vector and its corresponding covariance matrix.

5. The multi-target tracking method based on trajectory guidance as claimed in claim 4, characterized in that: The Kalman prediction phase includes: Among them, x k represents the estimated state vector at time k, F k ∈R n×n is the driving matrix, Q k represents the covariance matrix of the process noise; The Kalman update phase includes: p k / k =(I-k k H k )p k / k-1 in, is the measurement matrix, R k represents the covariance matrix of the measurement noise, z k is the measured value.

6. The multi-target tracking method based on trajectory guidance according to claim 1, characterized in that: The step S3 includes: setting a first confidence threshold and a second confidence threshold, defining a detection box with a confidence greater than the second confidence as a high-confidence detection, and performing a first match; defining a detection box with a confidence less than the second confidence and greater than the first confidence as a low-confidence detection box, and performing a second match.

7. The multi-target tracking method based on trajectory guidance according to claim 6, characterized in that: The first confidence threshold is set to 0.2, and the second confidence threshold is set to 0.

6.

8. The multi-target tracking method based on trajectory guidance according to claim 1, characterized in that: The appearance information of the target person in the detection result is updated by using the detection confidence as an adaptive exponential moving average method, and the appearance information of the target person is updated according to the following formula; in, is the appearance information of the i-th track corresponding to the t-frame image, α t is the smoothing index, f i t S represents the appearance information in the detection result of the t-th frame matching the i-th existing track. det is the detector confidence, and β is the detection confidence threshold.

9. The multi-target tracking method based on trajectory guidance according to claim 1, characterized in that: The motion similarity in step S3 is represented by calculating the intersection-over-union ratio of the target candidate frame and the detection frame output by the target detector; the appearance similarity is represented by calculating the cosine similarity of the appearance information of the existing trajectory and the appearance information of the target candidate frame; the speed direction is represented by calculating the cosine value of the angle between the motion direction of the existing trajectory and the motion direction of the target candidate frame.

10. The multi-target tracking method based on trajectory guidance according to claim 1, characterized in that: The step S4 includes: using the output target trajectory information to enhance the consistency area between the current t frame feature and the trajectory feature through a feature enhancement module, and suppressing the inconsistent area through a feature suppression module to achieve tracking of multiple targets.

Images