2D multi-object tracking method based on visibility and variant IoU

Abstract

The present invention relates to a 2D multi-target tracking method based on visibility and variant IoU, and belongs to the field of environmental perception technology. The method comprises: using a camera-based 2D detector to obtain two-dimensional detection information of a target at each moment, setting a detection confidence threshold, and dividing high-score detection and low-score detection; based on the trajectory frame and trajectory confidence predicted at the previous moment, calculating the visibility of each trajectory, setting a visibility threshold, and dividing active and inactive trajectories; performing variant IoU and appearance similarity cost fusion matching on high-score detection and active trajectory; adding unmatched active and inactive trajectories, and then performing variant IoU and appearance similarity cost fusion matching on the remaining unmatched high-score detection; performing variant IoU matching on the remaining unmatched trajectory and low-score detection; and managing the trajectory based on a trajectory management mechanism. The present invention can improve the accuracy of target tracking.

Description

Technical Field

[0001] The invention belongs to the field of environmental perception and relates to a 2D multi-target tracking method based on visibility and variant IoU. Background Art

[0002] Multi-Object-Tracking (MOT) aims to determine the spatiotemporal trajectory of each object in a video sequence and has been widely used in the fields of autonomous driving, security monitoring, and robotics. Common MOT paradigms include detection-based tracking, regression-based tracking, joint detection and appearance tracking, and attention-based tracking. Among these paradigms, the Tracking-by-Detection (TBD) paradigm is the mainstream paradigm of MOT, which includes target detection and data association. However, the accuracy of target detection and the precision of data association will seriously affect the tracking effect.

[0003] In the case of low-score true values ​​in target detection, some tracking methods divide detection into high-score detection and low-score detection, so that high-score detection is matched first and low-score detection is matched second, which effectively reduces missed detections and improves the recall rate. However, there are also high-confidence trajectories and low-confidence trajectories. Existing tracking methods lack a matching framework that can be graded from the perspective of trajectories, divide detections and trajectories, and design a reasonable matching framework so that objects belonging to the same true value can be accurately tracked.

[0004] Data association based on motion information and appearance information is the mainstream strategy for 2D multi-target tracking. There are also a few that are based on weak clues for association matching, such as speed direction, height, width, and trajectory confidence. At present, most multi-target tracking methods only weight motion and appearance information according to fixed weights, ignoring the fact that the use of motion information should be reduced in the case of occlusion. In the data association process, the cost function based on IoU only focuses on the degree of overlap of the object itself, and does not consider the area, length, width, and relative distance of the object. Some appearance-based MOT methods can achieve matching when IoU is 0, but in the case of occlusion, the extraction of appearance features will be affected by other neighboring objects. Some existing data association strategies based on IoU and appearance can improve the matching accuracy to a certain extent, but most fusion methods only adopt simple weighting and cannot adaptively achieve the fusion of the two, which affects the further improvement of tracking accuracy. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a 2D multi-target tracking method based on visibility and variant IoU to improve the object data association accuracy in an occluded environment, thereby providing stronger environmental perception for unmanned driving technology and laying the foundation for path planning and control decisions.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A 2D multi-target tracking method based on visibility and variant IoU, comprising:

[0008] S1. Use a camera-based 2D detector to obtain the two-dimensional detection information of the target at each moment, set the detection confidence threshold, define the detection greater than the detection confidence threshold as a high-score detection, and define the detection less than the detection confidence threshold as a low-score detection;

[0009] S2. Perform Kalman prediction on the trajectory of the previous moment to obtain the predicted trajectory in the current frame; based on the predicted trajectory frame and trajectory confidence, calculate the visibility of each trajectory, and set the visibility threshold. Define the trajectory greater than the visibility threshold as an active trajectory, and define the trajectory less than the visibility threshold as an inactive trajectory;

[0010] S3, the high-score detection and active track are matched by the cost fusion of variant IoU and appearance similarity to obtain unmatched high-score detection and unmatched active track;

[0011] S4, add the unmatched active and inactive trajectories, and then perform cost fusion matching of variant IoU and appearance similarity with the remaining unmatched high-score detections;

[0012] S5, perform variant IoU matching on the remaining unmatched trajectories and the low-score detections to obtain the final successfully matched detection-trajectory pairs, unmatched detections, and unmatched trajectories;

[0013] S6. Manage the trajectory based on the trajectory management mechanism.

[0014] Further, step S2 includes:

[0015] S21, performing Kalman prediction on the trajectory at the previous moment to obtain the predicted trajectory of the trajectory of the previous frame at the current moment;

[0016] S22. Calculate the intersection-and-union ratio of any two trajectories:

[0017]

[0018] Among them, b A 、b B They represent the bounding box of trajectory A and the bounding box of trajectory B respectively; if the trajectory confidence of trajectory A is greater than the trajectory confidence of trajectory B, then the intersection area is the area blocked by trajectory B, otherwise, it is the area blocked by trajectory A;

[0019] S23. Based on the area blocked by each track, the visibility is calculated as follows:

[0020]

[0021] In the formula, Δ represents the visibility of the trajectory, T area Represents the bounding box area of ​​the trajectory, T o-area Indicates the area where the trajectory is blocked;

[0022] S24: Based on the visibility of all predicted trajectories, a visibility threshold is set to divide the trajectories into active trajectories and inactive trajectories.

[0023] Further, step S3 includes:

[0024] S31. Calculate the area factor a between the high-score detection and the active track:

[0025]

[0026] In the formula, A T , A D Represent the area of ​​the trajectory box and the detection box respectively, A T∩D Represents the intersection area of ​​the trajectory box and the detection box, A C Represents the area of ​​the minimum bounding box of two rectangular boxes;

[0027] S32. Calculate the height factor hf between the high score detection and the active track:

[0028]

[0029] In the formula, h D 、h T Respectively represent the height of the detection box and the trajectory box, h C Represents the union height of the track box and the detection box;

[0030] S33. Calculate the distance factor d between the high score detection and the active track:

[0031]

[0032] In the formula, ρ represents the distance between the center points of the two boxes, and c represents the distance between the diagonal points of the minimum circumscribed bounding box of the trajectory box and the detection box;

[0033] S34. Calculate the variant IoU HYIoU based on the area factor a, height factor hf, and distance factor d:

[0034] HYIoU=a+IoU·hf@d

[0035] S35. Calculate the fusion cost of HYIoU and appearance similarity:

[0036]

[0037] In the formula, C, C Appr , C HYIoU They represent the correlation matrix, cosine distance matrix, and HYIoU distance matrix between active trajectories and high-score detections, respectively.

[0038] S36. Use Hungarian matching to complete the matching between the active track and the high-score detection, and obtain unmatched active track and unmatched high-score detection.

[0039] Further, step S4 includes:

[0040] S41, adding the unmatched active trajectory and the inactive trajectory as the trajectory to be matched; calculating the area factor a′ between the unmatched high-score detection and the trajectory to be matched:

[0041]

[0042] S42, calculate the height factor hf′ between the unmatched high-score detection and the trajectory to be matched:

[0043]

[0044] S43, calculate the distance factor d′ between the unmatched high-score detection and the track to be matched:

[0045]

[0046] S44. Calculate the variant IoU HYIoU′ according to the area factor a′, the height factor hf′, and the distance factor d′:

[0047] HYIoU′=a′+IoU.hf′.d′

[0048] S45. Calculate the fusion cost of HYIoU′ and appearance similarity:

[0049]

[0050] Among them, C′, C′ Appr , C′ HYIoU They represent the correlation matrix, cosine distance matrix, and HYIoU distance matrix between the unmatched high-score detections and the trajectories to be matched, respectively;

[0051] S46. Use Hungarian matching to complete the matching between the unmatched high-score detection and the trajectory to be matched, and obtain the unmatched trajectory and the unmatched detection.

[0052] Further, step S5 includes:

[0053] S51, calculate the area factor a″ between the low-score detection and the remaining unmatched tracks:

[0054]

[0055] S52, calculate the height factor hf″ between the low-score detection and the remaining unmatched tracks:

[0056]

[0057] S53, calculate the distance factor d″ between the low-score detection and the remaining unmatched tracks:

[0058]

[0059] S54. Calculate variant IoU HYIoU according to area factor a″, height factor hf″, and distance factor d″:

[0060] HYIoU″=a″+IoU.hf″.d″

[0061] S55. Use HYIoU″ as the cost to complete the matching between the low-score detection and the remaining unmatched tracks, and obtain the final successfully matched detection-track pairs, unmatched detections, and unmatched tracks.

[0062] Further, step S6 includes:

[0063] S61, performing Kalman update for the successfully matched detection-trajectory pair;

[0064] S62, for the detections that are not successfully matched, the detections that are greater than the threshold are initialized as new uncertain state trajectories, which need to be successfully matched at the next moment to be converted into a certain state trajectory;

[0065] S63, for the unmatched trajectory, increase the number of trajectory disappearance frames, and retain 30 frames for the unmatched trajectory.

[0066] The beneficial effects of the present invention are as follows: the present invention designs a variant IoU so that the IoU-based cost function can simultaneously consider information such as the area, length, width and relative distance of the tracked target, thereby greatly improving the data association accuracy of the target in an occluded environment; in addition, the present invention divides target detection into high-score detection and low-score detection, divides target trajectories into active trajectories and inactive trajectories according to confidence, and matches detection and trajectory based on an improved cost matching strategy, thereby improving target tracking accuracy.

[0067] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0069] Figure 1 It is a flowchart of the tracking method of the present invention;

[0070] Figure 2 Schematic diagram of the area factor of variant IoU;

[0071] Figure 3 Schematic diagram of the height factor of variant IoU;

[0072] Figure 4 Schematic diagram of the distance factor of variant IoU;

[0073] Figure 5 Schematic diagram of the visibility of the occluded object. DETAILED DESCRIPTION

[0074] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0075] In view of the fact that most of the multi-target tracking methods in the prior art only weight the motion and appearance information according to fixed weights, ignoring the fact that the use of motion information should be reduced in the case of occlusion, and the traditional IoU only focuses on the degree of overlap of objects, ignoring the relative area, relative height and relative distance between objects, an embodiment of the present invention proposes a 2D multi-target tracking method based on visibility and variant IoU, such as Figure 1 As shown, it includes:

[0076] Step 1: Use a camera-based 2D detector to obtain the two-dimensional detection information of the object at each moment, set the detection confidence threshold, define the detection greater than the detection confidence threshold as a high-score detection, and define the detection less than the detection confidence threshold as a low-score detection.

[0077] Among them, the information of the target object obtained by the 2D detector uses a two-dimensional bounding box, which can be expressed as boundingbox=(x1,y1,x2,y2,s), (x1,y1) represents the coordinates of the upper left corner of the object detection box, (x2,y2) represents the coordinates of the lower right corner of the object detection box, and s represents the detection confidence of the object.

[0078] Step 2: Perform Kalman prediction on the trajectory of the previous moment to obtain the predicted trajectory in the current frame. Based on the predicted trajectory frame and trajectory confidence, calculate the visibility of each trajectory, set the visibility threshold, define the trajectory greater than the visibility threshold as an active trajectory, and define the trajectory less than the visibility threshold as an inactive trajectory. The visibility of the trajectory is as follows: Figure 5 As shown, Figure 5 In the figure, the green font is the trajectory confidence and the blue font is the visibility of the occluded object.

[0079] The steps are as follows:

[0080] 1) Perform Kalman prediction on the trajectory of the previous moment to obtain the predicted trajectory of the previous frame at the current moment;

[0081] 2) Calculate the intersection over union (IoU) of trajectory A and trajectory B:

[0082]

[0083] Among them, b A 、b B They represent the bounding box of track A and the bounding box of track B respectively. If the track confidence of track A is greater than the track confidence of track B, then the intersection area is the area blocked by track B, otherwise, it is the area blocked by track A;

[0084] 3) Based on the area blocked by each track, the visibility calculation formula is as follows:

[0085]

[0086] Among them, α represents the visibility of the trajectory, T area Represents the bounding box area of ​​the trajectory, T o-area Indicates the area where the trajectory is blocked;

[0087] 4) Based on the visibility of all predicted trajectories, a visibility threshold is set to divide the trajectories into active trajectories and inactive trajectories.

[0088] Step 3: Perform cost fusion matching of variant IoU and appearance similarity on high-score detections and active trajectories to obtain unmatched high-score detections and unmatched active trajectories.

[0089] The specific steps are:

[0090] 1) Calculate the area factor a between high-score detections and active tracks:

[0091]

[0092] Among them, A T , A D Represent the area of ​​the trajectory box and the detection box respectively, A T∩D Represents the intersection area of ​​the trajectory box and the detection box, A C Represents the area of ​​the minimum bounding box of two rectangular boxes; where the area factor is Figure 2 As shown;

[0093] 2) Calculate the height factor hf between the high-score detection and the active track:

[0094]

[0095] Among them, h D 、h T are the heights of the detection box and the trajectory box, h C is the union height of the trajectory box and the detection box; the height factor is Figure 3 As shown;

[0096] 3) Calculate the distance factor d between the high-score detection and the active track:

[0097]

[0098] Among them, ρ represents the distance between the center points of the two boxes, c represents the distance between the diagonal points of the minimum circumscribed bounding box of the trajectory box and the detection box; the distance factor is as follows Figure 4 As shown;

[0099] 4) According to the area factor a, height factor hf, and distance factor d, calculate the variant IoU - HYIoU:

[0100] HYIoU=a+IoU·hf·d

[0101] 5) Calculate the fusion cost of HYIoU and appearance similarity:

[0102]

[0103] Among them, C, C Appr , C HYIoU They are the correlation matrix, cosine distance matrix, and HYIoU distance matrix between active trajectories and high-score detections, respectively.

[0104] 6) Use Hungarian matching to complete the matching between active trajectories and high-score detections, and obtain unmatched active trajectories and unmatched high-score detections.

[0105] Step 4: Add the unmatched active and inactive trajectories, and then perform cost fusion matching of variant IoU and appearance similarity with the remaining unmatched high-score detections;

[0106] The specific steps are:

[0107] 1) Add the unmatched active trajectories and inactive trajectories together to form the trajectory to be matched; calculate the area factor a′ between the unmatched high-score detection and the trajectory to be matched:

[0108]

[0109] 2) Calculate the height factor hf′ between the unmatched high-score detection and the track to be matched:

[0110]

[0111] 3) Calculate the distance factor d′ between the unmatched high-score detection and the trajectory to be matched:

[0112]

[0113] 4) According to the area factor a′, height factor hf′, and distance factor d′, calculate the variant IoU——HYIoU′:

[0114] HYIoU′=a′+IoU·hf′·d′

[0115] 5) Calculate the fusion cost of HYIoU′ and appearance similarity:

[0116]

[0117] Among them, C′, C′ Appr , C′ HYIoU They are the correlation matrix, cosine distance matrix, and HYIoU distance matrix between the unmatched high-score detections and the trajectories to be matched, respectively;

[0118] 6) Use Hungarian matching to complete the matching between the remaining high-score detections and the trajectories to be matched, and obtain unmatched trajectories and unmatched detections.

[0119] Step 5: Perform variant IoU matching on the remaining unmatched trajectories and low-score detections to obtain the final successfully matched detection-trajectory pairs, unmatched detections, and unmatched trajectories;

[0120] The specific steps are:

[0121] 1) Calculate the area factor a″ between the low-scoring detections and the remaining unmatched tracks:

[0122]

[0123] 2) Calculate the height factor hf″ between the low-scoring detections and the remaining unmatched tracks:

[0124]

[0125] 3) Calculate the distance factor d″ between the low-scoring detections and the remaining unmatched tracks:

[0126]

[0127] 4) According to the area factor a″, height factor hf″, and distance factor d″, calculate the variant IoU——HYIoU″:

[0128] HYIoU″=a″+IoU·hf″·d″

[0129] 5) Using HYIoU″ as the cost, the matching between low-score detections and remaining unmatched tracks is completed, and the final matched detection-track pairs, unmatched detections and unmatched tracks are obtained.

[0130] Step 6. For the successfully matched detection-trajectory pairs, perform Kalman update. For the unmatched detections, initialize the detections greater than the threshold as new uncertain trajectories, which need to be successfully matched at the next moment to be converted into certain trajectories. For the unmatched trajectories, increase the number of frames at which the trajectory disappears, and retain 30 frames for the unmatched trajectories.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A 2D multi-target tracking method based on visibility and variant IoU, characterized by: The method comprises the following steps: S1. Use a camera-based 2D detector to obtain the two-dimensional detection information of the target at each moment, set the detection confidence threshold, define the detection greater than the detection confidence threshold as a high-score detection, and define the detection less than the detection confidence threshold as a low-score detection; S2. Perform Kalman prediction on the trajectory of the previous moment to obtain the predicted trajectory in the current frame; based on the predicted trajectory frame and trajectory confidence, calculate the visibility of each trajectory, and set the visibility threshold. Define the trajectory greater than the visibility threshold as an active trajectory, and define the trajectory less than the visibility threshold as an inactive trajectory; S3, the high-score detection and active track are matched by the cost fusion of variant IoU and appearance similarity to obtain unmatched high-score detection and unmatched active track; S4, add the unmatched active and inactive trajectories, and then perform cost fusion matching of variant IoU and appearance similarity with the remaining unmatched high-score detections; S5, perform variant IoU matching on the remaining unmatched trajectories and the low-score detections to obtain the final successfully matched detection-trajectory pairs, unmatched detections, and unmatched trajectories; S6. Manage the trajectory based on the trajectory management mechanism.

2. The 2D multi-target tracking method according to claim 1, characterized in that: Step S2 includes: S21, performing Kalman prediction on the trajectory at the previous moment to obtain the predicted trajectory of the trajectory of the previous frame at the current moment; S22. Calculate the intersection-and-union ratio of any two trajectories: Among them, b A , b B They represent the bounding box of trajectory A and the bounding box of trajectory B respectively; if the trajectory confidence of trajectory A is greater than the trajectory confidence of trajectory B, then the intersection area is the area blocked by trajectory B, otherwise, it is the area blocked by trajectory A; S23. Based on the area blocked by each track, the visibility is calculated as follows: In the formula, α represents the visibility of the trajectory, T area Represents the bounding box area of ​​the trajectory, T o-area Indicates the area where the trajectory is blocked; S24: Based on the visibility of all predicted trajectories, a visibility threshold is set to divide the trajectories into active trajectories and inactive trajectories.

3. The 2D multi-target tracking method according to claim 1, characterized in that: Step S3 includes: S31. Calculate the area factor a between the high-score detection and the active track: In the formula, A T , A D Represent the area of ​​the trajectory box and the detection box respectively, A T∩D Represents the intersection area of ​​the trajectory box and the detection box, A C Represents the area of ​​the minimum bounding box of the detection box and trajectory box; S32. Calculate the height factor hf between the high score detection and the active track: In the formula, h D 、h T Respectively represent the height of the detection box and the trajectory box, h C Represents the union height of the track box and the detection box; S33. Calculate the distance factor d between the high score detection and the active track: In the formula, ρ represents the distance between the center points of the two boxes, and c represents the distance between the diagonal points of the minimum circumscribed bounding box of the trajectory box and the detection box; S34. Calculate the variant IoU HYIoU based on the area factor a, height factor hf, and distance factor d: HYIoU=a+IoU·hf·d S35. Calculate the fusion cost of HYIoU and appearance similarity: In the formula, C, C Appr , C HYIoU They represent the correlation matrix, cosine distance matrix, and HYIoU distance matrix between active trajectories and high-score detections, respectively. α represents the visibility of the trajectory. S36. Use Hungarian matching to complete the matching between the active track and the high-score detection, and obtain unmatched active track and unmatched high-score detection.

4. The 2D multi-target tracking method according to claim 1, characterized in that: Step S4 includes: S41, adding the unmatched active trajectory and the inactive trajectory as the trajectory to be matched; calculating the area factor a′ between the unmatched high-score detection and the trajectory to be matched: In the formula, A T , A D Represent the area of ​​the trajectory box and the detection box respectively, A T∩D Represents the intersection area of ​​the trajectory box and the detection box, A C Represents the area of ​​the minimum bounding box of the detection box and trajectory box; S42, calculate the height factor hf′ between the unmatched high-score detection and the trajectory to be matched: In the formula, h D 、h T Respectively represent the height of the detection box and the trajectory box, h C Represents the union height of the track box and the detection box; S43, calculate the distance factor d′ between the unmatched high-score detection and the track to be matched: Where ρ represents the distance between the center point of the trajectory box and the detection box, and c represents the distance between the diagonal point of the smallest circumscribed bounding box of the trajectory box and the detection box. S44. Calculate the variant IoU HYIoU′ according to the area factor a′, the height factor hf′, and the distance factor d′: HYIoU′=a′+IoU·hf′·d′ S45. Calculate the fusion cost of HYIoU′ and appearance similarity: Among them, C′, C′ Appr , C′ HYIoU They represent the correlation matrix, cosine distance matrix, and HYIoU distance matrix between the unmatched high-score detections and the trajectories to be matched, respectively. α represents the visibility of the trajectory. S46. Use Hungarian matching to complete the matching between the unmatched high-score detection and the trajectory to be matched, and obtain the unmatched trajectory and the unmatched detection.

5. The 2D multi-target tracking method according to claim 1, characterized in that: Step S5 includes: S51, calculate the area factor a″ between the low-score detection and the remaining unmatched tracks: In the formula, A T , A D Represent the area of ​​the trajectory box and the detection box respectively, A T∩D Represents the intersection area of ​​the trajectory box and the detection box, A C Represents the area of ​​the minimum bounding box of the detection box and trajectory box; S52, calculate the height factor hf″ between the low-score detection and the remaining unmatched tracks: In the formula, h D 、h T Respectively represent the height of the detection box and the trajectory box, h C Represents the union height of the track box and the detection box; S53, calculate the distance factor d″ between the low-score detection and the remaining unmatched tracks: Where ρ represents the distance between the center point of the trajectory box and the detection box, and c represents the distance between the diagonal point of the minimum circumscribed bounding box of the trajectory box and the detection box. S54. Calculate variant IoU HYIoU according to area factor a″, height factor hf″, and distance factor d″: HYIoU″=a″+IoU·hf″·d″ S55. Use HYIoU″ as the cost to complete the matching between the low-score detection and the remaining unmatched tracks, and obtain the final successfully matched detection-track pairs, unmatched detections, and unmatched tracks.

6. The 2D multi-target tracking method according to claim 1, characterized in that: Step S6 includes: S61, performing Kalman update for the successfully matched detection-trajectory pair; S62, for the detections that are not successfully matched, the detections that are greater than the threshold are initialized as new uncertain state trajectories, which need to be successfully matched at the next moment to be converted into a certain state trajectory; S63, for the unmatched trajectory, increase the number of trajectory disappearance frames, and retain 30 frames for the unmatched trajectory.

Images