A multi-target tracking method and system

By combining deep learning and Kalman filtering algorithms with similarity matching and high/low threshold strategies, the target tracking method is optimized, solving the problems of missed detection and false detection in existing target detection technologies and improving the safety of autonomous vehicles.

CN116703962BActive Publication Date: 2026-11-17ZHENGZHOU YUTONG BUS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310529165.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2026-11-17
Estimated Expiration
2043-05-11

AI Technical Summary

Technical Problem

Existing technologies suffer from missed detections and false detections in target tracking, leading to safety risks for autonomous vehicles, especially when using LiDAR point clouds, making it difficult to balance the accuracy of target detection.

Method used

A deep learning model is used to collect point cloud information in real time, and the Kalman filter algorithm is used to predict the target location. The target is filtered by similarity matching and high and low threshold strategies. The pedestrian target tracking is optimized by using secondary association conditions. The target is managed by combining timestamps and lifecycle to reduce false detections and false negatives.

Benefits of technology

It improves the accuracy of target detection, reduces the safety risks of autonomous vehicles, increases the target detection rate, and reduces false detections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116703962B_ABST
    Figure CN116703962B_ABST
Patent Text Reader

Abstract

This invention relates to the field of target tracking technology, and in particular to a multi-target tracking method and system. This solution tracks targets using point cloud deep learning, filters the target's position and angle, and uses similarity matching to associate targets in adjacent frames, stabilizing the target output state and improving the target detection rate. Simultaneously, based on the target's score, a high-low threshold algorithm is used to add the position, angle, and velocity information of new targets that meet the high-threshold strategy to the constructed tracking target information table before tracking, thus reducing false detections.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a multi-target tracking method and system. Background Technology

[0002] Object tracking is a crucial task in computer vision. The classic goal of object tracking is to locate the position of an object within each frame of a video image and generate its trajectory. Existing vision-based object tracking mainly falls into two categories: point object tracking and object tracking that occupies a certain area and possesses features such as texture and contour. Currently, object tracking technology is widely used in fields such as autonomous driving, intelligent video surveillance, human-computer interaction, robot vision navigation, virtual reality, and medical diagnostics. Its application in the field of autonomous driving, in particular, has attracted widespread attention.

[0003] Chinese invention patent application CN109118523A discloses an image target tracking method based on YOLO, with the following steps: S1, input video; S2, use the YOLO target detection network to detect targets and initialize the Kalman filter; S3, detect the current frame image. If a target is detected, proceed to step S4; otherwise, execute step S5; S4, calculate the intersection-over-union (IoU) ratio between the detected and predicted positions of the target in the current frame image. If the IoU ratio is greater than a preset threshold, use the detected position of the target as the target's position in the current frame; otherwise, return to step S2; S5, perform keypoint matching between the target's position in the previous frame and the predicted position of the target in the current frame. If the number of matching pairs is greater than a preset threshold, obtain the target's position in the current frame; otherwise, return to step S2; S6, check if the video detection has ended. If so, end the tracking; otherwise, return to step S3.

[0004] This approach has two drawbacks: 1) When associating the target's current frame position with its predicted position in the current frame, if the calculated intersection-union ratio (IU) is less than a pre-set threshold, the association is considered to have failed, leading to false positives and missed detections; 2) When tracking targets in the current frame, the target's score is not considered. Because targets inferred through deep learning may have different scores – some with lower scores may still be real targets, while others with higher scores may be false targets. Without filtering, directly establishing a tracking mechanism for all targets in the current frame can easily release false targets, resulting in multiple detections. Both of these drawbacks reduce the accuracy of target detection, potentially causing autonomous vehicles to brake incorrectly or fail to brake when they should, posing a driving safety risk.

[0005] LiDAR is a crucial sensor in the field of autonomous driving. Relying on the continuous scanning of LiDAR beams, the perception system of autonomous vehicles can acquire point cloud information of the surrounding scene. Deep learning-based object detection algorithms based on LiDAR point clouds filter detected bounding boxes according to a set score, ultimately outputting bounding boxes with a category score higher than the set value. However, setting a low threshold results in false positives, while setting a high threshold leads to missed detections; the algorithm struggles to balance these two scenarios. Furthermore, although LiDAR point clouds possess rich feature information, the relative position of the target to the autonomous vehicle, and the possibility of incomplete point cloud formations due to occlusion or intersections, can easily lead to missed detections and cause the autonomous vehicle to brake unnecessarily or fail to brake when it should, posing safety risks. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-target tracking method to solve the problem of missed and false detections in target detection in the prior art, which can easily cause traffic safety accidents; and to provide a multi-target tracking system for implementing the above method.

[0007] To address the aforementioned technical problems, this invention provides a multi-target tracking method, comprising the following steps:

[0008] 1) Collect point cloud information in real time and obtain the target of the current frame through a deep learning model;

[0009] 2) The Kalman filter algorithm is used to predict the position of the target box in the previous frame in the current frame. The corresponding predicted box is obtained and its similarity is matched with the actual detection box in the current frame. The targets associated by similarity matching are updated with their current position, angle and velocity information by the Kalman filter algorithm, and this information is put into the target set.

[0010] 3) If the target is detected in the first frame and meets the high threshold strategy, then the target's position, angle, and velocity information are added to the constructed tracking target information table; the high threshold strategy means that the target's deep learning score exceeds a set threshold.

[0011] 4) Traverse all targets in the target information table and delete targets that exceed the set storage time threshold from the target set; if the lifespan of a target in the target set is higher than the lifespan threshold, output the target being tracked; otherwise, do not output the target. The lifespan refers to the set cumulative number of frames in which the target has been detected. Beneficial effects: This invention tracks targets using point cloud deep learning, filters the target's position and angle, and uses similarity matching to associate targets in adjacent frames, stabilizing the target output state and improving the target detection rate. Simultaneously, based on the target's score, a high-low threshold algorithm is used to add the position, angle, and velocity information of new targets that meet the high-threshold strategy to the constructed target information table before tracking, thus reducing false detections.

[0012] Furthermore, for targets that are not associated through step 2), if the target is a pedestrian type target, then a secondary association is performed on the pedestrian type target using the secondary association condition. For pedestrian type targets that are associated through the secondary association condition, their current position, angle, and velocity information are updated using the Kalman filter algorithm and added to the target set. The secondary association condition refers to the similarity of at least one of the position, angle, velocity, and motion trajectory information between the predicted box and the actual detected box of the detected pedestrian type target reaching a set threshold.

[0013] Beneficial effects: Since pedestrian targets are relatively small and important, the actual detection box size of the pedestrian in the current frame is small and easily affected by measurement errors. At the same time, the predicted box size is also relatively small. Therefore, the cross-union ratio between the predicted box and the detection box in the current frame is sometimes small and difficult to match, which can easily lead to missed detections. Therefore, a secondary association condition is used to further filter pedestrian targets to reduce the missed detection rate of pedestrian targets.

[0014] Furthermore, targets that are still not associated using the secondary association condition include targets detected in the first frame and existing targets that do not appear in the actual detection box of the current frame. For targets detected in the first frame, the position, angle, and velocity information collected in the target box of the corresponding frame are added to the target set. For existing targets that do not appear in the actual detection box of the current frame, the position, angle, and velocity information predicted by the corresponding prediction box are added to the target set.

[0015] Beneficial effects: Targets that are not associated by similarity matching and secondary association conditions include targets detected in the first frame and existing targets that do not appear in the actual detection box in the current frame. Target information in these two cases is obtained through the target box and predicted box in the current frame, and added to the target set. This achieves the acquisition of information for all targets, thereby reducing the false negative rate of target detection.

[0016] Furthermore, if the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

[0017] Beneficial effect: By comparing the deep learning score of the detected target in the first frame with the set value, only targets with scores higher than the set value are tracked, which can reduce the false detection of targets.

[0018] Furthermore, if the time difference between the current timestamp and the last update timestamp of the target in the target set exceeds the set storage time threshold, or if the time period corresponding to the number of consecutively decreasing frames of the target's lifecycle in the target set exceeds the set storage time threshold, then the target is deleted.

[0019] Beneficial effect: Targets that have not appeared within a set time range can be filtered out by timestamp or target lifecycle. Since such targets have not been detected for a long time, it can be assumed that their probability of appearing in subsequent detection frames is extremely small and they no longer have tracking value. Therefore, they are deleted to reduce the false detection rate of real targets.

[0020] Furthermore, the similarity matching in step 2) refers to calculating the intersection-union ratio (IUU) between the predicted bounding box and the detected bounding box in the current frame, and then completing the matching using the Hungarian algorithm.

[0021] The present invention also provides a multi-target tracking system, the system including a processor and a memory, wherein the memory is used to store executable instructions of the processor; the processor is configured to perform the multi-target tracking method described above by executing the executable instructions. Attached Figure Description

[0022] Figure 1 This is a flowchart of the multi-target tracking method of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical principles and practical applications of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Examples of multi-target tracking methods:

[0025] This invention provides a multi-target tracking method. First, targets in a point cloud are acquired using a deep learning algorithm. The position and angle of the targets are then processed using a Kalman filter algorithm to stabilize the output state of the targets. Simultaneously, a high-threshold strategy is adopted based on the deep learning score of the targets to identify and track target types, which reduces false detections and improves the target detection rate. Then, different tracking strategies are adopted according to the target type. For targets of the pedestrian type that are not associated, further filtering is performed using historical information to reduce the false negative rate of pedestrian type targets. Finally, targets that meet the tracking conditions are derived based on the target lifecycle.

[0026] Figure 1 The flowchart of this method is shown below, and the specific steps are as follows:

[0027] (1) Collect point cloud information and obtain the target of the current frame through a deep learning model;

[0028] (2) Continuously update the timestamp;

[0029] (3) Use the Kalman filter algorithm to predict the position of the target box in the previous frame in the current frame to obtain the predicted box. The cross-union ratio between the predicted box and the actual detection box in the current frame is used as the similarity when the two are matched. Then, the matching is completed by the Hungarian algorithm.

[0030] For targets associated with the current frame and the predicted frame, the optimal position, optimal angle, and optimal velocity of the tracked target are updated using the Kalman filter algorithm (the calculation method for optimal position, optimal angle, and optimal velocity is as follows: a position-velocity-time model and an angle-time model are established in the Kalman filter algorithm, and the optimal position, optimal velocity, and optimal angle of the current frame are the posterior estimates from stage two of the Kalman filter algorithm), and then added to the target set;

[0031] This embodiment categorizes detected targets into two types: pedestrians and non-pedestrians. The target type can be inferred using deep learning algorithms. Since pedestrian targets are relatively small and important, the actual detection box size of a pedestrian in the current frame is small and easily affected by measurement errors. Simultaneously, the predicted box size is also relatively small. Therefore, the intersection-over-union (IoU) ratio between the predicted box and the current frame's detection box is sometimes low, making matching difficult and leading to missed detections. Thus, pedestrian targets that are not associated are further filtered using a secondary association condition. This secondary association condition refers to the similarity between the predicted box and the actual detection box of a detected pedestrian target after fusing information such as position, angle, velocity, and trajectory. If the set similarity threshold is met, the target is considered an associated pedestrian target between the current frame and the predicted frame. The Kalman filter algorithm is then used to update the target's current optimal position, optimal angle, and optimal velocity, and the target is added to the target set.

[0032] Targets in the current frame's target box and predicted box that are not associated by similarity matching and secondary association conditions include targets detected in the first frame (i.e., new targets appearing for the first time) and existing targets not appearing in the actual detection box of the current frame. For targets detected in the first frame, the position, angle, and velocity information collected from the corresponding target box in that frame are added to the target set; for existing targets not appearing in the actual detection box of the current frame, the position, angle, and velocity information predicted by the corresponding predicted box are added to the target set. In this embodiment, the secondary association condition refers to the similarity between the predicted box and the actual detection box of a pedestrian-type target after fusing information such as position, angle, velocity, and motion trajectory reaching a set threshold. As another implementation, the secondary association condition can also be the similarity of at least one of the information such as position, angle, velocity, and motion trajectory between the predicted box and the actual detection box of a pedestrian-type target.

[0033] (4) Calculate the velocity confidence of each target in the target set (the method for velocity confidence is to calculate the standard deviation of the velocity of the target in the last 5 frames. If the standard deviation is less than the set threshold, the velocity confidence of the target is considered to be high, otherwise it is considered to be low). Update the velocity confidence information of each target in the tracking target information table.

[0034] Since the type of the detected target has been identified in step (3), if the deep learning score of the detected target is less than the set value, the target is considered to have low credibility, that is, the target may not be the identified type or belong to any of the target types to be tracked, so it will no longer be tracked; if the deep learning score of the detected target is greater than the set value, the target is considered to have high credibility, that is, the actual type of the target is likely to match the identified type and has tracking value, so it will continue to be tracked.

[0035] Here, the tracked targets are divided into targets detected in the first frame and targets not detected in the first frame. For targets detected in the first frame, a high threshold strategy is set using deep learning scores. If the target detected in the first frame exceeds the set deep learning score value for the corresponding type (e.g., 0.7 or 0.8), it means that the target meets the high threshold strategy and the target type to which the target belongs can be identified. Therefore, the target needs to be tracked. Thus, a tracking target information table including information such as position, angle, and velocity is established for the target. The tracking target information table is initialized and maintained. Otherwise, it is not tracked.

[0036] For targets not detected in the first frame, if the actual detection box is detected in the frame, it is associated with the predicted box. After association, the filtered position, angle, and velocity values, as well as the timestamp of the corresponding frame, are updated in the tracking target information table. If the actual detection box is not detected in the frame, the position, angle, and velocity information predicted by the corresponding predicted box are updated in the tracking target information table.

[0037] Then, iterate through all targets in the target information table. If the time difference between the current timestamp and the last updated timestamp of the target in the target set exceeds the set storage time threshold, the target is considered to have disappeared and is therefore deleted from the target set. The set storage time threshold refers to the time difference between the current timestamp and the last updated timestamp of each target in the target set.

[0038] (5) Calculate the lifetime of each target in the target set. The lifetime refers to the number of frames in which the target appears in the actual detection frame, that is, the cumulative number of frames in which the target is detected. For example, if a pedestrian-type target appears in the 3rd frame of detection, its lifetime is 1; if it appears in the 4th frame, its lifetime is increased by 1 and becomes 2; if it does not appear in the 5th frame, its lifetime is decreased by 1 and becomes 1; if it appears in the 6th, 7th, and 8th frames, its lifetime is increased by 3 and becomes 4, and so on, to calculate the lifetime of the target.

[0039] If the lifespan of a target in the target set is greater than the set lifespan threshold (e.g., 3 or 4 frames), the target is considered to have high credibility and is pushed out to other modules outside the system for their use; otherwise, the target continues to be tracked within the system, and subsequent status is observed to determine whether to push it out.

[0040] (6) Jump to step (1) to perform the next round of target tracking.

[0041] This embodiment deletes targets from the target set based on timestamps. Alternatively, targets can be deleted based on the number of consecutive frames in which their lifespan decreases. For example, if a pedestrian-type target is not detected for four consecutive frames (meaning its lifespan has decreased by 4), it is considered to have disappeared and is therefore deleted from the target set.

[0042] The multi-target tracking method of the present invention can track and detect targets in the environment surrounding autonomous vehicles in various application scenarios. As other implementation methods, it can also be applied to fields such as intelligent video surveillance, human-computer interaction, robot vision navigation, virtual reality, and medical diagnosis.

[0043] Multi-target tracking system example:

[0044] The multi-target tracking system of this embodiment includes a processor and a memory. The processor executes a computer program stored in the memory to enable the present invention to implement the method of the above-described method embodiment. That is, the method in the above method embodiment should be understood as a flow of multi-target tracking methods that can be implemented by computer program instructions. These computer programs can be provided to issue instructions to the processor, causing the processor to execute these instructions to produce the functions specified for implementing the above-described method flow.

[0045] The processor referred to in this embodiment is a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA); the memory referred to in this embodiment includes a physical device for storing information, which usually involves digitizing the information and then storing it using media such as electricity, magnetism, or optics.

[0046] The system consisting of the aforementioned memory, processor, and computer program is implemented in a computer by the processor executing the corresponding program instructions. The processor can run various operating systems, such as Windows, Linux, Android, and iOS.

[0047] As an alternative implementation, the system may also include a display for showing the results of the calculated quantities for staff reference.

Claims

1. A multi-target tracking method characterized by, Includes the following steps: 1) Collect point cloud information in real time and obtain the target of the current frame through a deep learning model. The target includes its type. 2) The Kalman filter algorithm is used to predict the position of the target box in the previous frame in the current frame, and the corresponding predicted box is obtained. The predicted box is then matched with the actual detected box in the current frame to determine whether they are associated. If they are not associated and the target is a pedestrian, a secondary association condition is used to associate the target. The targets associated by similarity matching and the secondary association condition are updated with their current position, angle, and velocity information by the Kalman filter algorithm, and this information is added to the target set. The secondary association condition refers to the similarity of at least one of the position, angle, velocity, and motion trajectory information between the predicted box and the actual detected box of the pedestrian-type target reaching a set threshold. 3) If the target is not detected in the first frame, or if the target is detected in the first frame and meets the high threshold strategy, then the position, angle and velocity information of the target are added to the constructed tracking target information table; the high threshold strategy means that the deep learning score of the target exceeds the set threshold. 4) Traverse all targets in the target information table and delete targets that exceed the set storage time threshold from the target set; If the lifespan of a target in the target set is higher than the lifespan threshold, the tracked target is output; otherwise, the target is not output. The lifespan refers to the cumulative number of frames that the target has been detected.

2. The multi-target tracking method of claim 1, wherein, The secondary association condition refers to the similarity between the predicted bounding box of the pedestrian type target and the actual detected bounding box after fusing the position, angle, speed and motion trajectory information reaching a set threshold.

3. The multi-target tracking method according to claim 2, characterized in that, Targets that are still not associated using the secondary association conditions include targets detected in the first frame and existing targets that do not appear in the actual detection box of the current frame. For targets detected in the first frame, the position, angle and velocity information collected in the target box of the corresponding frame are put into the target set. For existing targets that do not appear in the actual detection box of the current frame, the position, angle and velocity information predicted by the corresponding prediction box are put into the target set.

4. The multi-target tracking method according to claim 1, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

5. The multi-target tracking method according to claim 2, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

6. The multi-target tracking method according to claim 3, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

7. The multi-target tracking method according to claim 4, characterized in that, If the time difference between the current timestamp and the last update timestamp of the target in the target set exceeds the set storage time threshold, or if the time period corresponding to the number of consecutively decreasing frames of the target's lifecycle in the target set exceeds the set storage time threshold, then the target is deleted.

8. The multi-target tracking method according to claim 1, characterized in that, The similarity matching in step 2) refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

9. The multi-target tracking method according to claim 2, characterized in that, The similarity matching in step 2) refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

10. The multi-target tracking method according to claim 3, characterized in that, The similarity matching in step 2) refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

11. A multi-target tracking system, characterized in that, Includes a processor and memory, the memory being used to store executable instructions for the processor; the processor is configured to execute a multi-target tracking method by executing the executable instructions, including: Real-time acquisition of point cloud information; obtaining the target in the current frame through a deep learning model; the target includes its type. The Kalman filter algorithm is used to predict the position of the target bounding box in the previous frame in the current frame, obtaining the corresponding predicted bounding box. This predicted bounding box is then matched with the actual detected bounding box in the current frame based on similarity to determine if they are associated. If they are not associated and the target is a pedestrian, a secondary association condition is used to re-associate the target. Targets associated through similarity matching and the secondary association condition are updated with their current position, angle, and velocity information using the Kalman filter algorithm, and this information is added to the target set. The secondary association condition refers to the similarity between the predicted bounding box and the actual detected bounding box of a pedestrian-type target reaching at least a set threshold in terms of position, angle, velocity, and motion trajectory information. If the target is not detected in the first frame, or if the target is detected in the first frame and meets the high threshold policy, then the target's position, angle, and velocity information are added to the constructed tracking target information table; the high threshold policy means that the target's deep learning score exceeds a set threshold. Traverse all targets in the target information table and delete targets that exceed the set storage time threshold from the target set; if the lifespan of a target in the target set is higher than the lifespan threshold, output the tracked target; otherwise, do not output the target; the lifespan refers to the set cumulative number of frames that the target has been detected.

12. The multi-target tracking system according to claim 11, characterized in that, The secondary association condition refers to the similarity between the predicted bounding box of the pedestrian type target and the actual detected bounding box after fusing the position, angle, speed and motion trajectory information reaching a set threshold.

13. The multi-target tracking system according to claim 12, characterized in that, Targets that are still not associated using the secondary association conditions include targets detected in the first frame and existing targets that do not appear in the actual detection box of the current frame. For targets detected in the first frame, the position, angle and velocity information collected in the target box of the corresponding frame are put into the target set. For existing targets that do not appear in the actual detection box of the current frame, the position, angle and velocity information predicted by the corresponding prediction box are put into the target set.

14. The multi-target tracking system according to claim 11, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

15. The multi-target tracking system according to claim 12, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

16. The multi-target tracking system according to claim 13, characterized in that, If the deep learning score of the detected target in the first frame is less than a set threshold, then the target will not be tracked.

17. The multi-target tracking system according to claim 14, characterized in that, If the time difference between the current timestamp and the last update timestamp of the target in the target set exceeds the set storage time threshold, or if the time period corresponding to the number of consecutively decreasing frames of the target's lifecycle in the target set exceeds the set storage time threshold, then the target is deleted.

18. The multi-target tracking system according to claim 11, characterized in that, The similarity matching refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

19. The multi-target tracking system according to claim 12, characterized in that, The similarity matching refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

20. The multi-target tracking system according to claim 13, characterized in that, The similarity matching refers to calculating the intersection-union ratio between the predicted bounding box and the current frame detection box, and then completing the matching using the Hungarian algorithm.

Citation Information

Patent Citations

  • Image object tracking method based on YOLO

    CN109118523A

  • Remote sensing video vehicle target detection and tracking method based on dynamic association model

    CN110390292A

  • Improved DeepSort target detection tracking method based on YOLOv4

    CN113160274A

  • Real-time multi-target tracking method and system in automatic driving

    CN114445453A

  • Aircraft multi-target tracking method based on improved YOLOV5 algorithm

    CN114529799A