A multi-target tracking method and device, storage medium, and terminal

Abstract

A multi-target tracking method and device, storage medium, and terminal, the method comprising: during the tracking of multiple targets, obtaining the tracking result of the first target among the multiple targets at the current moment; if the tracking result is an unmatched tracking, determining the cluster to which the first target belongs based on the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and similar motion attributes; based on the detection results of the targets other than the first target in the cluster at the current moment and the relative motion relationship between the targets in the cluster, restoring the trajectory of the first target at the current moment. The solution of the present invention can predict and track the trajectory of the obscured object, which is conducive to ensuring the integrity of target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a multi-target tracking method and device, a storage medium, and a terminal. Background Art

[0002] The purpose of Multiple Objects Tracking (MOT) is to estimate the trajectory of a target based on its identity in an image sequence or video. Recently, due to the advancement of object detection technology, many tracking-by-detection (TBD) techniques have made multi-target tracking develop rapidly. In detection-based tracking techniques, the target object is first detected, and then the tracking algorithm uses the detection results to estimate the target trajectory through a data association algorithm.

[0003] At present, there are many methods that have achieved good results in multi-target tracking, but most of the methods are to continuously improve the target detector so that the appearance features of the target extracted by the target detector are more accurate, thereby reducing the number of false detections and missed detections.

[0004] However, the existing technologies rarely involve trajectory prediction and recovery of occluded objects. As a result, most existing tracking algorithms cannot predict and track the trajectory of occluded objects due to poor appearance features, resulting in the inability to completely track the motion trajectory of the object. Summary of the invention

[0005] The technical problem solved by the present invention is how to achieve trajectory prediction and tracking of an obscured object so as to completely track the motion trajectory of the object.

[0006] To solve the above technical problems, an embodiment of the present invention provides a multi-target tracking method, comprising: during tracking of multiple targets, obtaining the tracking result of the first target among the multiple targets at the current moment; if the tracking result is an unmatched tracking, determining the cluster to which the first target belongs based on the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object properties and similar motion properties; and restoring the trajectory of the first target at the current moment based on the detection results of targets other than the first target in the cluster at the current moment and the relative motion relationship between the targets in the cluster.

[0007] Optionally, obtaining the tracking result of the first target among the multiple targets at the current moment includes: predicting the prediction result of the first target at the current moment based on the historical detection result of the first target; and performing target matching on the prediction result with the object detection result obtained by detection at the current moment to obtain the tracking result of the first target.

[0008] Optionally, for any one of the historical detection results, prediction results and object detection results, the detection result is represented based on a detection frame, the detection frame corresponds one-to-one to a dictionary, and the dictionary includes at least one of the following: identity information of the tracked object; the position, size, movement speed and size change speed of the detection frame; the tracking status of the trajectory; and the number of frames the trajectory is maintained.

[0009] Optionally, the target matching of the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target includes: when the tracking state of the trajectory in the historical detection results of the first target is a determined tracking state, cascade matching of the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target.

[0010] Optionally, the cascade matching refers to using data association algorithms of multiple cost matrices to perform target matching, and the data association algorithms of the multiple cost matrices include at least a first data association algorithm and a second data association algorithm; the cascade matching of the prediction result with the object detection result detected at the current moment to obtain the tracking result includes: using the first data association algorithm to perform target matching on the prediction result and the object detection result to obtain a first matching result; if the first matching result is not a matching pair, using the second data association algorithm to perform target matching on the prediction result and the object detection result to obtain a second matching result, and determining the second matching result as the tracking result.

[0011] Optionally, the target matching of the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target includes: when the tracking state of the trajectory in the historical detection results of the first target is uncertain tracking, using a single data association algorithm to target match the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target.

[0012] Optionally, the data association algorithms of the multiple cost matrices include: IoU matching and SIOA matching.

[0013] Optionally, determining the cluster to which the first target belongs based on the historical detection results of the multiple targets includes: determining the object attributes and motion attributes of each target based on the historical detection results of each target; searching from the multiple targets for a target that has similar motion attributes and the same object attributes as the first target and is spaced from the first target by a distance less than a preset threshold; if the search is successful, determining that the searched target forms the cluster with the first target.

[0014] Optionally, the object attributes include the identity category of the target; the motion attributes include: the motion speed of the target; the motion direction of the target; and the trajectory of the target.

[0015] Optionally, the method further includes: if the first target cannot form a cluster with other targets among the multiple targets, predicting the trajectory of the first target at the current moment according to a uniformly accelerated motion model.

[0016] Optionally, the method also includes: if the first target reappears within a preset matching time limit and matches the recovered or predicted trajectory of the first target at the current moment, the tracking state of the trajectory in the historical detection results of the first target is determined as uncertain tracking, and tracking is performed at the next moment.

[0017] Optionally, the method further includes: if the tracking result of the first target is a matching pair, updating the prediction result of the first target at the current moment by using an improved Kalman filter.

[0018] Optionally, before determining the cluster to which the first target belongs based on the historical detection results of the multiple targets, the method further includes: smoothing the historical detection results by using an improved Kalman filter.

[0019] To solve the above technical problems, an embodiment of the present invention also provides a multi-target tracking device, including: an acquisition module, used to obtain the tracking result of the first target among the multiple targets at the current moment during tracking of multiple targets; a determination module, if the tracking result is an unmatched tracking, determines the cluster to which the first target belongs based on the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object properties and motion properties; a recovery module, used to restore the trajectory of the first target at the current moment based on the detection results of the targets other than the first target in the cluster at the current moment and the relative motion relationship between the targets in the cluster.

[0020] In order to solve the above technical problem, an embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0021] To solve the above technical problem, an embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above method when running the computer program.

[0022] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0023] The embodiment of the present invention provides a multi-target tracking method, including: during the tracking of multiple targets, obtaining the tracking result of the first target among the multiple targets at the current moment; if the tracking result is an unmatched tracking, determining the cluster to which the first target belongs according to the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and similar motion attributes; according to the detection results of the targets other than the first target in the cluster at the current moment and the relative motion relationship between the targets in the cluster, restoring the trajectory of the first target at the current moment. Thus, the adoption of this embodiment can predict and track the trajectory of the obscured object, which is conducive to ensuring the integrity of target tracking. Specifically, when the tracking result is an unmatched tracking, it indicates that although the position of the first target at the current moment is predicted, the detection result (i.e., the detection frame) at the current moment does not match, that is, occlusion or detection frame loss occurs. Based on this embodiment, when the occlusion and detection frame loss problems occur in a multi-target motion scene, the targets are clustered by considering the relative motion relationship between the targets, and then the trajectory recovery of the occluded targets in the cluster is achieved by sharing features such as speed and acceleration. Therefore, when the first target is tracked again in the future, the occurrence of identity information matching failure can be effectively reduced.

[0024] Furthermore, if the first target cannot form a cluster with other targets in the multiple targets, the trajectory of the first target at the current moment is predicted according to the uniformly variable motion model. Thus, for individuals that cannot form a cluster, the uniformly variable motion model is used to predict the motion state of the individual during the occlusion period to achieve trajectory prediction, thereby also reducing the occurrence of identity information matching failures.

[0025] Furthermore, when the tracking state of the trajectory in the historical detection result of the first target is a determined tracking state, the prediction result is cascade matched with the object detection result obtained at the current moment to obtain the tracking result of the first target. Thus, the matching accuracy can be effectively improved.

[0026] Furthermore, if the trajectory of the first target at the current moment is still within the preset matching time limit, the tracking state of the trajectory in the historical detection result of the first target is determined as uncertain tracking, and the tracking at the next moment is performed. Thus, within the preset matching time limit, the predicted or restored trajectory of the first target at the current moment is set as unconfirmed tracking and enters the matching at the next moment to ensure that the tracking of the first target is still continued during the period when the first target is occluded. After the occlusion of the first target ends, the identity information can be quickly matched based on the predicted or restored trajectory of the first target, so as to match the re-detected object to the historical trajectory of the first target as soon as possible.

[0027] Furthermore, by adopting this implementation, it is possible to continuously track multiple targets around the unmanned vehicle using two-dimensional visual information, three-dimensional detection results, relative motion relationships between tracking targets, and motion models, so as to solve the problems of trajectory interruption and identity information conversion caused by occlusion between targets in multi-target tracking technology in unmanned driving scenarios. Specifically, when the vehicles or people around the unmanned vehicle are blocked, by adopting this implementation, the track of the blocked targets can be restored or predicted to ensure that the tracking of the targets around the unmanned vehicle is always complete, thereby making the unmanned vehicle safer and more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a multi-target tracking method according to an embodiment of the present invention;

[0029] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S101;

[0030] Figure 3 This is a flow chart of an embodiment of the present invention for predicting the position and size of a detection frame at the next moment through an improved Kalman filter;

[0031] Figure 4 is a flow chart of an occlusion recovery and trajectory prediction process according to an embodiment of the present invention;

[0032] Figure 5 It is a structural schematic diagram of a multi-target tracking device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] As mentioned in the background art, most existing tracking algorithms cannot predict and track the trajectory of occluded objects due to poor appearance features, resulting in the inability to completely track the motion trajectory of the object.

[0034] To solve the above technical problems, an embodiment of the present invention provides a multi-target tracking method, comprising: during tracking of multiple targets, obtaining the tracking result of the first target among the multiple targets at the current moment; if the tracking result is an unmatched tracking, determining the cluster to which the first target belongs based on the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object properties and similar motion properties; and restoring the trajectory of the first target at the current moment based on the detection results of targets other than the first target in the cluster at the current moment and the relative motion relationship between the targets in the cluster.

[0035] Therefore, the adoption of this implementation scheme can predict and track the trajectory of the obscured object, which is conducive to ensuring the integrity of the target tracking. Specifically, when the tracking result is an unmatched tracking, it means that although the position of the first target at the current moment is predicted, the detection result (i.e., the detection frame) at the current moment does not match, that is, occlusion or detection frame loss occurs. Based on this implementation scheme, when there are occlusion and detection frame loss problems in multi-target motion scenes, the targets are clustered by considering the relative motion relationship between the targets, and then the trajectory of the occluded targets in the cluster is restored by sharing features such as speed and acceleration. Therefore, when the first target is tracked again in the future, the occurrence of identity information matching failure can be effectively reduced.

[0036] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0037] Figure 1 The present invention is a flowchart of a multi-target tracking method according to an embodiment of the present invention.

[0038] This embodiment can be applied to unmanned driving scenarios, such as using a multi-target tracking technology based on a monocular camera and a three-dimensional laser radar to monitor the environment around an unmanned vehicle to obtain the motion trajectory of objects around the unmanned vehicle. This embodiment can be integrated into the on-board controller of an unmanned vehicle to achieve real-time target tracking of objects around the unmanned vehicle.

[0039] By adopting this implementation, it is possible to continuously track multiple targets around the unmanned vehicle using two-dimensional visual information, three-dimensional detection results, relative motion relationships between tracking targets, and motion models, so as to solve the problems of trajectory interruption and identity information conversion caused by occlusion between targets in multi-target tracking technology in unmanned driving scenarios. Specifically, when the vehicles or people around the unmanned vehicle are blocked, the implementation can be used to recover or predict the track of the blocked targets to ensure that the tracking of the targets around the unmanned vehicle is always complete, thereby making the unmanned vehicle safer and more reliable.

[0040] refer to Figure 1 The multi-target tracking method described in this embodiment may include the following steps:

[0041] Step S101, during tracking of multiple targets, obtaining a tracking result of a first target among the multiple targets at a current moment;

[0042] Step S102, if the tracking result is unmatched tracking, determining the cluster to which the first target belongs according to the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and similar motion attributes;

[0043] Step S103 , restoring the trajectory of the first target at the current moment according to the detection results of the targets in the cluster other than the first target at the current moment and the relative motion relationship between the targets in the cluster.

[0044] In the step S101, the tracking result of each target at the current moment is obtained, wherein each target can be used as the first target to execute the subsequent solution.

[0045] Furthermore, the target may be an object that can be sensed around the driverless car, and the object may include traffic participants such as people, cars, animals, plants, and traffic lights.

[0046] Furthermore, the detection result can be characterized based on the detection box. The detection box corresponds to the dictionary one by one, and the dictionary includes at least one of the following contents: the identity information (ID) of the tracked object (i.e., the target); the position, size, movement speed and size change speed of the detection box (bbox); the tracking status of the trajectory (is_confirmed_or_not); the number of frames the trajectory is maintained (age). The correspondence between the dictionary and the detection box can be stored in a database and retrieved when needed.

[0047] For example, the center point of the detection box can be used to represent the position of the detection box.

[0048] For example, the size of the detection box can be used to describe the three-dimensional boundary information of the target.

[0049] For example, the tracking state of a trajectory includes definite tracking and uncertain tracking. When the same target is detected in three consecutive frames, the tracking state of the trajectory is marked as definite tracking in the dictionary of the detection box corresponding to the target.

[0050] Furthermore, the detection result may be a three-dimensional detection result, or a three-dimensional and two-dimensional detection result. The detection result may be obtained from a target detector, the input of which is the environmental data around the driverless car collected by the monocular camera and / or radar.

[0051] Furthermore, the tracking result may refer to a matching result. During tracking, the tracking result of each target at each moment can be classified into unmatched detections, unmatched tracks, and matches (also called successful matches). Among them, unmatched tracking means that although the position of the target at the next moment is predicted, the detection box detected at the next moment does not match; unmatched detection means that there is no matching detection, indicating that a new target to be tracked has appeared.

[0052] In one specific implementation, reference Figure 2 , the step S101 may include the following steps:

[0053] Step S1011, predicting a prediction result of the first target at the current moment according to a historical detection result of the first target;

[0054] Step S1012: performing target matching on the prediction result and the object detection result obtained at the current moment to obtain the tracking result of the first target.

[0055] Specifically, in step S1011, the position and size of the detection frame at time (A-1) can be used to predict the position and size of the detection frame at the next time (ie, time A) through an improved Kalman filter (also called an improved three-dimensional Kalman filter).

[0056] Furthermore, the previous moment, the current moment, and the next moment may be determined according to the frame rate of the monocular camera, and the detection result obtained based on the single-frame image corresponds to the detection result at a certain moment.

[0057] refer to Figure 3 The specific process of predicting the position and size of the detection frame at the next moment through the improved Kalman filter may include the following steps:

[0058] Step a, setting parameters, wherein the parameters may include a state transition model F, an observation model H, an observation noise matrix R, and a process noise matrix Q.

[0059] For example, the state transition model F can be expressed in matrix form as shown in formula (1):

[0060]

[0061] In formula (1), dt represents the frame interval between the previous and next states. In this example, dt=1.

[0062] For example, the observation model H can be expressed in matrix form as shown in formula (2):

[0063]

[0064] For example, the observation noise matrix R and the process noise matrix Q can be adjusted according to the specific circumstances of the actual application.

[0065] Next, execute step b to initialize the system state X0 and the covariance matrix P0.

[0066] Next, execute step c to predict the system. The prediction process may include two aspects:

[0067] First, based on formula (3), the system state value at time k-1 is Predict the system state value at time k (i.e. predicted value):

[0068]

[0069] Among them, F k is the state transition model F at time k; B k is the transfer noise matrix.

[0070] In this example, the system status value corresponds to the detection result, that is, the position and size of the detection box.

[0071] Second, based on formula (4), the error covariance at time k-1 is Forecast the error covariance at time k

[0072]

[0073] Among them, F k is the state transition model F at time k; Q k is the process noise matrix Q at time k; F k The transpose of .

[0074] Next, step d is executed to determine whether the system state value at time k predicted in step c successfully matches the system state value at time k actually detected. That is, the contents described in step S1012 are executed.

[0075] In step S1012, the object detection result obtained at the current moment (i.e., the system state value at moment k actually detected) and the prediction result (i.e., the system state value at moment k predicted in step c) can be processed differently according to the tracking state of the trajectory.

[0076] For example, when the tracking state of the trajectory in the historical detection result of the first target is a determined tracking state, the prediction result is cascade matched with the object detection result obtained at the current moment to obtain the tracking result of the first target. The cascade matching refers to target matching using a data association algorithm with multiple cost matrices. Among them, a model uses multiple evaluation methods, and different cost matrices can be understood as different evaluation methods.

[0077] For another example, when the tracking state of the trajectory in the historical detection results of the first target is uncertain tracking, a single data association algorithm is used to match the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target.

[0078] The data association algorithm may include: Intersection-over-Union (IoU) matching and Sum-of-Intersection-Over-target-Area (SIOA) matching. Furthermore, the data association algorithm may also include appearance matching (i.e. visual matching).

[0079] The data association algorithms of the multiple cost matrices used in the cascade matching include at least a first data association algorithm and a second data association algorithm. Accordingly, the first data association algorithm is first used to target match the prediction result with the object detection result to obtain a first matching result; if the first matching result is not a matching pair, the second data association algorithm is used to target match the prediction result with the object detection result to obtain a second matching result, and the second matching result is determined as the tracking result.

[0080] For example, for uncertain tracking, SIOA matching can be performed directly between the prediction result and the current detection result.

[0081] For example, for the tracking of the determined state, the IOU matching is first performed based on formula (5) to obtain the first matching result:

[0082]

[0083] Among them, IoU is the first matching result; area(A) is the area of ​​the prediction result; area(B) is the area of ​​the detection result.

[0084] The first matching results can be divided into matching pairs 1 (matches_1), unmatched detections (unmatcheddetections_1) and unmatched tracks (unmatched tracks_1).

[0085] SIOA matching is performed again for unmatched detections_1 and unmatched tracks_1 to obtain a second matching result, which follows formula (6):

[0086]

[0087] Among them, SIOA is the second matching result.

[0088] The second matching result can also be divided into matching pairs (matches), unmatched detections (unmatched detections) and unmatched tracks (unmatched tracks). The second matching result is the final tracking result at the current moment.

[0089] Furthermore, the judgment result of step d is affirmative, that is, a successful match refers to the combined result of the matching pairs obtained through IOU matching and SIOU matching processing.

[0090] The inventor of this application has found through analysis that the prior art usually uses appearance matching when matching targets. The matching accuracy is heavily dependent on the imaging quality of the monocular camera, and is easily affected by the environment, resulting in inaccurate appearance matching, and the calculation complexity is high. To address this problem, this implementation adopts cascade matching, which can achieve high-accuracy target matching without obtaining appearance features such as color information and texture information of the target.

[0091] For uncertain tracking, SIOA is used for matching first. If the matching degree is less than the preset threshold, it is directly discarded to reduce the amount of calculation.

[0092] In a variation, the cascade matching may be three-stage, such as sequentially using appearance matching, IOU matching, and SIOA matching.

[0093] Further, when the judgment result of step d is positive, the updating process of step f can be performed, and the updating process has the following three stages:

[0094] First, the Kalman gain is calculated based on formula (7):

[0095]

[0096] Wherein, K is the Kalman gain, which is used to adjust the weight of the predicted value and the observed value (ie, the detected value); H k is the observation model at time k; H k The transpose of R k is the observation noise matrix at time k.

[0097] Second, based on formula (8), the predicted value at time k is corrected and updated according to the observed value at time k:

[0098]

[0099] in, is the updated predicted value at time k; is the predicted value at time k before the update; Z k is the observation value at time k (i.e., the detection result).

[0100] Third, update the covariance matrix based on formula (9):

[0101]

[0102] in, is the updated covariance matrix at time k; is the covariance matrix at time k before the update.

[0103] The next time step c is executed, the updated predicted value and covariance matrix based on formulas (8) and (9) are used. The predicted value can be corrected through the updating process.

[0104] Further, when the judgment result of step d is negative, the smoothing process of step e may be performed, and the smoothing process has the following three stages:

[0105] First, the Kalman smoothing matrix is ​​calculated based on formula (10):

[0106]

[0107] Wherein, J is the Kalman smoothing matrix; is the covariance matrix at time k+1.

[0108] Secondly, based on formula (11), the updated value at time K is smoothed and corrected according to the smoothed value at time k+1, that is, the state value of Kalman backpropagation is calculated:

[0109]

[0110] in, is the predicted value at time k after smoothing; is the predicted value at time k+1 after smoothing.

[0111] Third, based on formula (12), the covariance matrix at time k is updated, that is, the error covariance of Kalman backpropagation is calculated:

[0112]

[0113] in, is the updated covariance matrix at time k; is the updated covariance matrix at time k+1; J T is the transpose of J.

[0114] The next time step c is executed, the prediction value and covariance matrix updated based on formulas (11) and (12) are used.

[0115] In a specific implementation, if the tracking result of the first target at the current moment obtained in step S101 is a matching pair, the following steps can be directly executed: Figure 3 The updating process shown in step f is to update the prediction result of the first target at the current moment through the improved three-dimensional Kalman filter, and then enter the matching at the next moment.

[0116] In a specific implementation, if the tracking result of the first target at the current moment obtained in step S101 is unmatched tracking, occlusion recovery and trajectory prediction are performed, wherein occlusion recovery corresponds to step S102 and step S103.

[0117] Specifically, refer to Figure 4 ,The occlusion recovery and trajectory prediction process can include the following steps:

[0118] Step S1021, smoothing the historical detection results by using an improved Kalman filter;

[0119] Step S1022, determining the object attributes and motion attributes of each target according to the historical detection results of each target;

[0120] Step S1023, searching from the multiple targets for a target having the same motion attribute and object attribute as the first target and a distance from the first target less than a preset threshold;

[0121] If the search is successful in step S1023, step S1024 is executed to determine whether the searched target and the first target form the cluster, and then step S1026 is executed. Figure 1 Step S103 in the above method;

[0122] If the search in step S1023 is unsuccessful, that is, the first target cannot form a cluster with other targets in the multiple targets, step S1025 is executed to predict the trajectory of the first target at the current moment according to the uniformly accelerated motion model.

[0123] Further, the smoothing process described in step S1021 can refer to Figure 3 The smoothing operation can eliminate the incorrect speed, position, and size of the detection frame caused by low performance and instability of the target detector. The step S1021 can be a preprocessing step for improving the overall calculation efficiency.

[0124] Furthermore, the historical detection results of each target used in step S1022 may be pre-stored in a database. For example, the detection results of each target obtained at time A-1 are pre-stored, and when a target is found to be blocked at time A, the stored data is retrieved to determine whether a cluster is formed. This can improve the computing efficiency.

[0125] Furthermore, the object attributes may include the identity category of the target; the motion attributes include: the motion speed of the target; the motion direction of the target; and the trajectory of the target.

[0126] For example, the identity categories of the target include people and cars, which can be determined based on the identity information of the tracked object in the detection box corresponding dictionary.

[0127] For example, in step S1023, it can be determined whether the detection frames corrected in step S1021 can form a cluster based on the size, speed, and position information of the detection frames. For example, for the detection frames corresponding to all detected targets, the detection frames of the same category and similar size, the detection frames with a spacing less than the preset threshold, and the detection frames with similar movement speeds and the same movement directions can be classified into the same cluster. The same category means that only people and people, cars and cars can form a cluster, and people will not form a cluster with cars.

[0128] Furthermore, the advantage of using the uniform speed variation model is its high computational efficiency, because the unmanned driving scenario requires fast response and real-time performance.

[0129] In a specific implementation, during the process of predicting or recovering the trajectory of the first target at the current moment based on steps S1021 to S1025 and step S103, if the first target reappears within a preset matching time limit and matches the recovered or predicted trajectory of the first target at the current moment, the tracking state of the trajectory in the historical detection results of the first target can be determined as uncertain tracking, and tracking at the next moment can be performed.

[0130] Specifically, continue to refer to Figure 4 , step S1026 may be executed to verify whether the matching result is successfully matched based on the visual features, that is, to determine whether the reappearing first target matches the restored or predicted trajectory of the first target at the current moment based on the visual features.

[0131] For example, after the occlusion ends, the first target reappears, and it can be determined whether it is still within the preset matching time limit. If it is within the preset matching time limit, an appearance feature matrix can be constructed based on the visual features to verify whether the prediction or recovery of the tracking of the first target at the current moment can minimize the appearance feature matrix (the matrix is ​​0 if the two objects are the same, and non-0 if they are different). The appearance feature matrix is ​​used to characterize the visual matching degree. For another example, appearance matching can be replaced by IOU matching.

[0132] If the predicted or restored detection frame matches the detection frame of the reappearing first target in appearance, it indicates that the reappearing is the first target. Therefore, the tracking state of the trajectory in the historical detection result of the first target can be determined as uncertain tracking, and tracking at the next moment can be performed.

[0133] If the first target still does not appear or the appearance matching is unsuccessful when the preset matching time limit is exceeded, the tracking record of the first target is deleted.

[0134] Furthermore, the preset matching time limit can be determined according to the frame rate of the monocular camera. The higher the frame rate of the input image, the closer the position distance of the objects in two adjacent frames. The monocular camera can be set on an unmanned vehicle.

[0135] Further, if the verification result of step S1026 is positive, that is, a match is made, then execute Figure 3 Step f is shown.

[0136] In a specific implementation, if the tracking result of the first target at the current moment obtained in step S101 is an unmatched detection because a new target appears, it is set as a new track and initialized, and then enters the matching at the next moment.

[0137] Further, after creating a new track and executing step f, the newly created track can be added to a tracking list, and the tracking list includes the tracking status of each target in the multiple targets, such as trajectory, tracking results, etc.

[0138] Furthermore, after the tracking list is updated, step S101 may be restarted until all detection results are traversed.

[0139] From the above, by adopting this implementation scheme, the multi-target tracking algorithm based on monocular camera and 3D lidar is used in unmanned vehicles, which can accurately and stably track pedestrians or vehicles, and can solve the problems of trajectory interruption and identity information conversion caused by occlusion in the current multi-target tracking. At the same time, it predicts the motion trajectory of the occluded target, which provides a basis for the unmanned vehicle to adopt driving strategies such as following, overtaking, and emergency stopping, and provides a guarantee for the safe driving of the vehicle.

[0140] Specifically, this implementation scheme proposes a multi-target tracking technology based on a monocular camera and a 3D lidar, the purpose of which is to deal with the occlusion and detection frame loss problems in multi-target motion scenes, so as to improve the performance of the indicators (IDF1 and IDs) for identity information in the tracking algorithm. Based on the results of two-dimensional vision and three-dimensional point cloud detection, the algorithm of this implementation scheme can greatly improve the accuracy of target matching; by considering the relative motion relationship between targets, the targets are clustered, and by sharing features such as speed and acceleration, the trajectory of occluded targets in the cluster can be restored; for individuals that cannot form a cluster, the uniform speed model is used to predict their motion state during the occlusion period to achieve trajectory prediction, thereby reducing the occurrence of identity information matching failures.

[0141] Figure 5 Schematic diagram of the structure of a multi-target tracking device according to an embodiment of the present invention. A person skilled in the art will appreciate that the multi-target tracking device 5 according to this embodiment can be used to implement the above Figures 1 to 4 The method and technical solution described in the embodiments.

[0142] Specifically, refer to Figure 5 The multi-target tracking device 5 described in this embodiment may include: an acquisition module 51, used to obtain the tracking result of the first target among the multiple targets at the current moment during the tracking of multiple targets; a determination module 52, if the tracking result is an unmatched tracking, determines the cluster to which the first target belongs based on the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and similar motion attributes; a recovery module 53, used to restore the trajectory of the first target at the current moment based on the detection results of the targets in the cluster other than the first target at the current moment and the relative motion relationship between the targets in the cluster.

[0143] For more information about the working principle and working method of the multi-target tracking device 5, please refer to the above Figures 1 to 4 The relevant description in will not be repeated here.

[0144] Furthermore, an embodiment of the present invention also discloses a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor Figures 1 to 4 The method and technical solution described in the illustrated embodiment. Preferably, the storage medium may include a computer-readable storage medium such as a non-volatile memory or a non-transitory memory. The storage medium may include a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0145] Furthermore, an embodiment of the present invention also discloses a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 4 The method and technical solution described in the illustrated embodiment. Specifically, the terminal may be an on-board controller of the automobile.

[0146] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0147] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0148] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A multi-target tracking method, characterized in that: include: During tracking of multiple targets, obtaining a tracking result of a first target among the multiple targets at a current moment; If the tracking result is an unmatched tracking, determining the cluster to which the first target belongs according to the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and similar motion attributes; According to the detection results of the targets in the cluster other than the first target at the current moment and the relative motion relationship between the targets in the cluster, the trajectory of the first target at the current moment is restored by sharing the velocity and acceleration features; Determining the cluster to which the first target belongs according to historical detection results of the multiple targets includes: Determine the object attributes and motion attributes of each target based on the historical detection results of each target; Searching for a target from the multiple targets that has a motion attribute similar to that of the first target and an object attribute identical to that of the first target, and a distance from the first target that is less than a preset threshold; If the search is successful, it is determined that the searched target and the first target form the cluster.

2. The multi-target tracking method according to claim 1, characterized in that: The obtaining of the tracking result of the first target among the multiple targets at the current moment includes: Predicting a prediction result of the first target at the current moment according to a historical detection result of the first target; The prediction result is matched with the object detection result obtained at the current moment to obtain the tracking result of the first target.

3. The multi-target tracking method according to claim 2, characterized in that: For any of the historical detection results, prediction results and object detection results, the detection result is represented based on a detection frame, and the detection frame corresponds one-to-one to a dictionary, and the dictionary includes at least one of the following contents: identity information of the tracked object; the position, size, movement speed and size change speed of the detection frame; the tracking status of the trajectory; the number of frames the trajectory is maintained.

4. The multi-target tracking method according to claim 2, characterized in that: The performing target matching on the prediction result and the object detection result obtained by detection at the current moment to obtain the tracking result of the first target includes: When the tracking state of the trajectory in the historical detection result of the first target is a determined tracking state, the prediction result is cascade matched with the object detection result obtained by the current moment detection to obtain the tracking result of the first target.

5. The multi-target tracking method according to claim 4, characterized in that: The cascade matching refers to using a plurality of cost matrix data association algorithms to perform target matching, wherein the plurality of cost matrix data association algorithms include at least a first data association algorithm and a second data association algorithm; the cascade matching of the prediction result with the object detection result obtained at the current moment to obtain the tracking result includes: Using the first data association algorithm to perform target matching on the prediction result and the object detection result to obtain a first matching result; If the first matching result is not a matching pair, the second data association algorithm is used to perform target matching on the prediction result and the object detection result to obtain a second matching result, and the second matching result is determined as the tracking result.

6. The multi-target tracking method according to claim 2, characterized in that: The step of performing target matching on the prediction result and the object detection result obtained by detection at the current moment to obtain the tracking result of the first target includes: When the tracking state of the trajectory in the historical detection result of the first target is uncertain tracking, a single data association algorithm is used to match the prediction result with the object detection result obtained at the current moment to obtain the tracking result of the first target.

7. The multi-target tracking method according to claim 5 or 6, characterized in that: Data association algorithms for various cost matrices include: IoU matching and SIOA matching.

8. The multi-target tracking method according to claim 1, characterized in that: The object attributes include the identity category of the target; the motion attributes include: the motion speed of the target; the motion direction of the target; and the trajectory of the target.

9. The multi-target tracking method according to claim 1, characterized in that: Also includes: If the first target cannot form a cluster with other targets among the multiple targets, the trajectory of the first target at the current moment is predicted according to the uniformly accelerated motion model.

10. The multi-target tracking method according to claim 1 or 9, characterized in that: Also includes: If the first target reappears within the preset matching time limit and matches the recovered or predicted trajectory of the first target at the current moment, the tracking state of the trajectory in the historical detection results of the first target is determined as uncertain tracking, and tracking is performed at the next moment.

11. The multi-target tracking method according to claim 1, characterized in that: Also includes: If the tracking result of the first target is a matching pair, the prediction result of the first target at the current moment is updated by using an improved Kalman filter.

12. The multi-target tracking method according to claim 1, characterized in that: Before determining the cluster to which the first target belongs according to the historical detection results of the multiple targets, the method further includes: The historical detection results are smoothed by using an improved Kalman filter.

13. A multi-target tracking device, characterized in that: include: An acquisition module, used for acquiring, during the tracking of multiple targets, a tracking result of a first target among the multiple targets at a current moment; A determination module, if the tracking result is an unmatched tracking, determines the cluster to which the first target belongs according to the historical detection results of the multiple targets, wherein the multiple targets belonging to the same cluster have the same object attributes and motion attributes; A recovery module, configured to recover the trajectory of the first target at the current moment by sharing velocity and acceleration features according to the detection results of the targets in the cluster other than the first target at the current moment and the relative motion relationship between the targets in the cluster; The determining module performs the following steps: Determine the object attributes and motion attributes of each target based on the historical detection results of each target; Searching for a target from the multiple targets that has a motion attribute similar to that of the first target and an object attribute identical to that of the first target, and a distance from the first target that is less than a preset threshold; If the search is successful, it is determined that the searched target and the first target form the cluster.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are performed.

15. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 12 are performed.

Images