A multi-target tracking method and device in dense scenes based on density clustering

Abstract

The present application provides a method and device for tracking multiple targets in dense scenes based on density clustering, and relates to the technical field of multiple target tracking. The method includes: using the motion information of all moving targets in the previous image frame to obtain the prediction frames of all moving targets in the previous image frame in the current image frame; obtaining the detection frames of all moving targets in the current image frame output by the detector; classifying the density state of the detection frames of all moving targets; constructing a first association cost matrix between the detection frames of the sparse state category and all the prediction frames, and using a preset loose threshold to obtain the target sequence number of the detection frame that matches successfully and the prediction frame that does not match successfully; constructing a second association cost matrix between the detection frame of the dense state category and the prediction frame that does not match successfully, and using a preset strict threshold to obtain the target sequence number of the detection frame that matches successfully, wherein the loose threshold is greater than the strict threshold. The present application improves the matching accuracy of moving targets.

Description

Technical Field

[0001] The present application relates to the technical field of multi-target tracking, and in particular to a multi-target tracking method and device in a dense scene based on density clustering. Background Art

[0002] In the multi-target tracking method, the object similarity judgment function is usually used to first construct the cost matrix between the current frame detection frame and all active trajectories. Then, the matching algorithm such as the Hungarian algorithm is used to solve the matching relationship between trajectories based on the cost matrix. Finally, the algorithm performs trajectory management tasks such as trajectory registration, trajectory deletion, and trajectory update based on the detection frames and trajectories with various matching relationships. The object similarity judgment and cost matrix solution are collectively referred to as the data association process. The cost matrix solution algorithm is usually the Hungarian algorithm or the greedy algorithm. These solution algorithms usually require manual input of the association threshold, which is essentially to prevent the detection frame from matching the missed active trajectories on the one hand, and to avoid the situation where the object similarity judgment algorithm cannot measure the similarity between objects well on the other hand. The appearance of the association threshold ensures the stability of the solution.

[0003] The current mainstream multi-target tracking methods usually only use a fixed association threshold. However, considering the real scene, when the objects are densely spaced, a single association threshold is prone to mismatching between the detection box and the missed trajectory; and when the objects are sparse, missed associations are prone to occur. Summary of the invention

[0004] In view of this, the present application provides a multi-target tracking method and device in a dense scene based on density clustering to solve the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a multi-target tracking method in a dense scene based on density clustering, comprising:

[0006] Using the motion information of all moving objects in the previous image frame, the prediction boxes of all moving objects in the previous image frame in the current image frame are obtained;

[0007] Get the detection frames of all moving targets in the current image frame output by the detector;

[0008] Classify the density status of the detection frames of all moving targets;

[0009] Construct the first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and use the preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame;

[0010] A second correlation cost matrix is ​​constructed between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and a target sequence number of the successfully matched detection frame is obtained using a preset strict threshold, wherein the loose threshold is greater than the strict threshold.

[0011] Furthermore, when the detection frame is a three-dimensional detection frame, the detection frames of all moving targets are classified according to density states; including:

[0012] Step S1: Obtain the three-dimensional coordinates of the center point of each three-dimensional detection frame in the three-dimensional detection frame set;

[0013] Step S2: determine whether all the 3D detection frames in the 3D detection frame set are marked. If not, select an unmarked 3D detection frame p from the 3D detection frame set at random, mark the 3D detection frame p, and proceed to step S3; otherwise, proceed to step S7;

[0014] Step S3: Obtain the number n1 of 3D detection frames within the sphere with the 3D detection frame p as the center and the parameter ε as the radius, and determine whether n1≥Minpts1 is established, where Minpts1 is the first neighborhood density threshold. If yes, create a new cluster C1, add the 3D detection frame p to the new cluster C1, and proceed to step S4; otherwise, proceed to step S2;

[0015] Step S4: determine whether all the 3D detection frames in the detection frame subset within the ε neighborhood of the 3D detection frame p are marked. If yes, proceed to step S2; otherwise, proceed to step S5.

[0016] Step S5: randomly select a 3D detection frame p' from the detection frame subset within the ε neighborhood of the 3D detection frame p, mark the 3D detection frame p', and add the 3D detection frame p' to the cluster C1;

[0017] Step S6: Obtain the number n2 of 3D detection frames within the sphere with the 3D detection frame p' as the center and the parameter ε as the radius, and determine whether n2 ≥ Minpts1. If yes, add the 3D detection frames within the neighborhood of the 3D detection frame p' to cluster C1 and proceed to step S4; otherwise, proceed to step S4.

[0018] Step S7: define the category of the three-dimensional detection box in each cluster as a dense state, and define the three-dimensional detection box that does not belong to any cluster as a sparse state.

[0019] Furthermore, when the detection frame is a three-dimensional detection frame; constructing a first correlation cost matrix between the detection frame of the sparse state category and all the prediction frames, and using a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; including:

[0020] Using 3D Intersection-over-Union (GIoU)3d As a cost function, calculate the association value between the sparse 3D detection box and each 3D prediction box;

[0021] According to the association value between each sparse three-dimensional detection box and each three-dimensional prediction box, the first association cost matrix R is constructed. sparse ;

[0022] Determine the first association cost matrix R sparse Whether each element of is less than the first loose threshold θ1, if yes, the target serial number of the 3D detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the 3D trajectory frames that are not associated are obtained.

[0023] Furthermore, when the detection frame is a three-dimensional detection frame, a second correlation cost matrix is ​​constructed between the detection frame of the dense state category and the unmatched prediction frame, and a target sequence number of the successfully matched detection frame is obtained using a preset strict threshold, including:

[0024] Using 3D Intersection-over-Union (GIoU) 3d As a cost function, calculate the association value between the dense 3D detection box and each unmatched 3D prediction box;

[0025] According to the association value between each dense 3D detection frame and each unmatched 3D prediction frame, the second association cost matrix R is constructed. dense ;

[0026] Determine the second association cost matrix R dense Whether each element of is less than the first strict threshold σ1, if so, the target sequence number of the three-dimensional detection frame whose category is dense state and which is successfully matched is obtained.

[0027] Furthermore, the detection frame is a two-dimensional detection frame, and the detection frames of all moving targets are classified according to density states; including:

[0028] Step T1: Obtain the pixel plane center point of each two-dimensional detection frame in the two-dimensional detection frame set;

[0029] Step T2: determine whether all the two-dimensional detection frames in the two-dimensional detection frame set are marked. If not, select an unmarked two-dimensional detection frame q from the two-dimensional detection frame set at random, mark the two-dimensional detection frame q, and proceed to step T3; otherwise, proceed to step T7;

[0030] Step T3: Get the number m1 of detection frames within the circle with the two-dimensional detection frame q as the center and the parameter δ as the radius, and determine whether m1≥Minpts2 is true, where Minpts2 is the second neighborhood density threshold. If yes, create a new cluster C2, add the two-dimensional detection frame p to the new cluster C2, and go to step T4; otherwise, go to step T2;

[0031] Step T4: Determine whether all the two-dimensional detection frames in the detection frame subset within the δ neighborhood of the two-dimensional detection frame q have been marked. If yes, proceed to step T2; otherwise, proceed to step T5.

[0032] Step T5: randomly select a two-dimensional detection frame q' from the set of detection frames in the δ neighborhood of the two-dimensional detection frame q, mark the two-dimensional detection frame q', and add the two-dimensional detection frame q' to cluster C2;

[0033] Step T6: Get the number m2 of detection frames within the circle with the two-dimensional detection frame q' as the center and the parameter δ as the radius, and determine whether m2≥Minpts2. If yes, add the two-dimensional detection frames within the δ neighborhood of the two-dimensional detection frame q' to cluster C2 and go to step T4. Otherwise, go to step T4.

[0034] Step T7: define the category of the two-dimensional detection box in each cluster as a dense state, and define the detection box that does not belong to any cluster as a sparse state.

[0035] Furthermore, the detection frame is a two-dimensional detection frame, and the detection frames of all moving targets are classified according to density states; including:

[0036] Get the pixel plane center point of each two-dimensional detection box in the two-dimensional detection box set;

[0037] Using the OPTICS algorithm to process each two-dimensional detection frame in the two-dimensional detection frame set, multiple clusters are obtained;

[0038] The category of the two-dimensional detection box in each cluster is defined as a dense state, and the detection box that does not belong to any cluster is defined as a sparse state.

[0039] Furthermore, when the detection frame is a two-dimensional detection frame; constructing a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, using a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; including:

[0040] Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the sparse two-dimensional detection box and each two-dimensional prediction box;

[0041] According to the association value between each sparse two-dimensional detection box and each two-dimensional prediction box, the first association cost matrix R is constructed. sparse ;

[0042] Determine the first association cost matrix R sparse Whether each element of is less than the second loose threshold θ2, if so, the target serial number of the two-dimensional detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the unassociated two-dimensional trajectory frames are obtained.

[0043] Furthermore, when the detection frame is a two-dimensional detection frame; constructing a second correlation cost matrix between the detection frame of the dense state category and the unmatched prediction frame, and using a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, including:

[0044] Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the dense 2D detection box and each unmatched 2D prediction box;

[0045] According to the association value between each dense two-dimensional detection box and each unmatched two-dimensional prediction box, the second association cost matrix R is constructed. dense ;

[0046] Determine the second association cost matrix R dense Whether each element of is less than the second strict threshold σ2, if so, the target serial number of the two-dimensional detection frame whose category is dense state and whose matching is successful is obtained.

[0047] In a second aspect, an embodiment of the present application provides a multi-target tracking device in a dense scene based on density clustering, comprising:

[0048] A prediction unit, used to obtain a prediction frame of all moving objects in the previous image frame in the current image frame by using the motion information of all moving objects in the previous image frame;

[0049] An acquisition unit, used for acquiring the detection frames of all moving targets in the current image frame output by the detector;

[0050] The classification unit is used to classify the density state of the detection frames of all moving targets;

[0051] A first matching unit is used to construct a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame by using a preset loose threshold;

[0052] The second matching unit is used to construct a second correlation cost matrix between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and use a preset strict threshold to obtain a target sequence number of the successfully matched detection frame, wherein the loose threshold is greater than the strict threshold.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the method of the embodiment of the present application when executed by a processor.

[0055] The present application improves the matching accuracy of moving targets by classifying targets of different densities and then matching them. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 A flowchart of a multi-target tracking method in a dense scene based on density clustering provided in an embodiment of the present application;

[0058] Figure 2 A functional structure diagram of a multi-target tracking device in a dense scene based on density clustering provided in an embodiment of the present application;

[0059] Figure 3 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0062] First, the technical terms involved in the embodiments of the present application are briefly introduced.

[0063] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a representative density-based clustering algorithm. It defines a cluster as the largest set of density-connected points, can divide areas with sufficiently high density into clusters, and can find clusters of any shape in a noisy spatial database.

[0064] OPTICS (Ordering points to identify the clustering Structure) is an improved version of DBSCAN, so the OPTICS algorithm is also a density-based clustering algorithm. In the DBSCAN algorithm, two parameters need to be input: ∈ and MinPts. Choosing different parameters will lead to different final clustering results. Therefore, DBCSAN is too sensitive to input parameters. The OPTICS algorithm is proposed to help the DBTCAN algorithm select appropriate parameters and reduce the sensitivity of input parameters. OPTICS is mainly an improvement for the over-sensitivity of the input parameter ∈. The input parameters of OPTICS and DBSCNA are the same (∈ and MinPts). Although the OPTICS algorithm also requires two input parameters, the algorithm is insensitive to the ∈ input (∈ is generally fixed to infinity). At the same time, the algorithm does not explicitly generate data clusters, but only sorts the objects in the data set to obtain an ordered list of objects. Through the ordered list, a decision diagram can be obtained. The decision diagram can be used to detect clusters in data sets with different ∈ parameters, that is: first, an ordered list is obtained through a fixed MinPts and infinite ∈, and then a decision diagram is obtained. The decision diagram can be used to know the clustering of the data when ∈ takes a specific value (for example, ∈=3).

[0065] After introducing the technical terms involved in this application, the design concept of the embodiments of this application is briefly introduced next.

[0066] Multi-target tracking algorithms can track common objects in common scenarios, such as indoor or outdoor scenes and targets such as vehicles and pedestrians. Multi-target tracking can be divided into two-dimensional multi-target tracking and three-dimensional multi-target tracking. Among them, two-dimensional multi-target tracking usually uses the two-dimensional detection box information output by the detector, including the position information and size information of the detection box in the two-dimensional plane, and tracks its trajectory in the two-dimensional coordinate system. Three-dimensional multi-target tracking usually uses the three-dimensional detection box information output by the detector, including the size information and position information of the detection box in the three-dimensional space, and tracks its trajectory in the three-dimensional coordinate system.

[0067] Object similarity judgment and cost matrix solution are collectively referred to as the data association process. The cost matrix solution algorithm is usually the Hungarian algorithm or the greedy algorithm. These solution algorithms usually require manual input of the association threshold, which is essentially to prevent the detection box from matching the active trajectory of the missed detection on the one hand, and to avoid the situation where the object similarity judgment algorithm cannot measure the similarity between objects well on the other hand. The emergence of the association threshold ensures the stability of the solution. At present, only a fixed association threshold is usually used. However, considering the real scene, when the objects are dense, a more stringent association threshold should be used to avoid possible mismatches between the detection box and the missed detection trajectory, and when the objects are sparse, a looser association threshold should be used to leave a certain error space for the motion prediction algorithm to prevent missed associations. In addition, since there are many types of objects in the scene, such as buses, cars, bicycles, and pedestrians, the sizes of these objects are very different, so different standards should also be used to judge the density between objects.

[0068] For example, when it comes to pedestrian tracking, traditional pedestrian tracking algorithms are generally targeted at pedestrian tracking in dense scenes. When associating data, a relatively strict single threshold is generally used to avoid mismatching, thereby obtaining relatively good pedestrian tracking results in dense scenes. Although such methods are simple, convenient, and highly operational, in real scenes, the density of tracked pedestrians changes over time, and the density of different areas in the same frame may also vary greatly. The same threshold selection will lead to missed associations in dense states and mismatches in sparse states, resulting in weak algorithm generalization and difficulty in applying to real scenes.

[0069] In order to achieve the optimal match of data association and improve tracking accuracy, this application provides a multi-target tracking method in dense scenes based on density clustering. This method uses the DBSCAN algorithm or the OPTICS algorithm to judge and classify the sparsity of the tracking targets in the real-time scene, and uses different tracking thresholds for dense target clusters and sparse targets to achieve the optimal match, reduce the mismatch caused by different target densities, and improve the tracking performance in a targeted manner. This method has strong generalization and can be used for a variety of data sets and real-world situations.

[0070] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0071] like Figure 1 As shown, the present application implements a method for multi-target tracking in a dense scene based on density clustering, comprising the following steps:

[0072] Step 101: using the motion information of all moving objects in the previous image frame, obtaining the predicted frames of all moving objects in the previous image frame in the current image frame;

[0073] Among them, directly using the trajectory features of historical frames will lead to a large inter-frame displacement and a decrease in matching accuracy. Therefore, for the trajectories that are still active in the current frame, motion prediction is performed through trajectory prediction methods (such as filtering methods, time series neural networks, etc.) to predict the positions of these trajectories in the current frame. Suppose the predicted trajectory frame T t-1 The number is N_tra.

[0074] Step 102: Obtain detection frames of all moving objects in the current image frame output by the detector;

[0075] Among them, 2D multi-target tracking usually uses the 2D detection frame information output by the detector, including the position and size information of the detection frame in the 2D plane, and tracks its trajectory in the 2D coordinate system. 3D multi-target tracking usually uses the 3D detection frame information output by the detector, including the size and position information of the detection frame in the 3D space, and tracks its trajectory in the 3D coordinate system.

[0076] Step 103: classifying the density states of the detection frames of all moving targets;

[0077] There are three implementation methods for this step, which are described in detail below.

[0078] For the three-dimensional detection frame, the DBSCAN algorithm is used. The first implementation method is:

[0079] Since the density of each tracking target in the scene is different, the DBSCAN algorithm is used to judge the density of the target and cluster it. Assume that the number of 3D detection frames of the current frame sent by the 3D detector is N_det, and the coordinates of the center point of the detection frame are [x, y, z]. The distance between any two points reflects the density between the points, indicating whether the points can be clustered. The specific process is as follows:

[0080] Step S1: Obtain the three-dimensional coordinates of the center point of each three-dimensional detection frame in the three-dimensional detection frame set;

[0081] Step S2: determine whether all the 3D detection frames in the 3D detection frame set are marked. If not, select an unmarked 3D detection frame p from the 3D detection frame set at random, mark the 3D detection frame p, and proceed to step S3; otherwise, proceed to step S7;

[0082] Step S3: Obtain the number n1 of 3D detection frames within the sphere with the 3D detection frame p as the center and the parameter ε as the radius, and determine whether n1≥Minpts1 is established, where Minpts1 is the first neighborhood density threshold. If yes, create a new cluster C1, add the 3D detection frame p to the new cluster C1, and proceed to step S4; otherwise, proceed to step S2;

[0083] Step S4: determine whether all the 3D detection frames in the detection frame subset within the ε neighborhood of the 3D detection frame p have been marked. If yes, proceed to step S2; otherwise, proceed to step S5.

[0084] Step S5: randomly select a 3D detection frame p' from the detection frame subset within the ε neighborhood of the 3D detection frame p, mark the 3D detection frame p', and add the 3D detection frame p' to the cluster C1;

[0085] Step S6: Obtain the number n2 of 3D detection frames within the sphere with the 3D detection frame p' as the center and the parameter ε as the radius, and determine whether n2 ≥ Minpts1. If yes, add the 3D detection frames within the neighborhood of the 3D detection frame p' to cluster C1 and proceed to step S4; otherwise, proceed to step S4.

[0086] Step S7: define the category of the three-dimensional detection box in each cluster as a dense state, and define the three-dimensional detection box (sparse point) that does not belong to any cluster as a sparse state.

[0087] Therefore, the detection frame set is divided into a dense state detection frame set and a sparse state detection frame set, and:

[0088] N_det=N_det dense +N_det sparse

[0089] Where: N_det dense D is the dense state detection box dense Number, N_det sparse is the sparse state detection box D sparse number.

[0090] For the two-dimensional detection box, the DBSCAN algorithm is used, the second implementation method:

[0091] Step T1: Obtain the pixel plane center point of each two-dimensional detection frame in the two-dimensional detection frame set;

[0092] Step T2: determine whether all the two-dimensional detection frames in the two-dimensional detection frame set are marked. If not, select an unmarked two-dimensional detection frame q from the two-dimensional detection frame set at random, mark the two-dimensional detection frame q, and proceed to step T3; otherwise, proceed to step T7;

[0093] Step T3: Get the number m1 of detection frames within the circle with the two-dimensional detection frame q as the center and the parameter δ as the radius, and determine whether m1≥Minpts2 is true, where Minpts2 is the second neighborhood density threshold. If yes, create a new cluster C2, add the two-dimensional detection frame p to the new cluster C2, and go to step T4; otherwise, go to step T2;

[0094] Step T4: Determine whether all the two-dimensional detection frames in the detection frame subset within the δ neighborhood of the two-dimensional detection frame q have been marked. If yes, proceed to step T2; otherwise, proceed to step T5.

[0095] Step T5: randomly select a two-dimensional detection frame q' from the set of detection frames in the δ neighborhood of the two-dimensional detection frame q, mark the two-dimensional detection frame q', and add the two-dimensional detection frame q' to cluster C2;

[0096] Step T6: Get the number m2 of detection frames within the circle with the two-dimensional detection frame q' as the center and the parameter δ as the radius, and determine whether m2≥Minpts2. If yes, add the two-dimensional detection frames within the δ neighborhood of the two-dimensional detection frame q' to cluster C2 and go to step T4. Otherwise, go to step T4.

[0097] Step T7: define the category of the two-dimensional detection box in each cluster as a dense state, and define the detection box that does not belong to any cluster as a sparse state.

[0098] For the two-dimensional detection frame, the OPTICS algorithm is used, the third implementation method:

[0099] Get the pixel plane center point of each two-dimensional detection box in the two-dimensional detection box set;

[0100] Using the OPTICS algorithm to process each two-dimensional detection frame in the two-dimensional detection frame set, multiple clusters are obtained;

[0101] The category of the two-dimensional detection box in each cluster is defined as a dense state, and the detection box that does not belong to any cluster is defined as a sparse state.

[0102] Step 104: construct a first correlation cost matrix between the detection frame of the sparse state category and all prediction frames, and use a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame;

[0103] When the detection frame is a three-dimensional detection frame, this step includes:

[0104] Using 3D Intersection-over-Union (GIoU) 3d (Generalized InterTection over Union) is used as the cost function to calculate the association value between the sparse 3D detection box and each 3D prediction box;

[0105]

[0106] Among them A v is the three-dimensional trajectory box representing the trajectory in the current frame state after the trajectory is predicted, and B v The 3D detection box of the current frame detector, C v A v With B v The minimum three-dimensional closure of ;

[0107] According to the association value between each sparse three-dimensional detection box and each three-dimensional prediction box, the first association cost matrix R is constructed. sparse , where the first correlation cost matrix R sparse Each element in is 1-associated value;

[0108] Determine the first association cost matrix R sparse Whether each element of is less than the first loose threshold θ1, if yes, the target serial number of the 3D detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the 3D trajectory frames that are not associated are obtained.

[0109] Preferably, when the target is a pedestrian, the value range of the first loose threshold θ1 is: 0.9≤θ1≤1.2.

[0110] When the detection frame is a two-dimensional detection frame, this step includes:

[0111] Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the sparse two-dimensional detection box and each two-dimensional prediction box;

[0112] IOU 2d The calculation is described as follows:

[0113]

[0114] Among them, A and B are the two-dimensional detection box and the two-dimensional prediction box in the pixel plane, A∩B represents the intersection area between the detection box and the prediction box; A∪B represents the union area between the detection box and the prediction box.

[0115] According to the association value between each sparse two-dimensional detection box and each two-dimensional prediction box, the first association cost matrix R is constructed.sparse ; Among them, the first correlation cost matrix R sparse Each element in is 1-associated value;

[0116] Determine the first association cost matrix R sparse Whether each element of is less than the second loose threshold θ2, if so, the target serial number of the two-dimensional detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the unassociated two-dimensional trajectory frames are obtained.

[0117] Step 105: construct a second correlation cost matrix between the detection frame of the dense state category and the unmatched prediction frame, and use a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, wherein the loose threshold is greater than the strict threshold.

[0118] In this embodiment, when the detection frame is a three-dimensional detection frame, the step specifically includes:

[0119] Using 3D Intersection-over-Union (GIoU) 3d As a cost function, calculate the association value between the dense 3D detection box and each unmatched 3D prediction box;

[0120] According to the association value between each dense 3D detection frame and each unmatched 3D prediction frame, the second association cost matrix R is constructed. dense , where the second correlation cost matrix R dense Each element in is 1-associated value;

[0121] Determine the second association cost matrix R dense Whether each element of is less than the first strict threshold σ1, if so, the target sequence number of the three-dimensional detection frame whose category is dense state and which is successfully matched is obtained.

[0122] Preferably, when the target is a pedestrian, the value range of the first strict threshold σ1 is: 1.35≤σ1≤1.5.

[0123] In this embodiment, when the detection frame is a two-dimensional detection frame, the step includes:

[0124] Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the dense 2D detection box and each unmatched 2D prediction box;

[0125] According to the association value between each dense two-dimensional detection box and each unmatched two-dimensional prediction box, the second association cost matrix R is constructed. dense , where the second correlation cost matrix R dense Each element in is 1-associated value;

[0126] Determine the second association cost matrix R dense Whether each element of is less than the second strict threshold σ2, if so, the target serial number of the two-dimensional detection frame whose category is dense state and whose matching is successful is obtained.

[0127] Based on the above embodiments, the present application embodiment provides a multi-target tracking device in a dense scene based on density clustering, see Figure 2 As shown, the multi-target tracking device 200 in a dense scene based on density clustering provided in an embodiment of the present application at least includes:

[0128] The prediction unit 201 is used to obtain the prediction frames of all the moving objects in the previous image frame in the current image frame by using the motion information of all the moving objects in the previous image frame;

[0129] An acquisition unit 202 is used to acquire the detection frames of all moving objects in the current image frame output by the detector;

[0130] A classification unit 203 is used to classify the density states of the detection frames of all moving objects;

[0131] A first matching unit 204 is used to construct a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame by using a preset loose threshold;

[0132] The second matching unit 205 is used to construct a second correlation cost matrix between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and obtain the target sequence number of the successfully matched detection frame using a preset strict threshold, wherein the loose threshold is greater than the strict threshold.

[0133] It should be noted that the principle of solving the technical problem of the multi-target tracking device 200 in a dense scene based on density clustering provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the multi-target tracking device 200 in a dense scene based on density clustering provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0134] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 3 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the multi-target tracking method in a dense scene based on density clustering provided in the embodiment of the present application is implemented.

[0135] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0136] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (Random Access Memory, RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (Read Only Memory, ROM) 3023 .

[0137] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0138] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 3 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant ArrayT of lndependentDiTkT, RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0139] It should be noted that Figure 3 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0140] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the method provided by the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multi-target tracking method in a dense scene based on density clustering provided by the embodiment of the present application by executing the built-in or installed executable program.

[0141] The method provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the multi-target tracking method in a dense scene based on density clustering provided in the embodiment of the present application.

[0142] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0144] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0145] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multi-target tracking method in dense scenes based on density clustering, characterized in that: include: Using the motion information of all moving objects in the previous image frame, the prediction boxes of all moving objects in the previous image frame in the current image frame are obtained; Obtain the detection frames of all moving targets in the current image frame output by the detector; Classify the density status of the detection frames of all moving targets; Construct the first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and use the preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; Constructing a second correlation cost matrix between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and using a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, wherein the loose threshold is greater than the strict threshold; When the detection frame is a three-dimensional detection frame, the density state of the detection frames of all moving targets is classified; including: Step S1: Obtain the three-dimensional coordinates of the center point of each three-dimensional detection frame in the three-dimensional detection frame set; Step S2: determine whether all the 3D detection frames in the 3D detection frame set are marked. If not, select an unmarked 3D detection frame p from the 3D detection frame set at random, mark the 3D detection frame p, and proceed to step S3; otherwise, proceed to step S7; Step S3: Obtain the number n1 of 3D detection frames within the sphere with the 3D detection frame p as the center and the parameter ε as the radius, and determine whether n1≥Minpts1 is established, where Minpts1 is the first neighborhood density threshold. If yes, create a new cluster C1, add the 3D detection frame p to the new cluster C1, and proceed to step S4; otherwise, proceed to step S2; Step S4: determine whether all the 3D detection frames in the detection frame subset within the ε neighborhood of the 3D detection frame p are marked. If yes, proceed to step S2; otherwise, proceed to step S5. Step S5: randomly select a 3D detection frame p' from the detection frame subset within the ε neighborhood of the 3D detection frame p, mark the 3D detection frame p', and add the 3D detection frame p' to the cluster C1; Step S6: Obtain the number n2 of 3D detection frames within the sphere with the 3D detection frame p' as the center and the parameter ε as the radius, and determine whether n2 ≥ Minpts1. If yes, add the 3D detection frames within the neighborhood of the 3D detection frame p' to cluster C1 and proceed to step S4; otherwise, proceed to step S4. Step S7: define the category of the three-dimensional detection box in each cluster as a dense state, and define the three-dimensional detection box that does not belong to any cluster as a sparse state.

2. The method according to claim 1, characterized in that When the detection frame is a three-dimensional detection frame; constructing a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and using a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; including: Using 3D Intersection-over-Union (GIoU) 3d As a cost function, calculate the association value between the sparse 3D detection box and each 3D prediction box; According to the association value between each sparse three-dimensional detection box and each three-dimensional prediction box, the first association cost matrix R is constructed. sparse ; Determine the first association cost matrix R sparse Whether each element of is less than the first loose threshold θ1, if yes, the target serial number of the 3D detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the 3D trajectory frames that are not associated are obtained.

3. The method according to claim 2, characterized in that When the detection frame is a three-dimensional detection frame; construct a second correlation cost matrix between the detection frame of the dense state category and the unmatched prediction frame, and use a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, including: Using 3D Intersection-over-Union (GIoU) 3d As a cost function, calculate the association value between the dense 3D detection box and each unmatched 3D prediction box; According to the association value between each dense 3D detection frame and each unmatched 3D prediction frame, the second association cost matrix R is constructed. dense ; Determine the second association cost matrix R dense Whether each element of is less than the first strict threshold σ1, if so, the target sequence number of the three-dimensional detection frame whose category is dense state and which is successfully matched is obtained.

4. A multi-target tracking method in dense scenes based on density clustering, characterized in that: include: Using the motion information of all moving objects in the previous image frame, the prediction boxes of all moving objects in the previous image frame in the current image frame are obtained; Obtain the detection frames of all moving targets in the current image frame output by the detector; Classify the density status of the detection frames of all moving targets; Construct the first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and use the preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; Constructing a second correlation cost matrix between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and using a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, wherein the loose threshold is greater than the strict threshold; The detection frame is a two-dimensional detection frame, and the density state of the detection frames of all moving targets is classified; including: Step T1: Obtain the pixel plane center point of each two-dimensional detection frame in the two-dimensional detection frame set; Step T2: determine whether all the two-dimensional detection frames in the two-dimensional detection frame set are marked. If not, select an unmarked two-dimensional detection frame q from the two-dimensional detection frame set at random, mark the two-dimensional detection frame q, and proceed to step T3; otherwise, proceed to step T7; Step T3: Get the number of detection frames m1 within the circle with the two-dimensional detection frame q as the center and the parameter δ as the radius, and determine whether m1≥Minpts2 is true, where Minpts2 is the second neighborhood density threshold. If yes, create a new cluster C2 and add the two-dimensional detection frame p to the new cluster C2, and enter Step T4; otherwise, go to step T2; Step T4: Determine whether all the two-dimensional detection frames in the detection frame subset within the δ neighborhood of the two-dimensional detection frame q have been marked. If yes, proceed to step T2; otherwise, proceed to step T5. Step T5: randomly select a two-dimensional detection frame q' from the set of detection frames in the δ neighborhood of the two-dimensional detection frame q, mark the two-dimensional detection frame q', and add the two-dimensional detection frame q' to cluster C2; Step T6: Get the number m2 of detection frames within the circle with the two-dimensional detection frame q' as the center and the parameter δ as the radius, and determine whether m2≥Minpts2. If yes, add the two-dimensional detection frames within the δ neighborhood of the two-dimensional detection frame q' to cluster C2 and go to step T4. Otherwise, go to step T4. Step T7: define the category of the two-dimensional detection box in each cluster as a dense state, and define the detection box that does not belong to any cluster as a sparse state.

5. The method according to claim 4, characterized in that When the detection frame is a two-dimensional detection frame; constructing a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and using a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; including: Using 2D Intersection-Union (IOU) 2d As a cost function, calculate the association value between the sparse two-dimensional detection box and each two-dimensional prediction box; According to the association value between each sparse two-dimensional detection box and each two-dimensional prediction box, the first association cost matrix R is constructed. sparse ; Determine the first association cost matrix R sparse Whether each element of is less than the second loose threshold θ2, if so, the target serial number of the two-dimensional detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the unassociated two-dimensional trajectory frames are obtained.

6. The method according to claim 5, characterized in that When the detection frame is a two-dimensional detection frame; construct a second correlation cost matrix between the detection frame of the dense state category and the unmatched prediction frame, and use a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, including: Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the dense 2D detection box and each unmatched 2D prediction box; According to the association value between each dense two-dimensional detection box and each unmatched two-dimensional prediction box, the second association cost matrix R is constructed. dense ; Determine the second association cost matrix R dense Whether each element of is less than the second strict threshold σ2, if so, the target serial number of the two-dimensional detection frame of the dense state whose category is successfully matched is obtained.

7. A multi-target tracking method in dense scenes based on density clustering, characterized in that: include: Using the motion information of all moving objects in the previous image frame, the prediction boxes of all moving objects in the previous image frame in the current image frame are obtained; Obtain the detection frames of all moving targets in the current image frame output by the detector; Classify the density status of the detection frames of all moving targets; Construct the first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and use the preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; Constructing a second correlation cost matrix between the detection frame of the dense state category and the prediction frame that has not been successfully matched, and using a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, wherein the loose threshold is greater than the strict threshold; The detection frame is a two-dimensional detection frame, and the density state of the detection frames of all moving targets is classified; including: Get the pixel plane center point of each two-dimensional detection box in the two-dimensional detection box set; Using the OPTICS algorithm to process each two-dimensional detection frame in the two-dimensional detection frame set, multiple clusters are obtained; The category of the two-dimensional detection box in each cluster is defined as a dense state, and the detection box that does not belong to any cluster is defined as a sparse state.

8. The method according to claim 7, characterized in that When the detection frame is a two-dimensional detection frame; constructing a first correlation cost matrix between the detection frame of the sparse state and all the prediction frames, and using a preset loose threshold to obtain the target sequence number of the successfully matched detection frame and the unmatched prediction frame; including: Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the sparse two-dimensional detection box and each two-dimensional prediction box; According to the association value between each sparse two-dimensional detection box and each two-dimensional prediction box, the first association cost matrix R is constructed. sparse ; Determine the first association cost matrix R sparse Whether each element of is less than the second loose threshold θ2, if so, the target serial number of the two-dimensional detection frame of the successfully matched category in the sparse state is obtained, otherwise, all the unassociated two-dimensional trajectory frames are obtained.

9. The method according to claim 8, characterized in that When the detection frame is a two-dimensional detection frame; construct a second correlation cost matrix between the detection frame of the dense state category and the unmatched prediction frame, and use a preset strict threshold to obtain the target sequence number of the successfully matched detection frame, including: Using two-dimensional intersection-and-union (IOU) 2d As a cost function, calculate the association value between the dense 2D detection box and each unmatched 2D prediction box; According to the association value between each dense two-dimensional detection box and each unmatched two-dimensional prediction box, the second association cost matrix R is constructed. dense ; Determine the second association cost matrix R dense Whether each element of is less than the second strict threshold σ2, if so, the target serial number of the two-dimensional detection frame of the dense state whose category is successfully matched is obtained.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 3, any one of claims 4 to 6, or any one of claims 7 to 9 is implemented.

Images