Multi-target detection method and related device

Abstract

The invention discloses a multi-target detection method and a related device. The method comprises the following steps: acquiring a cost matrix of m rows and n columns; obtaining the track length of each track in the cost matrix, and determining the pixel coordinates of an image where a target corresponding to the track is located; for each track, determining an adjustment threshold value corresponding to the track based on the track length of the track, the pixel coordinates of the image where the target corresponding to the track is located and a threshold value calculation formula; constructing an expansion matrix based on the adjustment threshold value of each track and the cost matrix; and obtaining a matching result based on the expansion matrix and a bipartite graph matching algorithm. In this way, according to the method, adaptive adjustment thresholds are given to the tracks with different track lengths and different positions of the corresponding targets in the pixel coordinate system; and the adjustment thresholds are fused into a cost matrix to construct an expansion matrix, and the expansion matrix is used as an input matrix of a bipartite graph matching algorithm to obtain a matching result, so that the accuracy and the detection efficiency of the matching result are finally improved.

Description

Technical Field

[0001] The present application relates to the field of multi-target detection technology and tracking, and in particular to a multi-target detection method and related devices. Background Art

[0002] In the multi-target detection and tracking scenario, if the detection data of the same target in different frames are incorrectly associated with different trajectories, it will trigger the trajectory splitting problem, which will damage the accuracy and stability of multi-target tracking.

[0003] Traditional target detection schemes use deep learning networks to extract target appearance features, then convert the extracted target appearance features into similarity indicators and integrate them into a metric matrix, and input the optimized metric matrix into a bipartite graph matching algorithm to obtain the matching relationship between targets; that is, the traditional scheme uses appearance features to improve matching accuracy and alleviate the problem of trajectory splitting.

[0004] However, in scenarios where the appearance features are very similar, such as in traffic scenarios where vehicles of the same color and style frequently appear, it is difficult to effectively distinguish different targets in real time by appearance features, and noise may even be introduced, resulting in matching errors. Therefore, how to improve the accuracy and real-time performance of multi-target detection has become one of the technical issues that need to be urgently solved in the current field of target detection technology. Summary of the invention

[0005] Based on the above problems, the present application provides a multi-target detection method, which can effectively improve the matching quality between the detection box and the prediction box, alleviate the problem of trajectory splitting, and improve the robustness and accuracy of multi-target detection.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] The first aspect of the present application provides a multi-target detection method, comprising:

[0008] Get a cost matrix of m rows and n columns; m is the number of trajectories, and n is the number of detection boxes;

[0009] Obtaining the trajectory length of each trajectory in the cost matrix, and determining the pixel coordinates of the image where the target corresponding to the trajectory is located;

[0010] For each of the trajectories, based on the trajectory length of the trajectory, the pixel coordinates of the image where the target corresponding to the trajectory is located, and a threshold calculation formula, determining an adjustment threshold corresponding to the trajectory;

[0011] constructing an expansion matrix based on the adjustment threshold of each of the trajectories and the cost matrix;

[0012] Based on the extended matrix and the bipartite graph matching algorithm, a matching result is obtained.

[0013] In an alternative implementation, the threshold calculation formula includes a first formula and a second formula; for each of the trajectories, based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and the threshold calculation formula, determining the adjustment threshold corresponding to the trajectory includes:

[0014] Based on the trajectory length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image, determining whether the trajectory is a target trajectory;

[0015] If the trajectory is the target trajectory, determining the adjustment threshold of the trajectory based on the length of the trajectory and the first formula;

[0016] If the trajectory is not the target trajectory, using the preset target value in the second formula as the adjustment threshold of the trajectory.

[0017] In an alternative implementation, the expression of the first formula is specifically: E i = A + B × (C - D i ) / C; the E i is the adjustment threshold of the i-th trajectory, the A, B, and C are preset positive numbers; the D i is the trajectory length of the i-th trajectory; i is a positive integer greater than or equal to 1.

[0018] In an alternative implementation, the determining whether the trajectory is a target trajectory based on the trajectory length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image includes:

[0019] If the trajectory length of the trajectory is less than a preset trajectory length threshold, and the pixel coordinates of the target corresponding to the trajectory in the image indicate that the target is located within a preset edge area of the pixel coordinate system, determining that the trajectory is the target trajectory.

[0020] In an alternative implementation, the constructing an extended matrix based on the adjustment threshold of each of the trajectories and the cost matrix includes:

[0021] Constructing a zero matrix with (m + n) rows and (m + n) columns;

[0022] Assigning the cost matrix to the first m rows and the first n columns of the zero matrix to obtain a first matrix;

[0023] For each trajectory corresponding to each row in the first m rows of the first matrix, obtaining the adjustment threshold of the trajectory, and filling the obtained adjustment threshold of the trajectory into the columns from the (n + 1)-th column to the (m + n)-th column corresponding to the row to obtain a second matrix;

[0024] Determine the maximum adjustment threshold corresponding to all trajectories in the cost matrix, and fill the maximum adjustment threshold into the sub-matrix of the second matrix as the extended matrix; the sub-matrix is a matrix constructed from the (m + 1)-th row to the (m + n)-th row of the second matrix and the first n columns of the second matrix.

[0025] In an alternative implementation, the step of obtaining an m-by-n cost matrix includes:

[0026] Obtain the intersection over union (IoU) matrix of m rows and n columns; where m is the number of trajectories and n is the number of detection boxes;

[0027] Take the difference between the m-by-n identity matrix and the IoU matrix as the cost matrix.

[0028] In an alternative implementation, the bipartite graph matching algorithm is the Jonker-Volgenant algorithm.

[0029] A second aspect of the present application discloses a multi-object detection device, including:

[0030] A cost matrix acquisition module, configured to obtain an m-by-n cost matrix; where m is the number of trajectories and n is the number of detection boxes;

[0031] A trajectory data acquisition module, configured to obtain the trajectory length of each trajectory in the cost matrix and determine the pixel coordinates of the target corresponding to the trajectory in the image;

[0032] An adjustment threshold determination module, configured to, for each trajectory, determine the adjustment threshold corresponding to the trajectory based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and a threshold calculation formula;

[0033] An extended matrix construction module, configured to construct an extended matrix based on the adjustment threshold of each trajectory and the cost matrix;

[0034] A matching result acquisition module, configured to obtain a matching result based on the extended matrix and the bipartite graph matching algorithm.

[0035] A third aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any implementation manner of the first aspect are implemented.

[0036] A fourth aspect of the present application provides an electronic device, including:

[0037] A memory, on which a computer program is stored;

[0038] A processor for executing the computer program in the memory to implement the steps of the method introduced in any implementation manner of the first aspect.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] The multi-object detection method disclosed in the present application dynamically adjusts the threshold for different trajectories according to the trajectory length (reflecting the stability of the target movement) and the pixel coordinates (characterizing the distribution of the target in space). For trajectories with different lengths and corresponding targets at different positions in the pixel coordinate system, appropriate adjustment thresholds are assigned respectively. Subsequently, these adjustment thresholds are incorporated into the cost matrix to construct an extended matrix, and this extended matrix is used as the input matrix of the bipartite graph matching algorithm; finally, the accuracy and detection efficiency of the matching result are improved; the problems of low accuracy and poor real-time performance existing in the traditional solution when using appearance features to improve the matching accuracy are solved, and it is particularly suitable for complex dynamic scenarios with frequent traffic tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 It is a flowchart of a multi-object detection method provided by an embodiment of the present application;

[0043] Figure 2 It is a flowchart of determining the adjustment threshold of a trajectory provided by an embodiment of the present application;

[0044] Figure 3 It is a schematic structural diagram of a multi-object detection device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] In a complex multi-object detection and tracking scenario, the target detection system needs to continuously and accurately monitor and record the states of multiple targets. During this process, if the detection data generated by the same target in different frames is wrongly associated with different trajectories due to various reasons, it will inevitably trigger the trajectory splitting problem, seriously damaging the accuracy of multi-object tracking, causing the system to deviate in judging the actual movement path of the target, unable to truthfully reflect the real movement trajectory of the target, and greatly affecting the stability of the system.

[0046] In traditional multi-target detection schemes, deep learning networks are used to extract appearance feature vectors such as shape and color of targets in image data; the extracted appearance feature vectors are then converted into similarity indicators and integrated into a metric matrix; the optimized metric matrix is ​​then input into a bipartite graph matching algorithm to obtain the target matching relationship; that is, traditional schemes use the appearance features of targets in images to improve matching accuracy, support solutions to trajectory splitting problems, reduce the risk of misassociation of detection data for the same target, ensure that target trajectories are complete and accurate, and maintain stable operation of the multi-target tracking system.

[0047] In the field of multi-target detection, the accuracy and real-time performance of matching results are crucial. Accurate detection results can clearly define key information such as the category and location of each target, laying a solid foundation for subsequent analysis and decision-making; and real-time performance ensures that the system responds to dynamic changes in targets in a timely manner, quickly outputs detection results in scenarios such as smart security and autonomous driving, and ensures efficient operation of the system.

[0048] Due to the shortcomings of insufficient appearance feature differentiation and large amount of calculation in traditional solutions, it is difficult to meet the requirements of accuracy and real-time matching results in the field of multi-target detection. For example, in scenes with large similarity of appearance features, such as transportation scenes where vehicles of the same color and style frequently appear, it is difficult to effectively distinguish different targets in real time through appearance features, and noise may even be introduced, resulting in matching errors. For another example, in scenes with high frame rates or a large number of targets, the calculation process of extracting target appearance features is very complicated, which will significantly increase the computational burden of the target detection algorithm and affect real-time performance.

[0049] Therefore, how to improve the accuracy and real-time performance of multi-target detection has become one of the technical problems that need to be urgently solved in the current field of target detection technology.

[0050] Based on the above problems, the present application discloses a multi-target detection method, including: obtaining a cost matrix of m rows and n columns; obtaining the trajectory length of each trajectory in the cost matrix, and determining the pixel coordinates of the image where the target corresponding to the trajectory is located; for each trajectory, based on the trajectory length of the trajectory, the pixel coordinates of the image where the target corresponding to the trajectory is located and the threshold calculation formula, determining the adjustment threshold corresponding to the trajectory; constructing an extension matrix based on the adjustment threshold of each trajectory and the cost matrix; obtaining the matching result based on the extension matrix and the bipartite graph matching algorithm. In this way, the present application assigns adaptive adjustment thresholds to trajectories with different trajectory lengths and corresponding targets at different positions in the pixel coordinate system; and integrates these adjustment thresholds into the extension matrix method to improve the accuracy of the matching results and the detection efficiency; solves the problems of low precision and poor real-time performance in the traditional solution when using appearance features to improve the matching accuracy, and is particularly suitable for complex dynamic scenes with frequent automatic driving and vehicles.

[0051] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of this application without creative efforts belong to the scope of protection of this application.

[0052] Figure 1 It is a flowchart of a multi-object detection method provided by an embodiment of this application. In combination with Figure 1 As shown, the multi-object detection method disclosed in this application includes:

[0053] S101, obtain a cost matrix of m rows and n columns.

[0054] It should be noted that m in this application is the number of trajectories, and n is the number of detection frames.

[0055] In an alternative implementation, the step of obtaining a cost matrix of m rows and n columns is:

[0056] First, obtain an intersection over union (IoU) matrix of m rows and n columns.

[0057] Intersection over Union (IoU) is a commonly used evaluation metric in computer vision tasks such as object detection and image segmentation, and is used to measure the overlap degree between the predicted result and the true result.

[0058] In the field of multi-object detection, the intersection area of the predicted box and the detection box can be calculated first; then the union area of the predicted box and the detection box can be calculated; and then the ratio of the intersection area to the union area is used as the IoU.

[0059] The value range of IoU is between 0 and 1. The closer the value of IoU is to 1, the higher the overlap degree between the predicted box and the detection box, and the better the prediction effect; the closer the value of IoU is to 0, the lower the overlap degree, and the worse the prediction effect.

[0060] It can be understood that in a multi-object detection scenario, there will be multiple predicted boxes and multiple detection boxes at the same time; the IoU matrix is a matrix formed by arranging the IoU values obtained by calculating the IoU of each predicted box with each detection box in a certain order.

[0061] Exemplarily, there are m1 predicted boxes and n1 detection boxes, and the IoU matrix is an m1×n1 matrix. Among them, the element in the i-th row and j-th column of the IoU matrix represents the IoU of the i-th predicted box and the j-th detection box.

[0062] Then, the difference between the m×n identity matrix and the intersection over union matrix is used as the cost matrix.

[0063] It can be understood that the larger the value of an element in the cost matrix, the greater the difference between the corresponding predicted bounding box and the detected bounding box, and the higher the associated cost; the smaller the value of the element, the smaller the difference between the two, and the lower the associated cost.

[0064] Through the cost matrix, it is convenient to find the optimal matching combination in subsequent algorithms to minimize the overall association cost, thereby achieving accurate object detection and data association.

[0065] S102, obtain the trajectory length of each trajectory in the cost matrix, and determine the pixel coordinates of the target corresponding to the trajectory in the image where the target is located.

[0066] In the field of multi-object detection, the trajectory length of each trajectory refers to the length of the path passed by the target in consecutive frame images. The trajectory length can intuitively reflect the motion stability of the target in the scene. If the trajectory length is short, it means that the target has appeared in the scene for a short time and has not moved a large range; if the trajectory length is long, it means that the target has persisted in the scene for a long time and has accumulated a long motion path.

[0067] In an alternative implementation, the trajectory length of each trajectory in the cost trajectory can be determined based on frame count. Specifically, during the target tracking process, the frame count of the trajectory is incremented by one for each frame passed, so as to record the trajectory length. Exemplarily, from the frame when the target is first detected and tracking starts to the current frame, the number of frames passed is the trajectory length.

[0068] In another alternative implementation, the trajectory length of each trajectory in the cost trajectory can be determined based on the time step. Specifically, the trajectory length can be determined by recording the number of time steps passed from the start of the trajectory to the current moment.

[0069] In an alternative implementation, the pixel coordinates of the target corresponding to the trajectory in the image where the target is located can be determined based on the bounding box.

[0070] For example, a bounding box can be generated for the image where the target corresponding to the trajectory is located, and the pixel coordinates of the centroid of the bounding box in the pixel coordinate system are used as the pixel coordinates of the target corresponding to the trajectory in the image where the target is located.

[0071] For another example, a bounding box can be generated for the image where the target corresponding to the trajectory is located, and the pixel coordinates of a certain corner point of the bounding box, such as the pixel coordinates of the lower right corner point, are used as the pixel coordinates of the target corresponding to the trajectory in the image where the target is located.

[0072] Among them, the image where the target corresponding to the trajectory is located can be the image included in the detection box of the latest frame associated with the trajectory, or the image included in the key detection box corresponding to the trajectory.

[0073] It should be noted that in this application, there is no limitation on the specific implementation method of determining the trajectory length of each trajectory in the cost matrix and determining the pixel coordinates of the image where the target corresponding to the trajectory is located. Those skilled in the art can select a suitable solution according to actual needs to determine the trajectory length of each trajectory in the cost matrix and determine the pixel coordinates of the image where the target corresponding to the trajectory is located.

[0074] S103. For each of the trajectories, based on the trajectory length of the trajectory, the pixel coordinates of the image where the target corresponding to the trajectory is located, and the threshold calculation formula, determine the adjustment threshold corresponding to the trajectory.

[0075] In multi-object tracking algorithms, for vehicles such as motorcycles and electric vehicles in a uniform motion process, the Kalman filter is usually used to predict their motion trajectories. However, when such vehicles appear from the lower edge of the image, even if they are approximately in uniform motion in actual motion, in the pixel coordinate system, the motion of the detection box does not present a state of uniform linear motion. This difference will cause the prediction result of the Kalman filter to deviate, thereby affecting the accurate tracking of the target.

[0076] In different stages of the trajectory association of multi-object tracking algorithms, the prediction accuracy of the Kalman filter will also be different. For example, when the target first appears, the prediction of the Kalman filter may not be accurate enough, while after the target's motion stabilizes, the prediction accuracy will be significantly improved.

[0077] In this application, based on the trajectory length and the pixel coordinates of the image where the target corresponding to the trajectory is located, it is determined whether the target corresponding to the trajectory is a newly appeared target and whether it is a target with an unstable motion state. After determining a newly appeared and unstable target, the relevant parameters in the input matrix of the bipartite graph matching algorithm are adjusted to make up for the prediction defect of the Kalman filter in this scenario, reduce the probability of incorrect matching caused by inaccurate prediction, and thus improve the overall performance and accuracy of the multi-object tracking algorithm.

[0078] The threshold calculation formula in this application includes a first formula and a second formula. The specific expression of the threshold calculation formula is:

[0079]

[0080] Among them, E in the threshold calculation formula i is the adjustment threshold of the i-th trajectory, A, B, and C are preset positive numbers; D i is the trajectory length of the i-th trajectory; i is a positive integer greater than or equal to 1.

[0081] Exemplarily, in this application, the value of A is preset to 0.5; the value of B is preset to 0.2; the value of C is preset to 5. In this way, the expression of the threshold calculation formula is converted to:

[0082]

[0083] After determining the threshold calculation formula, for each trajectory, based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and the threshold calculation formula, the adjustment threshold corresponding to the trajectory can be determined.

[0084] Figure 2 It is a flowchart for determining the adjustment threshold of a trajectory provided by an embodiment of this application. Combining Figure 2 as shown, the process of determining the adjustment threshold of a trajectory includes:

[0085] S201, based on the trajectory length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image, determine whether the trajectory is a target trajectory.

[0086] Specifically, if the trajectory length of the trajectory is less than the preset trajectory length threshold, and the pixel coordinates of the target corresponding to the trajectory indicate that the target is located within the preset edge area of the pixel coordinate system, then it is determined that the trajectory is a target trajectory.

[0087] In this application, A in the threshold calculation formula is used as the preset trajectory length threshold; that is, if A is 5, the preset trajectory length threshold is 5.

[0088] After this application obtains the pixel coordinates of the target corresponding to the trajectory in the image, it can calculate the vertical distance between the pixel coordinates and the horizontal axis of the pixel coordinate system; if the calculated vertical distance is less than the preset distance threshold, it is considered that the target corresponding to the trajectory is located within the preset edge area of the pixel coordinate system.

[0089] Exemplarily, take the key detection box of trajectory 1 as the image where the target corresponding to trajectory 1 is located; take the pixel coordinates of the lower right corner of the key detection box as the pixel coordinates of the target in the image in the pixel coordinate system; calculate the vertical distance between the pixel coordinates of the lower right corner and the horizontal axis in the pixel coordinate system; if the calculated vertical distance is 50 pixels, which is less than the preset distance threshold of 100 pixels, it is considered that the target corresponding to trajectory 1 is at the edge position in the pixel coordinate system, and it is considered that the picture corresponding to the trajectory is within the preset edge area of the pixel coordinate system.

[0090] It should be noted that in this application, the minimum value of the preset distance threshold is not limited, and those skilled in the art can set the distance threshold by themselves.

[0091] S202, if the trajectory is the target trajectory, determine the adjustment threshold for the trajectory based on the length of the trajectory and the first formula.

[0092] Specifically, if it is determined that trajectory 1 is the target trajectory, substitute the trajectory length of trajectory 1 into the first formula in the threshold calculation formula to obtain the adjustment threshold for trajectory 1.

[0093] S203, if the trajectory is not the target trajectory, use the preset target value in the second formula as the adjustment threshold for the trajectory.

[0094] Specifically, if it is determined that trajectory 1 is not the target trajectory, directly use the preset adjustment threshold in formula 2 as the adjustment threshold for trajectory 1.

[0095] As can be seen from the foregoing, A in the threshold calculation formula is a preset trajectory length threshold; A, B, and C are preset positive numbers; therefore, the calculation result of the first formula in the threshold calculation formula is greater than the calculation result of the second formula. That is, if the trajectory is the target trajectory, a larger adjustment threshold value can be configured for the trajectory through formula one in the threshold calculation formula, that is, a looser adjustment threshold is configured for the trajectory to avoid matching failures caused by inaccurate prediction; if the trajectory is not the target trajectory, a smaller adjustment threshold value can be configured for the trajectory through formula two in the threshold calculation formula, that is, a stricter adjustment threshold is configured for the trajectory, thereby reducing the occurrence of unmatched events. In this way, by assigning corresponding adjustment thresholds to different trajectories, the matching accuracy is ultimately improved and the multi-target tracking effect is optimized.

[0096] S104, construct an extended matrix based on the adjustment threshold of each trajectory and the cost matrix.

[0097] After obtaining an m-row and n-column cost matrix and the adjustment threshold of each trajectory in the cost matrix, an extended matrix can be constructed. This process includes:

[0098] The first step is to construct a zero matrix with (m + n) rows and (m + n) columns.

[0099] Exemplarily, m in this application is 3 and n is 4.

[0100] In this step, a 7-row and 7-column zero matrix needs to be constructed.

[0101] The second step is to assign the cost matrix to the first m rows and the first n columns of the zero matrix to obtain the first matrix.

[0102] Exemplarily, the expression of the cost matrix in this application is:

[0103]

[0104] Assign the 3-row and 4-column cost matrix to the first 3 rows and the first 4 columns of a 7-row and 7-column zero matrix to obtain the first matrix. The expression of the first matrix is as follows:

[0105]

[0106] In the third step, for each trajectory corresponding to each row in the first m rows of the first matrix, obtain the adjustment threshold of the trajectory, and fill the obtained adjustment threshold of the trajectory into the (n + 1)-th column to the (m + n)-th column corresponding to that row as the second matrix.

[0107] Obtain the first m rows in the first matrix, that is, obtain the adjustment thresholds of the trajectories corresponding to each row in the first 3 rows.

[0108] Exemplarily, the adjustment threshold of the trajectory corresponding to the first row in the first matrix is 0.7; the adjustment threshold of the trajectory corresponding to the second row in the first matrix is 0.6; the adjustment threshold corresponding to the third row in the first matrix is 0.5. Fill the obtained adjustment thresholds of the trajectories corresponding to each row in the first 3 rows of the first matrix into the (n + 1)-th column to the (m + n)-th column corresponding to that row, that is, fill into the 5th column to the 7th column corresponding to that row, to obtain the second matrix. The expression of the second matrix is as follows:

[0109]

[0110] In the fourth step, determine the maximum adjustment threshold corresponding to all the trajectories in the cost matrix, and fill the maximum adjustment threshold into the sub-matrix of the second matrix as the extended matrix.

[0111] Wherein, the sub-matrix is a matrix constructed by the (m + 1)-th row to the (m + n)-th row in the second matrix and the first n columns in the second matrix.

[0112] Exemplarily, the maximum adjustment threshold corresponding to all the trajectories in the 3-row and 4-column cost matrix is 0.7; fill the adjustment threshold 0.7 into the sub-matrix of the second matrix to obtain the extended matrix.

[0113] Wherein, the sub-matrix of the second matrix is a matrix constructed by the (m + 1)-th row to the (m + n)-th row in the second matrix and the first n columns in the second matrix. In this example, the sub-matrix of the second matrix is a matrix constructed by the 4th row to the 7th row in the second matrix and the first 4 columns.

[0114] Exemplarily, the expression of the extended matrix in this embodiment is as follows:

[0115]

[0116] S105, based on the extended matrix and the bipartite graph matching algorithm, obtain the matching result.

[0117] The bipartite graph matching algorithm in this application can be the Jonker-Volgenant algorithm, the Hungarian algorithm, or the Kuhn-Munkres algorithm.

[0118] To ensure the matching effect, the Jonker-Volgenant algorithm is adopted as the bipartite graph matching algorithm in this application. After obtaining the extended matrix, the extended matrix is used as the input matrix of the Jonker-Volgenant algorithm. The Jonker-Volgenant algorithm continuously searches for augmenting paths based on the extended matrix and finally obtains the matching result.

[0119] It can be understood that in the field of multi-object detection and tracking technology, the obtained matching result indicates the correspondence between the detected object and the target trajectory in the tracker.

[0120] For example, (d1, t1) indicates that the detection result d1 matches the trajectory t1, and (d2, t2) indicates that the detection result d2 matches the trajectory t2.

[0121] In summary, the core of the multi-object detection method disclosed in this application is to dynamically adjust the adjustment threshold corresponding to each trajectory in the matching process according to the motion state of the trajectory, that is, the trajectory length and the pixel coordinates of the target corresponding to the trajectory in the image; then incorporate these adjustment thresholds into the cost matrix to construct an extended matrix, and use this extended matrix as the input matrix of the bipartite graph matching algorithm; finally, improve the accuracy and detection efficiency of the matching result; solve the problems of low accuracy and poor real-time performance in the traditional solution when using appearance features to improve the matching accuracy, and is particularly suitable for complex dynamic scenarios with frequent occurrences of autonomous driving and transportation vehicles.

[0122] Based on the multi-object detection method disclosed in the foregoing embodiments, this application also discloses a multi-object detection device. Figure 3 It is a schematic structural diagram of the multi-object detection device provided in the embodiments of this application. Combining Figure 3 As shown, the multi-object detection device 300 disclosed in this application includes:

[0123] A cost matrix acquisition module 301, configured to acquire a cost matrix with m rows and n columns; where m is the number of trajectories and n is the number of detection frames;

[0124] A trajectory data acquisition module 302, configured to acquire the trajectory length of each trajectory in the cost matrix and determine the pixel coordinates of the target corresponding to the trajectory in the image;

[0125] An adjustment threshold determination module 303, configured to determine the adjustment threshold corresponding to each trajectory based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and a threshold calculation formula;

[0126] An extended matrix construction module 304, configured to construct an extended matrix based on the adjustment threshold of each of the trajectories and the cost matrix;

[0127] A matching result acquisition module 305, configured to obtain a matching result based on the extended matrix and the bipartite graph matching algorithm.

[0128] In an optional implementation manner, the adjustment threshold determination module 303 includes:

[0129] A target trajectory determination unit, configured to determine whether a trajectory is a target trajectory based on the length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image where the target is located;

[0130] A first adjustment threshold determination unit, configured to, if the trajectory is the target trajectory, determine the adjustment threshold of the trajectory based on the length of the trajectory and the first formula;

[0131] A second adjustment threshold determination unit, configured to, if the trajectory is not the target trajectory, use the preset target value in the second formula as the adjustment threshold of the trajectory.

[0132] In an optional implementation manner, the target trajectory determination unit includes:

[0133] A target trajectory determination subunit, configured to, if the length of the trajectory is less than a preset trajectory length threshold and the pixel coordinates of the target corresponding to the trajectory indicate that the target is located within a preset edge area of the pixel coordinate system, determine that the trajectory is the target trajectory.

[0134] In an optional implementation manner, the extended matrix construction module 304 includes:

[0135] A zero matrix determination unit, configured to construct a zero matrix with (m + n) rows and (m + n) columns;

[0136] A first matrix determination unit, configured to assign the cost matrix to the first m rows and the first n columns of the zero matrix to obtain a first matrix;

[0137] A second matrix determination unit, configured to, for each trajectory corresponding to each row in the first m rows of the first matrix, obtain the adjustment threshold of the trajectory, and fill the obtained adjustment threshold of the trajectory into the (n + 1)-th column to the (m + n)-th column corresponding to the row as the second matrix;

[0138] An expansion matrix determination unit for determining the maximum adjustment threshold corresponding to all trajectories in the cost matrix and filling the maximum adjustment threshold into a sub-matrix of the second matrix as the expansion matrix; the sub-matrix is a matrix constructed from the (m + 1)-th row to the (m + n)-th row of the second matrix and the first n columns of the second matrix.

[0139] In an alternative implementation, the cost matrix acquisition module 301 includes:

[0140] An intersection over union (IoU) matrix determination unit for obtaining the m-by-n IoU matrix; where m is the number of trajectories and n is the number of detection boxes;

[0141] A cost matrix determination unit for taking the difference between the m-by-n identity matrix and the IoU matrix as the cost matrix.

[0142] Based on the multi-object detection method and device provided in the foregoing embodiments, correspondingly, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, it implements some or all of the steps in the multi-object detection method mentioned above.

[0143] Based on the multi-object detection method and device provided in the foregoing embodiments, the present application also provides an electronic device, including:

[0144] A memory having a computer program stored thereon;

[0145] A processor for executing the computer program in the memory to implement some or all of the steps in the multi-object detection method provided in the foregoing embodiments.

[0146] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0147] As described above, it is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-object detection method, characterized in that, The method includes: Obtain a cost matrix of m rows and n columns; where m is the number of trajectories and n is the number of detection boxes; Obtain the trajectory length of each trajectory in the cost matrix, and determine the pixel coordinates of the target corresponding to the trajectory in the image; For each trajectory, based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and a threshold calculation formula, determine the adjustment threshold corresponding to the trajectory; Based on the adjustment threshold of each trajectory and the cost matrix, construct an extended matrix; Based on the extended matrix and the bipartite graph matching algorithm, obtain a matching result.

2. The method according to claim 1, characterized in that, The threshold calculation formula includes a first formula and a second formula; for each trajectory, based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and the threshold calculation formula, determining the adjustment threshold corresponding to the trajectory includes: Based on the trajectory length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image, determine whether the trajectory is a target trajectory; If the trajectory is the target trajectory, then based on the length of the trajectory and the first formula, determine the adjustment threshold of the trajectory; If the trajectory is not the target trajectory, then use the preset target value in the second formula as the adjustment threshold of the trajectory.

3. The method according to claim 2, characterized in that, The expression of the first formula is specifically: E i = A + B × (C - D i ) / C; the E i is the adjustment threshold of the i-th trajectory, and the A, B, and C are preset positive numbers; the D i is the trajectory length of the i-th trajectory; i is a positive integer greater than or equal to 1.

4. The method according to claim 2, characterized in that, The determining whether the trajectory is a target trajectory based on the trajectory length of the trajectory and the pixel coordinates of the target corresponding to the trajectory in the image includes: If the trajectory length of the trajectory is less than a preset trajectory length threshold, and the pixel coordinates of the target corresponding to the trajectory in the image indicate that the target is located within a preset edge area of the pixel coordinate system, then determine that the trajectory is the target trajectory.

5. The method according to claim 1, characterized in that, The constructing an extended matrix based on the adjustment threshold of each trajectory and the cost matrix includes: Construct a zero matrix of (m + n) rows and (m + n) columns; Assign the cost matrix to the first m rows and the first n columns of the zero matrix to obtain a first matrix; For each row corresponding to a trajectory in the first m rows of the first matrix, obtain the adjustment threshold of the trajectory, and fill the obtained adjustment threshold of the trajectory into the columns from the (n + 1)-th column to the (m + n)-th column corresponding to the row to obtain a second matrix; Determine the maximum adjustment threshold corresponding to all trajectories in the cost matrix, and fill the maximum adjustment threshold into a sub-matrix of the second matrix as the extended matrix; the sub-matrix is a matrix constructed by the (m + 1)-th row to the (m + n)-th row of the second matrix and the first n columns of the second matrix.

6. The method according to claim 1, wherein The step of obtaining a cost matrix of m rows and n columns includes: Obtain the intersection over union matrix of m rows and n columns; where m is the number of trajectories and n is the number of detection boxes; Take the difference between the m - row and n - column identity matrix and the intersection over union matrix as the cost matrix.

7. The method according to claim 1, characterized in that, The bipartite graph matching algorithm is the Jonker - Volgenant algorithm.

8. A multi-object detection device, characterized in that, The device includes: A cost matrix acquisition module for obtaining a cost matrix of m rows and n columns; where m is the number of trajectories and n is the number of detection boxes; A trajectory data acquisition module, configured to acquire the trajectory length of each trajectory in the cost matrix and determine the pixel coordinates of the target corresponding to the trajectory in the image; An adjustment threshold determination module, configured to, for each of the trajectories, determine an adjustment threshold corresponding to the trajectory based on the trajectory length of the trajectory, the pixel coordinates of the target corresponding to the trajectory in the image, and a threshold calculation formula; An extended matrix construction module, configured to construct an extended matrix based on the adjustment threshold of each of the trajectories and the cost matrix; A matching result acquisition module, configured to obtain a matching result based on the extended matrix and a bipartite graph matching algorithm.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, Comprising: A memory, on which a computer program is stored; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images