A target detection method, device and electronic equipment

Abstract

The present disclosure provides a target detection method and device and electronic equipment, relating to the field of artificial intelligence, in particular to automatic driving, computer vision and deep learning technology. The specific scheme is: performing target detection on a to-be-detected frame by a first target detection algorithm to obtain a detection target set; determining a first cost matrix based on the detection target set and an actual labeled target set of the to-be-detected frame, any element in the first cost matrix being an intersection-over-union between a detection target in the detection target set and a labeled target in the actual labeled target set; and performing problem target extraction and classification on the detection target set and the actual labeled target set according to the first cost matrix to obtain at least one problem target set of the to-be-detected frame, the problem categories of the at least one problem target set being different, and the at least one problem target set being used for updating the first target detection algorithm, so as to improve the robustness of the target detection algorithm.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to automatic driving, computer vision and deep learning technology, and more particularly to a target detection method and device and electronic equipment. BACKGROUND

[0002] Target detection is a common means in computer vision tasks, and obstacles and other targets are detected, which can be used in the fields of intelligent transportation and intelligent driving. The performance of a target detection algorithm directly affects the effect of target detection, and therefore it is necessary to continuously update the target detection algorithm.

[0003] At present, the target detection algorithm is updated by evaluating the target detection results of the target detection algorithm. In the process of evaluating the detection results of the target detection algorithm, a common method is to calculate common indicators such as accuracy and recall rate based on the detection results. SUMMARY

[0004] The present disclosure provides a target detection method, device and electronic equipment.

[0005] In a first aspect, an embodiment of the present disclosure provides a target detection method, comprising:

[0006] performing target detection on a to-be-detected frame by a first target detection algorithm to obtain a detection target set;

[0007] determining a first cost matrix based on the detection target set and a labeled target set of the to-be-detected frame, wherein any element in the first cost matrix is an intersection-over-union between a detection target in the detection target set and a labeled target in the labeled target set;

[0008] performing problem target extraction and classification on the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the to-be-detected frame, wherein one problem target set corresponds to one problem category, and the at least one problem target set is used to update the first target detection algorithm.

[0009] In the target detection method of the embodiments of the present disclosure, after the first cost matrix is determined based on the detection target set and the labeled target set of the frame to be detected, the detection target set and the labeled target set can be subjected to problem target extraction and classification according to the first cost matrix, so as to obtain at least one problem target set of the frame to be detected, one problem target set corresponding to one problem category, that is, the problem category of the problem target can be subdivided, and the problem targets of the same problem category are aggregated in the same problem target set, and the obtained at least one problem target set can be used to update the first target detection algorithm, so as to improve the robustness of the first target detection algorithm.

[0010] In a second aspect, an embodiment of the present disclosure provides a target detection device, and the device comprises:

[0011] a target detection module, configured to detect targets in a frame to be detected by using a first target detection algorithm to obtain a detection target set;

[0012] a first determination module, configured to determine a first cost matrix based on the detection target set and a labeled target set of the frame to be detected, any element in the first cost matrix being an intersection-over-union ratio between a detection target in the detection target set and a labeled target in the labeled target set;

[0013] a target set determination module, configured to extract and classify problem targets from the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the frame to be detected, one problem target set corresponding to one problem category, and the at least one problem target set being used to update the first target detection algorithm.

[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory connected with the at least one processor in communication; wherein

[0017] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the target detection method provided in the first aspect of the present disclosure.

[0018] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to enable the computer to execute the target detection method provided in the first aspect of the present disclosure.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the target detection method provided by the first aspect of the present disclosure.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0022] Figure 1 is one of the flowcharts of the target detection method provided by an embodiment of the present disclosure;

[0023] Figure 2 is the second flowchart of the target detection method provided by an embodiment of the present disclosure;

[0024] Figure 3 is the flowchart of obtaining the at least one problem target set of the point cloud frame in the target detection method provided by an embodiment of the present disclosure;

[0025] Figure 4 is the flowchart of obtaining the version error by version comparison in the target detection method provided by an embodiment of the present disclosure;

[0026] Figure 5 is one of the flowcharts of the target detection method provided by an embodiment of the present disclosure;

[0027] Figure 6 is the second flowchart of the target detection method provided by an embodiment of the present disclosure;

[0028] Figure 7 is the schematic diagram of drawing picture X1 provided by an embodiment of the present disclosure;

[0029] Figure 8 is the schematic diagram of drawing picture X2 provided by an embodiment of the present disclosure;

[0030] Figure 9 is one of the structural diagrams of the target detection device provided by an embodiment of the present disclosure;

[0031] Figure 10 is the second structural diagram of the target detection device provided by an embodiment of the present disclosure;

[0032] Figure 11is a structure diagram of a target detection device of one embodiment provided by the present disclosure;

[0033] Figure 12 is a block diagram of an electronic device for implementing a target detection method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of embodiments of the present disclosure by a person of ordinary skill in the art, and should be considered as illustrative only. Accordingly, persons of ordinary skill in the art will recognize that there are various changes and modifications that can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0035] As shown in Figure 1 According to an embodiment of the present disclosure, the present disclosure provides a target detection method, which can be applied to an electronic device, and the method comprises:

[0036] Step S101: performing target detection on the to-be-detected frame by a first target detection algorithm to obtain a detection target set,

[0037] There are various types of target detection algorithms, and in the present embodiment, the first target detection algorithm is not specifically limited. The to-be-detected frame is subjected to target detection by the first target detection algorithm to obtain a detection target set, which can also be referred to as a detection result (DT) set. The detection target set includes at least one detection target, and also includes related information of each detection target and a prediction score. The detection target can be understood as a detection box, and the related information can include but is not limited to three-dimensional coordinates, size information, detection category and heading angle. The size information can include length, width and height. In one example, the to-be-detected frame can include but is not limited to a point cloud frame. The prediction score of the detection target can represent the probability that the detection target is the corresponding detection category. It should be noted that the detection box and the annotation box in the present embodiment can be a three-dimensional box (3dbox) or a two-dimensional box.

[0038] Step S102: determining a first cost matrix based on the detection target set and an annotation target set of the to-be-detected frame;

[0039] Any element in the first cost matrix is the intersection over union (IoU) between a detection target in the detection target set and an annotation target in the annotation target set.

[0040] The target in the to-be-tested frame is labeled in advance to obtain a labeled target set of the to-be-tested frame. The labeled target set can also be understood as a ground truth (GT) target set. The labeled target set includes at least one labeled target and can also include related information of each labeled target. The labeled target can be understood as a ground truth target. The labeled target can be understood as a labeled box.

[0041] The point cloud frame sequence and the image frame sequence are collected at a roadside collection point. After preprocessing operations such as frame extraction, the point cloud frame can be labeled in combination with the point cloud frame and the corresponding image frame. In one example, the to-be-tested frame can be a to-be-tested point cloud frame. In the process of labeling the to-be-tested point cloud frame, the to-be-tested point cloud frame can be labeled in combination with the to-be-tested point cloud frame and the corresponding image frame to obtain a labeled target set.

[0042] In one example, the intersection over union can be a three-dimensional intersection over union (3D IoU). The detection box of the detection target and the labeled box of the labeled target are both cubes. If there is no intersection between the detection target A and the labeled target B, that is, there is no intersection between the detection box and the labeled box, then IoU = 0. If there is an intersection between the detection target A and the labeled target B, then A∩B represents the volume of the intersection of A and B, and A∪B represents the volume of the union of A and B. If the detection box and the labeled box are both rectangles, the intersection over union is a two-dimensional intersection over union. If there is an intersection between the detection target A and the labeled target B, then A∩B represents the area of the intersection of A and B, and A∪B represents the area of the union of A and B.

[0043] In this embodiment, the intersection over union between the detection target in the detection target set and the labeled target in the labeled target set is included in the first cost matrix. For example, the labeled target set includes I labeled targets, and the detection target set includes J detection targets. The labeled target is taken as a row, and the detection target is taken as a column. The first cost matrix has I rows and J columns and includes I×J elements. Any element in the first cost matrix is the intersection over union between one labeled target and one detection target. For example, the element in the i-th row and the j-th column of the first cost matrix is the intersection over union between the i-th labeled target and the j-th detection target.

[0044] Step S103: According to the first cost matrix, the detection target set and the labeled target set are subjected to problem target extraction and classification to obtain at least one problem target set of the to-be-tested frame. One problem target set corresponds to one problem category. The at least one problem target set is used to update the first target detection algorithm.

[0045] Since the detection target set is detected by the target detection algorithm, there may be detection errors (problems), that is, there may be problem targets in the obtained detection target set. In addition, for the labeled target set, there may be undetected targets, that is, missed detection, which can also be considered as problem targets. In the embodiment, the first cost matrix can be used to extract and classify problem targets from the detection target set and the labeled target set, and at least one problem target set of the to-be-tested frame is obtained.

[0046] One problem target set corresponds to one problem category, and problem targets of the same problem category are concentrated in the same problem target set. Since there is at least one problem target set, there is at least one problem category, and the number of problem categories is the same as the number of problem target sets. In one example, the at least one problem category can include, but is not limited to, at least one of the following five categories: missed detection, false detection, detection category error, position error, and error in both detection category and position.

[0047] The obtained at least one problem target set can be used to update the first target detection algorithm to improve the reliability and robustness of the first target detection algorithm. In one example, in the process of updating the first target detection algorithm, the number of problem targets in each problem target set in the at least one problem target set can be counted first to obtain the number of problem targets in the at least one problem target set. According to the number of problem targets in the at least one problem target set and the problem category of the at least one problem target set, the first target detection algorithm is updated. For example, if the number of problem targets of detection category error in the at least one problem category is large, the loss, weight, feature, etc. of the classification in the first target detection algorithm can be adjusted to realize algorithm update. For another example, if the number of problem targets of position error in the at least one problem category is large, further analysis can be performed on whether the position is inaccurate, the size is inaccurate, or the angle is inaccurate, and the anchor box regression of the first target detection algorithm is optimized and adjusted to realize algorithm update. For another example, if the number of problem targets of background false detection in the at least one problem category is large, the sample matching strategy and foreground and background classification features of the first target detection algorithm can be adjusted to realize algorithm update.

[0048] In the target detection method of the embodiments of the present disclosure, after the first cost matrix is determined based on the detection target set and the labeled target set of the frame to be detected, the detection target set and the labeled target set can be subjected to problem target extraction and classification according to the first cost matrix, so as to obtain at least one problem target set of the frame to be detected, one problem target set corresponding to one problem category, that is, the problem category of the problem target can be subdivided, and the problem targets of the same problem category are aggregated in the same problem target set, and the obtained at least one problem target set can be used to update the first target detection algorithm, so as to improve the robustness of the first target detection algorithm. Subsequently, the updated first target detection algorithm is used for target detection, so as to improve the accuracy of target detection.

[0049] In one embodiment, the detection target set and the labeled target set are subjected to problem target extraction and classification according to the first cost matrix, to obtain at least one problem target set of the frame to be detected, including:

[0050] The detection target set and the labeled target set are subjected to target matching according to the first cost matrix, to obtain a first matching result.

[0051] The detection target set and the labeled target set are subjected to problem target extraction and classification based on the first matching result and the first cost matrix, to obtain at least one problem target set of the frame to be detected.

[0052] After the first cost matrix is obtained, the detection targets in the detection target set and the labeled targets in the labeled target set need to be subjected to preliminary target matching through the first cost matrix, that is, to determine which detection targets and labeled targets are preliminarily matched successfully, which detection targets are not matched successfully, and which labeled targets are not matched successfully, to obtain a first matching result, that is, the matching result of each detection target and the matching result of each labeled target in the first matching result. Then, the detection target set and the labeled target set are subjected to problem target extraction and classification based on the first matching result and the first cost matrix, to obtain at least one problem target set of the frame to be detected, so as to improve the accuracy of determining the problem target set.

[0053] In one example, the detection target set and the labeled target set are subjected to target matching according to the first cost matrix, to obtain a first matching result, which can be subjected to target matching through the Hungarian algorithm according to the first cost matrix, to obtain the first matching result. The Hungarian algorithm is a matching algorithm, which can also be referred to as a combinatorial optimization algorithm, and the specific process thereof will not be described herein. In this example, the Hungarian algorithm is used in the first cost matrix to match the detection target set and the labeled target set, so as to reduce the matching complexity and improve the matching efficiency.

[0054] In one embodiment, based on the first matching result and the first cost matrix, the detection target set and the labeled target set are subjected to problem target extraction and classification to obtain at least one problem target set of the frame under test, including:

[0055] According to the first matching result and the first cost matrix, a first unmatched target set and a first matched target pair set are obtained from the detection target set and the labeled target set, wherein any matched target pair in the first matched target pair set includes a detection target and a matched labeled target, and the intersection-over-union between any matched target pair is greater than 0;

[0056] For each unmatched target in the first unmatched target set, if the unmatched target belongs to the labeled target set, the unmatched target is added to the problem target set of the missed detection category, and if the unmatched target belongs to the detection target set, the unmatched target is added to the problem target set of the false detection category;

[0057] Based on the first cost matrix and the detection category of the target in the first matched target pair set, the first matched target pair set is subjected to problem target extraction and classification to obtain M problem target sets, where M is a positive integer.

[0058] The targets that are not matched successfully after the preliminary matching by the first cost matrix belong to the first unmatched target set. It should be noted that the first matching result is the result of the preliminary matching by the first cost matrix, but if the intersection-over-union between two targets that are matched initially is not greater than 0, it can be determined that these two targets are not matched successfully finally and belong to the first unmatched target set. It can be understood that the first unmatched target set includes the targets that are not matched successfully according to the first cost matrix, and the initial matched pairs (any initial matched pair includes a detection target and a labeled target that is matched successfully initially with the detection target) that are matched successfully according to the first cost matrix, and the intersection-over-union between any initial matched pair is not greater than 0, while the detection target and the labeled target in any matched target pair in the first matched target pair set are not only matched successfully initially, but also the intersection-over-union between them is greater than 0, which indicates that the final matching is successful. It should be noted that the final unmatched detection target and labeled target indicate that the detection target is incorrect detection, i.e., a problem target. However, the final matched detection target and labeled target indicate that the detection target is matched successfully based on the first cost matrix and the first matching result, but it does not mean that the detection target is necessarily correct detection, i.e., it does not mean that the detection target is necessarily not a problem target. Further problem target extraction and classification of the first matched target pair set based on the detection category of the target in the first matched target pair set and the first cost matrix are needed to obtain M problem target sets.

[0059] In the embodiment, for the unmatched targets in the first unmatched target set, the unmatched targets can be added to the problem target set of the corresponding category according to which one of the detection target set and the labeled target set the unmatched targets belong to, for example, if a certain unmatched target in the first unmatched target set is a target in the labeled target set, the unmatched target can be added to the problem target set of the Miss error category, and if the unmatched target is a target in the detection target set, the unmatched target can be added to the problem target set of the False error category (also referred to as the background False error category). For the first matched target pair set, the problem target extraction and classification need to be further performed according to the first cost matrix and the detection categories of the targets in the first matched target pair set, to obtain M problem target sets. It should be noted that the detection categories are the detection categories obtained by performing target detection using the first target detection algorithm, for example, the detection category of the detected target A1 is a vehicle, the detection category of the detected target A2 is a pedestrian, and the detection category of the detected target A3 is a dog.

[0060] In the embodiment, first, the first unmatched target set and the first matched target pair set can be obtained from the detection target set and the labeled target set using the first matching result and the first cost matrix, and then the first unmatched target set and the first matched target pair set can be classified using different classification manners, that is, for each unmatched target in the first unmatched target set, if the unmatched target belongs to the labeled target set, the unmatched target is added to the problem target set of the Miss error category, and if the unmatched target belongs to the detection target set, the unmatched target is added to the problem target set of the False error category, and for the first matched target pair set, the problem target extraction and classification need to be performed on the first matched target pair set by using the first cost matrix and the detection categories of the targets in the first matched target pair set, to obtain M problem target sets, so that the problem target extraction and classification of the detection target set and the labeled target set are realized, at least one problem target set of the frame under test is obtained, and the accuracy of the problem target extraction and classification can be improved.

[0061] In one embodiment, the problem target extraction and classification of the first matched target pair set are performed based on the first cost matrix and the detection categories of the targets in the first matched target pair set, to obtain M problem target sets, including any one of the following:

[0062] in a case that the intersection over union between the first matched target pair in the first cost matrix is less than the first threshold value and the detection class between the detection target and the labeled target in the first matched target pair is matched, adding the detection target in the first matched target pair into the problem target set of the position error category;

[0063] in a case that the intersection over union between the first matched target pair in the first cost matrix is less than the first threshold value and the detection class between the detection target and the labeled target in the first matched target pair is matched, adding the detection target in the first matched target pair into the problem target set of the position error category;

[0064] in a case that the intersection over union between the first matched target pair in the first cost matrix is less than the first threshold value and the detection class between the detection target and the labeled target in the first matched target pair is matched, adding the detection target in the first matched target pair into the problem target set of the position error category;

[0065] It should be noted that the first threshold value can be pre-set according to experience, and the specific value of the first threshold value is not limited in the embodiment of the present disclosure. For example, for a motor vehicle, it can be set to 0.5, and for the remaining small targets, it can be set to 0.25. In one example, the detection class mismatch can be inconsistent or different, and the detection class match can be consistent or the same.

[0066] In the process of extracting and classifying the question targets from the first matching target pair set in the embodiment, for any matching target pair in the first matching target pair set, the intersection-over-union between the matching target pair can be obtained from the first cost matrix. If the intersection-over-union between the matching target pair is greater than or equal to the first threshold, and the detection class between the detection target and the labeled target in the matching target pair is not matched, it means that the detection class of the detection target in the matching target pair is wrong (Class error), and the detection target is a question target of detection class error. The detection target in the matching target pair can be added to the question target set of the detection class error category. If the intersection-over-union between the matching target pair is less than the first threshold, and the detection class between the detection target and the labeled target in the matching target pair is matched, it means that the detection position of the detection target in the matching target pair is wrong (location error), and the detection target is a question target of position error. The detection target in the matching target pair can be added to the question target set of the position error category. If the intersection-over-union between the matching target pair is less than the first threshold, and the detection class between the detection target and the labeled target in the matching target pair is not matched, it means that the detection target in the matching target pair is not only a detection class error but also a detection position error, and the detection target is a question target of detection class error and position error. The detection target in the matching target pair can be added to the question target set of the class error and location error category. In addition, if the intersection-over-union between the matching target pair is greater than or equal to the first threshold, and the detection class between the detection target and the labeled target in the matching target pair is matched, it means that the detection target in the matching target pair is correctly detected, that is, a true positive, and is not a question target, and does not need to be added to any question target set. Each matching target pair in the first matching target pair set can be judged whether it is a question target and classified through the similar process, so as to extract and classify the question targets from the first matching target pair set and obtain M question target sets.

[0067] In the embodiment, the extraction and classification of the question targets from the first matching target pair set can be realized by comparing the intersection-over-union between the matching target pair with the first threshold and judging whether the detection classes between the matching target pair are the same, and adding the detection target in the matching target pair to the corresponding question target set according to the comparison result and the detection class judgment result, so as to improve the accuracy of the question target classification.

[0068] In one embodiment, after obtaining the at least one question target set of the to-be-detected frame, the method further includes:

[0069] obtain a second problem target set of the to-be-tested frame, the second problem target set being a problem target set corresponding to the second target detection algorithm, the version of the first target detection algorithm being different from the version of the second target detection algorithm;

[0070] determine a second cost matrix based on the first problem target set and the second problem target set, any element in the second cost matrix being an intersection-union ratio between a problem target in the first problem target set and a problem target in the second problem target set, the first problem target set being a problem target set corresponding to the first target detection algorithm, the first problem target set including at least one problem target set;

[0071] perform problem labeling on the first problem target set and the second problem target set according to the second cost matrix, the labeled first problem target set and the labeled second problem target set being used for updating the first target detection algorithm.

[0072] It can be understood that the first target detection algorithm and the second target detection algorithm are different versions of the same target detection algorithm, the detection target set obtained by performing target detection on the to-be-tested frame by the first target detection algorithm is regarded as a first detection target set, the second detection target set is obtained by performing target detection on the to-be-tested frame by the second target detection algorithm in advance, a third cost matrix is determined based on the second detection target set and the labeled target set of the to-be-tested frame, any element in the third cost matrix being an intersection-union ratio between a detection target in the second detection target set and a labeled target in the labeled target set; problem targets are extracted and classified from the second detection target set and the labeled target set according to the third cost matrix, to obtain the second problem target set of the to-be-tested frame, it can be understood that the obtained second problem target set corresponds to the second target algorithm, the second problem target set includes H problem target sets, H being at least one, one problem target set in the H problem target sets corresponding to one problem category, the H problem target sets corresponding to H problem categories. In addition, it needs to be noted that the process of determining the second problem target set of the to-be-tested frame by the second target detection algorithm is similar to the process of determining the first problem target set of the to-be-tested frame by the first target detection algorithm, the difference being that the target detection algorithms are different, that is, the target detection algorithms are different versions.

[0073] The at least one problem target set determined based on the first target detection algorithm can be regarded as the first problem target set, after the at least one problem target set of the to-be-tested frame is obtained, the second problem target set corresponding to the second target detection algorithm can be obtained, then the first problem target set corresponding to the first target detection algorithm and the second problem target set of the second target detection algorithm are compared, error information between the different versions of the target detection algorithms is obtained, and the first problem target set and the second problem target set are labeled.

[0074] Specifically, the second cost matrix can be determined by using the first problem target set and the second problem target set, any element in the second cost matrix being an intersection-union ratio between a problem target in the first problem target set and a problem target in the second problem target set, for example, the number of problem targets in the second problem target set being P (a positive integer), the number of problem targets in the first problem target set being Q (a positive integer), the second cost matrix having P rows and Q columns, including P*Q elements, the element in the pth row and the qth column of the second cost matrix being an intersection-union ratio between the pth problem target in the second problem target set and the qth problem target in the first problem target set.

[0075] The first target detection algorithm can be understood as the current latest version of the target detection algorithm, and the second target detection algorithm can be a historical version of the target detection algorithm. The historical version of the target detection algorithm can be determined from multiple historical versions according to a user selection, or the historical version of the target detection algorithm is the latest version in the multiple historical versions. Since the detection results of different versions of the target detection algorithm can be different, the new version can solve the problems of the historical version, or can introduce new problems. In this embodiment, the first problem target set and the second problem target set can be marked with problems, for example, the first problem target set and the second problem target set can be marked with problems according to the second cost matrix. An example, the problem marking result of a problem target can include that the problem has been fixed, the problem has newly appeared, or the problem has not been solved.

[0076] In this embodiment, the first problem target set and the second problem target set can be marked with problems through version comparison. Whether the first target detection algorithm introduces problems, whether the first target detection algorithm fixes previous problems, and which problems are not fixed can be determined. The marked first problem target set and the marked second problem target set can be used for updating the first target detection algorithm, so that the reliability and robustness of the first target detection algorithm can be improved.

[0077] In one embodiment, the first problem target set and the second problem target set are marked with problems according to the second cost matrix, including:

[0078] The first problem target set and the second problem target set are matched according to the second cost matrix to obtain a second matching result.

[0079] The first problem target set and the second problem target set are marked with problems by using the second matching result and the second cost matrix.

[0080] After the second cost matrix is obtained, the problem targets in the first problem target set and the problem targets in the second problem target set are preliminarily matched through the second cost matrix, that is, it is determined which problem targets are preliminarily matched successfully and which problem targets are not matched successfully, to obtain a second matching result, that is, the matching result of each problem target in the first problem target set and the second problem target set in the second matching result. Then, the first problem target set and the second problem target set are labeled by using the second matching result and the second cost matrix, so as to improve the accuracy of the problem labeling.

[0081] In one example, according to the second cost matrix, the target matching of the first problem target set and the second problem target set is performed to obtain the second matching result. According to the second cost matrix, the target matching of the first problem target set and the second problem target set can be performed by using the Hungarian algorithm to obtain the second matching result, so as to reduce the matching complexity of the first problem target set and the second problem target set and improve the matching efficiency.

[0082] In one embodiment, the problem labeling of the first problem target set and the second problem target set is performed by using the second matching result and the second cost matrix, including:

[0083] According to the second matching result and the second cost matrix, a second unmatched target set and a second matched target pair set are obtained from the first problem target set and the second problem target set. Any matched target pair in the second matched target pair set includes a problem target in the first problem target set and a matched problem target in the second problem target set, and the intersection-union ratio between any matched target pair in the second matched target pair set is greater than 0.

[0084] For each unmatched target in the second unmatched target set, if the unmatched target belongs to the second problem target set, the unmatched target is labeled as a problem that has been repaired; if the unmatched target belongs to the first problem set, the unmatched target is labeled as a newly appeared problem.

[0085] Based on the second cost matrix, the problem labeling of the second matched target pair set is performed.

[0086] The target that is not matched successfully after the preliminary matching by the second cost matrix belongs to the second unmatched target set. It should be noted that the second matching result is the result of the preliminary matching by the second cost matrix, but if the intersection-union ratio between the two targets that are preliminarily matched is not greater than 0, it can be determined that the two targets are finally not matched successfully and belong to the second unmatched target set. It can be understood that the second unmatched target set includes the problem targets that are not matched successfully according to the second cost matrix, from the first problem target set and the second problem target set, and the preliminary matching pairs (any preliminary matching pair includes two problem targets that are matched, one of which belongs to the first problem target set and the other of which belongs to the second problem target set) that are matched successfully according to the second cost matrix, and the intersection-union ratio between any preliminary matching pair is not greater than 0, while the two problem targets in any matching target pair in the second matching target pair set are not only initially matched successfully, but also the intersection-union ratio between the two problem targets is greater than 0, which means that the final matching is successful.

[0087] In the embodiment, first, the second unmatched target set and the second matching target pair set can be obtained from the first problem target set and the second problem target set by using the second matching result and the second cost matrix. Subsequently, in the process of problem labeling for the second unmatched target set and the second matching target pair set, for the unmatched target in the second unmatched target set, if the unmatched target is a target in the second problem target set, that is, the unmatched target belongs to the second problem target set, the unmatched target can be labeled as a problem that is repaired, and if the unmatched target is a target in the first problem target set, that is, the unmatched target belongs to the first problem target set, the unmatched target can be labeled as a problem that newly appears. For the second matching target pair set, problem labeling for the second matching target pair set is performed by using the second cost matrix. In this way, problem labeling for the first problem target set and the second problem target set is implemented, that is, in the embodiment, the second unmatched target set and the second matching target pair set can be obtained from the first problem target set and the second problem target set, different labeling manners are used for the second unmatched target set and the second matching target pair set, so that problem labeling for the targets is implemented, and the accuracy of problem labeling can be improved.

[0088] In one embodiment, the problem labeling for the second matching target pair set based on the second cost matrix includes any of the following:

[0089] In a case where the intersection-union ratio between the second matching target pairs in the second cost matrix is greater than or equal to the second threshold value, the problem targets in the second matching target pair are labeled as problems that are not solved, and the second matching target pair is any matching target pair in the second matching target pair set;

[0090] In a case that the intersection union ratio between the second matched target pair in the second cost matrix is less than the second threshold value, marking the first problem target in the second matched target pair as problem newly occurring, marking the second problem target in the second matched target pair as problem repaired, the first problem target belonging to the first problem target set, and the second problem target belonging to the second problem target set.

[0091] It should be noted that the second threshold value can be pre-set according to experience, and the specific value of the second threshold value is not limited in the embodiment of the present disclosure. The second threshold value can be the same as or different from the first threshold value.

[0092] In the process of marking the problems of the second matched target pair set in the embodiment, for any matched target pair in the second matched target pair set, the intersection union ratio between the matched target pair can be obtained from the second cost matrix. If the intersection union ratio between the matched target pair is greater than or equal to the second threshold value, it indicates that the problem of the problem target in the matched target pair is not solved, and the problem target in the matched target pair is marked as problem not solved. If the intersection union ratio between the matched target pair is less than the second threshold value, since the matched target pair includes two problem targets, i.e., the first problem target and the second problem target, the marking result thereof needs to be determined according to the problem target set to which the target belongs. For example, the first problem target in the matched target pair belonging to the first problem target set can be marked as problem newly occurring, and the second problem target in the matched target pair belonging to the second problem target set can be marked as problem repaired.

[0093] In the embodiment, the problems of the second matched target pair set can be marked by comparing the intersection union ratio between the matched target pair with the second threshold value, marking the problem target in the matched target pair as the corresponding result according to the comparison result, and improving the accuracy of problem marking.

[0094] In one embodiment, the frame to be tested includes N point cloud frames, N is a positive integer, and any frame in the N point cloud frames corresponds to at least one problem target set;

[0095] In the embodiment, the method further includes:

[0096] counting the problem target sets of the N point cloud frames to obtain a target statistical result;

[0097] The target statistical result includes at least one of the following:

[0098] The first statistical result includes N first sub-statistical results, and any first sub-statistical result includes a frame number and a problem target in the point cloud frame corresponding to the frame number.

[0099] The second statistical result includes K second sub-statistical results, and each second sub-statistical result includes a problem category and a total number of problem targets corresponding to the problem category, K being a positive integer;

[0100] The third statistical result includes L third sub-statistical results, and each third sub-statistical result includes a problem category, a frame number of each point cloud frame in the N point cloud frames, and a number of problem targets in the point cloud frame corresponding to the frame number and belonging to the problem category, L being a positive integer.

[0101] It can be understood that the first statistical result is a result of frame-by-frame statistics, wherein each first sub-statistical result includes a frame number and problem targets in the point cloud frame corresponding to the frame number, the second statistical result is a result of problem category-by-problem category statistics, wherein each second sub-statistical result includes a problem category and a total number of problem targets corresponding to the problem category, and the third statistical result is a result of problem category-by-frame statistics, wherein each third sub-statistical result includes a problem category, a frame number of each point cloud frame in the N point cloud frames, and a number of problem targets in the point cloud frame corresponding to the frame number and belonging to the problem category. The problem categories in the target statistical result belong to the at least one problem category described above.

[0102] It should be noted that for a point cloud frame, if at least one problem target set of the point cloud frame is regarded as a first problem target set, for the N point cloud frames, N first problem target sets can be obtained, and for the second target detection algorithm, N second problem target sets can be similarly obtained. The N first problem target sets correspond one-to-one to the N point cloud frames, and the N second problem target sets correspond one-to-one to the N point cloud frames. It can be understood that the statistics of the problem target sets of the N point cloud frames is to statistically analyze all the problem target sets of the N point cloud frames, that is, to statistically analyze the N first problem target sets.

[0103] In this embodiment, the first statistical result, the second statistical result, and the second statistical result can be obtained by performing statistics according to different statistical rules, respectively, thereby improving the flexibility of statistics and obtaining different forms of statistical results for the user to view.

[0104] In one embodiment, the target statistical result includes the third statistical result;

[0105] After obtaining the target statistical result by statistically analyzing the N point cloud frames, the following operations are further included:

[0106] At least one of the preset number of problem targets of the reference problem category and the first preset percentage and the number of problem targets of each point cloud frame in the N point cloud frames under the reference problem category are used to filter the N point cloud frames to obtain first target point cloud frames, and the reference problem category is any one of the problem categories in the third statistical result.

[0107] The first target point cloud frame, the rendering picture of the first target point cloud frame, the detection target set of the first target point cloud frame, and the labeled target set of the first target point cloud frame are stored.

[0108] If the target statistical result includes the third statistical result, the third statistical result includes L third sub-statistical results, any one of the third sub-statistical results includes a problem category, the third statistical result includes L problem categories, the reference problem category is any one of the L problem categories, and at least one of the preset problem quantity of the reference problem category and the first preset percentage of the reference problem category and the problem target quantity of each point cloud frame in the N point cloud frames under the reference problem category can be used to filter the N point cloud frames to obtain the first target point cloud frame, which can be used for research and development, that is, in the research and development process, the target detection algorithm can be iteratively updated by analyzing the problem targets in the first target point cloud frame obtained after filtering, for example, the target detection algorithm can be iteratively updated by the first problem target set after classification of the first target point cloud frame, avoiding analyzing the problem targets of all point cloud frames, reducing the calculation amount, and more targetedly updating the algorithm, improving the algorithm updating accuracy.

[0109] In one example, the point cloud frames in which the problem target quantity of the reference problem category in the N point cloud frames is less than or equal to the preset problem quantity of the reference problem category can be filtered out, and the point cloud frames in which the problem target quantity of the reference problem category in the N point cloud frames is greater than the preset problem quantity of the reference problem category can be retained to obtain the first target point cloud frame. It can be understood that the problem target quantity of the reference problem category in the first target point cloud frame is greater than the preset problem quantity of the reference problem category. In another example, the point cloud frames in which the problem target quantity of the reference problem category in the N point cloud frames is not in the top T can be filtered out, and the point cloud frames in which the problem target quantity of the reference problem category in the N point cloud frames is in the top T can be retained to obtain the first target point cloud frame. It can be understood that the problem target quantity of the reference problem category in the first target point cloud frame is in the top T among the problem target quantities of the reference problem category in the N point cloud frames. Wherein, T can be the product of N and the first preset percentage, for example, N is 100 and the first preset percentage is 20%, it can be understood that the point cloud frames in which the problem target quantity of the reference problem category is in the top 20 among the problem target quantities of the reference problem category in the 100 point cloud frames need to be retained as the first target point cloud frame.

[0110] It should be noted that the frame number of the first target point cloud frame is one or more, and the drawing picture of the first target point cloud frame can be a picture drawn based on the first target point cloud frame and the related information of the problem target in the first problem target set of the first target point cloud frame, or a picture drawn based on the first target point cloud frame, the related information of the problem target in the first problem target set of the first target point cloud frame, and the problem marking result, the related information of the problem target in the second problem target set of the first target point cloud frame, and the problem marking result.

[0111] In the embodiment, after filtering and determining the first target point cloud frame, the first target point cloud frame, the drawing picture of the first target point cloud frame, the detection target set of the first target point cloud frame, and the labeled target set of the first target point cloud frame can be stored, so as to reduce the storage amount, avoid occupying more storage space, and save storage space.

[0112] In one embodiment, the frame to be tested includes N point cloud frames, N is a positive integer, and any frame of the N point cloud frames corresponds to at least one problem target set;

[0113] After marking the first problem target set and the second problem target set according to the second cost matrix, the method further includes:

[0114] The marked first problem target set of the N point cloud frames and the marked second problem target set of the N point cloud frames are counted to obtain a reference statistical result, and the reference statistical result includes at least one of the following:

[0115] The fourth statistical result includes R fourth sub-statistical results, any fourth sub-statistical result includes a problem marking result and a total number of targets corresponding to the problem marking result in the N point cloud frames, and R is a positive integer;

[0116] The fifth statistical result includes S fifth sub-statistical results, any fifth sub-statistical result includes a problem marking result, a frame number of each point cloud frame in the N point cloud frames, and a number of targets corresponding to the problem marking result in the point cloud frame corresponding to each point cloud frame number, and S is a positive integer.

[0117] It can be understood that the fourth statistical result is a result counted according to a problem marking result (which can be understood as a problem marking classification or a problem marking category), any fourth sub-statistical result includes a problem marking result and a total number of targets corresponding to the problem marking result in the N point cloud frames, and the fifth statistical result is a result counted according to a problem marking result and a frame, any fifth sub-statistical result includes a problem marking result, a frame number of each point cloud frame in the N point cloud frames, and a number of targets corresponding to the problem marking result in the point cloud frame corresponding to each point cloud frame number.

[0118] In the embodiment, the fourth statistical result and the fifth statistical result can be obtained respectively according to different statistical rules, the flexibility of the statistics is improved, and different forms of statistical results can be obtained for the user to view.

[0119] In one embodiment, the reference statistical result includes the fifth statistical result.

[0120] The method further includes:

[0121] The N point cloud frames are filtered according to at least one of the preset quantity threshold of the candidate problem marking result and the second preset percentage and the target quantity of each point cloud frame in the N point cloud frames under the candidate problem marking result, to obtain second target point cloud frames, and the candidate problem marking result is any marking result in the problem marking result of the fifth statistical result.

[0122] The second target point cloud frames, the drawing pictures of the second target point cloud frames, the detection target sets of the second target point cloud frames, and the labeled target sets of the second target point cloud frames are stored.

[0123] If the reference statistical result includes the fifth statistical result, the fifth statistical result includes S fifth sub-statistical results, any fifth sub-statistical result includes a problem marking result, the fifth statistical result includes S problem marking results, the candidate problem marking result is any marking result in the S problem marking results, the N point cloud frames can be filtered according to at least one of the preset quantity threshold of the candidate problem marking result and the second preset percentage of the candidate problem marking result and the target quantity of each point cloud frame in the N point cloud frames under the candidate problem marking result, to obtain second target point cloud frames, which can be used for research and development, that is, in the research and development process, the target detection algorithm can be iteratively updated by analyzing the problem targets in the second target point cloud frames obtained after filtering, for example, the target detection algorithm can be iteratively updated by using the first problem target set after marking and the second problem target set after marking of the second target point cloud frames, the problem targets in all point cloud frames are avoided to be analyzed, the calculation amount is reduced, the algorithm updating accuracy is improved, and the algorithm updating is more targeted.

[0124] In one example, the point cloud frames in which the number of target points of the candidate problem labeling result is less than or equal to the preset threshold of the number of target points of the candidate problem labeling result are filtered out, and the point cloud frames in which the number of target points of the candidate problem labeling result is greater than the preset threshold of the number of target points of the candidate problem labeling result are retained to obtain the second target point cloud frames. It can be understood that the number of target points of the candidate problem labeling result in the second target point cloud frames is greater than the preset threshold of the number of target points of the candidate problem labeling result. In another example, the point cloud frames in which the number of target points of the candidate problem labeling result is not in the top U are filtered out, and the point cloud frames in which the number of target points of the candidate problem labeling result is in the top U are retained to obtain the second target point cloud frames. It can be understood that the number of target points of the candidate problem labeling result in the second target point cloud frames is in the top U among the number of target points of the candidate problem labeling result in the N point cloud frames. Wherein, U can be the product of N and the second preset percentage. For example, N is 100, and the second preset percentage is 20%. It can be understood that the point cloud frames in which the number of target points of the candidate problem labeling result is in the top 20 among the number of target points of the candidate problem labeling result in the 100 point cloud frames are retained as the second target point cloud frames.

[0125] It should be noted that the number of the second target point cloud frames is one or more, and the drawing picture of the second target point cloud frames can be a picture drawn based on the second target point cloud frames and the related information of the problem targets in the first problem target set of the second target point cloud frames, or a picture drawn based on the second target point cloud frames, the related information of the problem targets in the first problem target set of the second target point cloud frames, and the problem labeling result, the related information of the problem targets in the second problem target set of the second target point cloud frames, and the problem labeling result.

[0126] In this embodiment, after filtering and determining the second target point cloud frames, the second target point cloud frames, the drawing picture of the second target point cloud frames, the detection target set of the second target point cloud frames, and the labeling target set of the second target point cloud frames can be stored, which can reduce the storage amount, avoid occupying too much storage space, and save storage space.

[0127] In one example, during storage, the directory for storage can be determined according to the accuracy and recall rate of the point cloud frames. For example, for any point cloud frame in the first target point cloud frames or any point cloud frame in the second target point cloud frames, the accuracy and recall rate of the point cloud frame can be calculated according to at least one problem target set of the point cloud frame. In the case where the accuracy is less than the preset accuracy threshold and / or the recall rate is less than the preset recall rate threshold, the point cloud frame, the drawing picture of the point cloud frame, the detection target set of the point cloud frame, and the labeling target set of the point cloud frame can be stored in a first directory, for example, a check directory. Otherwise, they are stored in a second directory, for example, a pass directory.

[0128] In one example, for any point cloud frame in the first target point cloud frame or for any point cloud frame in the second target point cloud frame, at least one problem target set of the point cloud frame obtained according to the problem target extraction and classification of the detection target set of the point cloud frame and the labeled target set of the point cloud frame, the number of true positive (TP) targets in the point cloud frame, the number of false positive (FP) targets in the point cloud frame, and the number of false negative (FN) targets in the point cloud frame can be counted, and the accuracy Precision of the point cloud frame can be determined by the following formula:

[0129]

[0130] wherein ∑TP represents the number of true positive targets, ∑(TP+FP) represents the sum of the number of true positive targets and the number of false positive targets, and can be understood as the total number of targets in the detection target set of the point cloud frame.

[0131] The accuracy Recall of the point cloud frame can be determined by the following formula:

[0132]

[0133] wherein ∑(TP+FN) represents the sum of the number of true positive targets and the number of false negative targets, and can be understood as the total number of targets in the labeled target set of the point cloud frame.

[0134] The process of the above method will be specifically described in the following specific embodiments, taking N point cloud frames as an example.

[0135] As shown in the following figure, the overall flow of the method of the present embodiment is as follows: Figure 2

[0136] First, for each point cloud frame, read the labeled results of the point cloud frame and the detection results under the first target detection algorithm of the current test version, and generate a labeled target set and a detection target set;

[0137] Then, compare the detection target set with the labeled target set to obtain error information, that is, to obtain at least one problem target set (the first problem target set) of the point cloud frame;

[0138] Secondly, calculate the accuracy and recall of the point cloud frame;

[0139] Further, determine whether to perform version comparison;

[0140] ​If it is determined to perform version comparison, the first target detection algorithm of the current detection version and the second target detection algorithm of the target historical version are compared, the first problem target set of the first target detection algorithm and the second problem target set of the second target detection algorithm are marked, and the problem marking results of the first problem target set and the second problem target set are obtained, that is, the version error is obtained, which can also be understood as obtaining the marked first problem target set and the marked second problem target set.

[0141] Subsequently, the version error of the point cloud frame is used to create a blank canvas, draw the ground and the target point cloud, obtain a first preliminary picture, and draw a target box (which can include a detection box and a labeling box) on the basis of the first preliminary picture using at least one problem target set of the point cloud frame to obtain a first drawing picture;

[0142] If it is determined not to perform version comparison, a blank canvas can be created, the ground and the target point cloud are drawn, a second preliminary picture is obtained, and a target box is drawn on the basis of the second preliminary picture using at least one problem target set of the point cloud frame to obtain a second drawing picture;

[0143] It is determined whether the accuracy and the recall rate meet the condition, for example, whether the accuracy is less than a preset accuracy threshold and whether the recall rate is less than a preset recall threshold, if the accuracy is less than the preset accuracy threshold and the recall rate is less than the preset recall threshold, it is determined that the condition is met, otherwise the condition is not met;

[0144] If the condition is met, the point cloud frame, the drawing picture (the first drawing picture or the second drawing picture) of the point cloud frame, the detection target set and the labeling target set of the point cloud frame are stored in the check directory;

[0145] If the condition is not met, the point cloud frame, the drawing picture (the first drawing picture or the second drawing picture) of the point cloud frame, the detection target set and the labeling target set of the point cloud frame are stored in the pass directory.

[0146] The preset accuracy threshold and the preset recall threshold can be set according to the research and development stage, if it is in the early stage of research and development, the average accuracy and the average recall rate on the entire data set can be set, if the accuracy and the recall rate are greater than or equal to the corresponding average value, the point cloud frames and the corresponding results are stored in the pass directory, and the pictures generated by the point cloud frames can be ignored to reduce the workload, otherwise the attention is focused on the pictures generated by the frames with accuracy and recall rate less than the average value; if it is in the later stage of research and development, most of the missed detection and false detection have been basically eliminated, and fine positioning of the problem is required, the preset accuracy threshold and the preset recall threshold can be directly set to fixed values, such as 95%, and the frames with accuracy and recall rate less than 95% are checked.

[0147] Whether to perform version comparison is selected according to iteration requirements, if version comparison is needed, a benchmark version (target historical version) of the problem target set needs to be generated in advance as input, compared with the problem target set of the current test version, and the version error is output, if no comparison is needed, the visualization drawing is directly performed.

[0148] In addition, it should be noted that the target box is drawn with the relevant information of the target, the problem category and the version error information as input, and the drawing mask is read to select whether to draw the target of the current category. If only the detection result of the error (false detection) is wanted to be viewed, the drawing switch of the correctly detected target can be turned off, and the correctly detected target will not be drawn; or the specific error of each category is wanted to be viewed, the current error category is turned on and the other categories are turned off, and then the drawing is performed.

[0149] As shown in FIG. 6, the process of obtaining at least one problem target set of the point cloud frame is as follows: Figure 3

[0150] Firstly, the annotation target set (GT) and the detection target set (DT) of the point cloud frame are taken as input, the 3D IoU between each target in the two sets is calculated, and the first cost matrix is obtained, the annotation target is taken as row and the detection target is taken as column;

[0151] Secondly, the initial matching of the annotation target set and the detection target set in the first cost matrix is performed by using the Hungarian algorithm, that is, whether the targets in the annotation target set and the detection target set match is determined, if the two targets are preliminarily matched successfully and the intersection over union of the two targets in the first cost matrix is greater than 0, it is represented that the two targets are matched, otherwise the two targets are not matched;

[0152] For the unmatched target, it is determined whether it is in the detection target set, if it is in the detection target set, it can be determined as a background error, if it is not in the detection target set, it is in the annotation target set, it can be determined as a miss error.

[0153] For the matched target, it is further determined whether the corresponding 3D IoU is greater than the first threshold;

[0154] If the 3D IoU is greater than or equal to the first threshold and the detection categories of the two matched targets are the same, it can be determined as a true positive, if the 3D IoU is greater than or equal to the first threshold but the detection categories of the two matched targets are different, it can be determined that the detection target is a class error;

[0155] ​If the two matched targets have the same detection category and the value is less than the first threshold, it is determined that the detection target is a location error; if the two matched targets have different detection categories and the value is less than the first threshold, it is determined that the detection target is a class error and a location error.

[0156] Through the above problem target classification, the problem of the detection effect of the current test version can be refined, and time and effort can be focused on the head problem, so as to break down the problem one by one. For example, if the number of problem targets of detection category error is large, the loss, weight, feature, etc. of the classification in the first target detection algorithm can be adjusted to realize algorithm updating; for another example, if the number of problem targets of location error is large, further analysis can be made on whether the position is inaccurate, the size is inaccurate or the angle is inaccurate, and the anchor box regression of the first target detection algorithm can be optimized and adjusted to realize algorithm updating; for another example, if the number of problem targets of background false detection is large, the sample matching strategy and the foreground and background classification features of the first target detection algorithm can be adjusted to realize algorithm updating.

[0157] As shown in FIG. 6, the process of obtaining the version error is as follows: Figure 4

[0158] Firstly, the problem target set (the first problem target set) of the comparison version (test version, i.e. the current test version) and the problem target set (the second problem target set) of the benchmark version (base version) are taken as inputs, and the 3D IoU between each target in the two sets is calculated to obtain a second cost matrix, with the targets of the benchmark version as rows and the targets of the comparison version as columns;

[0159] Secondly, the initial matching of the first problem target set and the second problem target set is performed in the second cost matrix by using the Hungarian algorithm, and if the initial matching of two targets is successful and the corresponding intersection-over-union in the second cost matrix of the two targets is greater than 0, it means that the two targets are matched, otherwise the targets are not matched;

[0160] For the unmatched targets, if they are in the second problem target set, they are marked as fixed problems, and if they are in the first problem target set, they are marked as new problems;

[0161] For the matched targets, it is further judged whether the corresponding 3D IoU is greater than a second threshold;

[0162] If the corresponding 3D IoU is greater than or equal to the second threshold, the matched target is marked as a remain problem; ​

[0163] If less than the second threshold, mark the targets in the matched targets belonging to the second problem target set as problem repaired, and mark the targets in the matched targets belonging to the first problem target set as problem newly appeared.

[0164] Through version comparison, the newly added problems of the current test version can be quickly located, and it can be determined whether more problems are introduced in the current test version. If more problems are introduced, it means that the current test version is unqualified. It can also be found whether the previous problems are repaired in the current test version. If many problems are repaired, it means that the current test version has a better effect. It can also be judged which problems have not been repaired, and the subsequent problems need to be solved.

[0165] As shown in Figure 5 After the problem classification of each frame in the N point cloud frames is performed and the first problem target set of the N point cloud frames is determined, the first problem target set of the N point cloud frames can be counted and stored, and the specific process is as follows:

[0166] The first problem target set of each frame is recorded, and all problem targets in the first problem target set of the N point cloud frames are summarized. Specifically, the related information of the problem targets can be stored in a Json file according to frames. It can be understood that the counting result is the first counting result, and the specific form is as follows:

[0167] {“frame number”: [[target1], [target2] …], “frame number”: [[target1], [target2] …]};

[0168] The target problem details of each frame are counted, that is, the number of problem targets of each problem category in each frame is counted, the target problem details of the N point cloud frames are summarized, the total number of problem targets of each problem category is counted, and the second counting result is obtained. The specific form is as follows:

[0169] {“problem category 1”: total number, “problem category 2”: total number, …};

[0170] For example, the total number of missed detections is 65444, the total number of false detections is 30650, the total number of detection categories and positions that are both wrong is 6478, the total number of position errors is 17761, and the total number of detection category errors is 5751.

[0171] According to the number of problem targets of each problem category appearing in each frame, the frame number is arranged in descending order in each problem category, and the frame number with the most problem targets is arranged in the front, and then stored in the json. It can be understood that the counting result obtained in this way is the third counting result (problem category sorting result), and the specific form is as follows:

[0172] {“Problem Category 1”:{“Frame Number 1”:”Quantity 1”,“Frame Number 2”:”Quantity 2”……},“Problem Category 2”:{“Frame Number 1”:”Quantity 1”,“Frame Number 2”:”Quantity 2”……}……};

[0173] For example, the number of missed detections for point cloud frames with frame number 007671 was 36, the number of missed detections for point cloud frames with frame number 005218 was 29, and the number of missed detections for point cloud frames with frame number 003952 was 28, etc.

[0174] It should be noted that the above-mentioned problem categories may include missed detections, false detections, incorrect detection categories, incorrect locations, and both incorrect detection categories and locations. N point cloud frames can be filtered according to the target number of problems in a certain problem category (e.g., missed detections) using a corresponding preset problem count or a first preset percentage (top 20%) to obtain the first target point cloud frame. The first target point cloud frame, its rendered image, the set of detected targets, and the set of labeled targets are then stored in the corresponding directories according to the problem category. Filtering by the first preset percentage can be used in the early stages of development when there are many problems, allowing for the selection of the most problematic issues. Filtering by the preset problem count can be used in the later stages of development to select all frames where the number of problems does not meet expectations.

[0175] like Figure 6 As shown, after version comparison, the first problem target set after labeling N point cloud frames and the second problem target set after labeling N point cloud frames can be statistically analyzed and stored. The specific process is as follows:

[0176] The comparison results for each frame are detailed, specifically the number of targets for each problem labeling result in each frame. The comparison results for N point cloud frames are then summarized, and the total number of targets for each problem labeling result is calculated to obtain the fourth statistical result. This summarized statistical result is stored in a JSON file. The specific format of the fourth statistical result is as follows:

[0177] {“Question Marking Result 1”: Total, “Question Marking Result 2”: Total, ...};

[0178] This way, it becomes clearer how many new issues have been added, how many issues have been fixed, and how many issues remain unresolved.

[0179] For example, the total number of newly emerging problems is 100, the total number of problems that have been fixed is 150, and the total number of unresolved problems is 130.

[0180] The target number of each frame version compared with the problem marking result is obtained, the frame number is arranged in descending order in each problem marking result, the frame number with the largest target number is arranged in the front, and then stored in the json. It can be understood that the statistical result obtained in this way is the fifth statistical result (version comparison sorting result), and the specific form is as follows:

[0181] {“problem marking result 1”:{“frame number 1”:“number 1”,“frame number 2”:“number 2”……},“problem marking result 2”:{“frame number 1”:“number 1”,“frame number 2”:“number 2”……}……};

[0182] For example, the number of unresolved problems in the point cloud frame with frame number 007671 is 47, the number of unresolved problems in the point cloud frame with frame number 006373 is 44, and the number of unresolved problems in the point cloud frame with frame number 005190 is 43.

[0183] It should be noted that the above problem marking result can include problem newly appearing, problem being repaired, and problem being unresolved.

[0184] According to the target number of a certain problem marking result (for example, problem newly appearing), the N point cloud frames can be filtered by using the corresponding preset number threshold or the second preset percentage (the first 20%), to obtain a second target point cloud frame. The second target point cloud frame, the drawing picture of the second target point cloud frame, the detection target set of the second target point cloud frame, and the annotation target set of the second target point cloud frame are stored in the corresponding directory according to the problem marking result. If a new problem appears in the current test version, only the head problem of the newly added problem (problem newly appearing) needs to be checked, and there is no need to find the problem in the whole data set, which greatly improves the efficiency.

[0185] For a certain point cloud frame F1, if the version error information is obtained through version comparison, and the corresponding second drawing picture X1 is obtained by using the version error information for drawing, as Figure 7As shown in the figure, the frame number of the point cloud frame F1 is 000042, the number of target objects with the problem label result of remain in the version error information is 11, the number of target objects with the problem label result of fixed in the version error information is 4, the number of target objects with the problem label result of new in the version error information is 4, the precision of the point cloud frame F1 is 94.78%, the recall is 88.62%, the number of Miss_Err is 9, the number of BGround_Err is 1, the number of Cls&Loc_Err is 2, the number of Loc_Err is 2, the number of Cls_Err is 1, the total number of target objects in the labeled target object set of the point cloud frame F1 (TP_Gt) is 109, and the total number of target objects in the detected target object set of the point cloud frame F1 (TP_Dt) is 109.

[0186] For a point cloud frame F2, if the version error information is obtained through version comparison, the corresponding second drawing picture X2 is obtained by drawing using the version error information, as shown in the figure. Figure 8 As shown in the figure, the frame number of the point cloud frame F2 is 000048, the number of target objects with the problem label result of remain in the version error information is 14, the number of target objects with the problem label result of fixed in the version error information is 4, the number of target objects with the problem label result of new in the version error information is 3, the precision of the point cloud frame F2 is 89.19%, the recall is 86.09%, the number of Miss_Err is 5, the number of BGround_Err is 1, the number of Cls&Loc_Err is 3, the number of Loc_Err is 8, the number of Cls_Err is 0, the total number of target objects in the labeled target object set of the point cloud frame F2 (TP_Gt) is 99, and the total number of target objects in the detected target object set of the point cloud frame F2 (TP_Dt) is 99.

[0187] By the method of the embodiments of the present disclosure, the laser radar target detection result is further classified and filtered, the focusing problem can be found faster, the problem repairing efficiency is accelerated, the product is better improved, another dimension other than the general index is adopted, the problems of the current test version are clearer, the problems can be solved more targetedly, the problems of the current test version can be quickly located, the problems can be accurately classified, the communication cost caused by the fuzzy classification is avoided, the manpower cost is reduced, and the iteration efficiency is improved. The time saving and efficiency improvement of the embodiments of the present disclosure are more obvious when the data amount is large. Originally, 10000 point cloud frames need to be checked manually, and after the method, the number can be reduced by 10 times or more. At present, various manufacturers are also laying out the vehicle-road cooperation roadside sensing system, and the present disclosure can promote the acceleration of the iteration and update of the roadside laser radar algorithm and establish a leading advantage in the field.

[0188] In addition, the classification and filtering method in the scheme of the embodiments of the present disclosure can be applied in the early, middle and late stages of research and development, so that the problems of the current test version can be better and faster found and classified, the problems can be quickly located, the problems of the current test version can be solved and repaired, the rapid iteration can be realized, the product quality of the roadside laser radar is ensured, and the laser radar target detection is more robust.

[0189] As shown in Figure 9 According to the embodiments of the present disclosure, the present disclosure also provides a target detection device 900, the device comprising:

[0190] A target detection module 901 is configured to perform target detection on a to-be-detected frame by using a first target detection algorithm to obtain a detection target set.

[0191] A first determination module 902 is configured to determine a first cost matrix based on the detection target set and a labeled target set of the to-be-detected frame, wherein any element in the first cost matrix is an intersection-over-union between a detection target in the detection target set and a labeled target in the labeled target set.

[0192] A target set determination module 903 is configured to perform problem target extraction and classification on the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the to-be-detected frame, wherein one problem target set corresponds to one problem category, and the at least one problem target set is used to update the first target detection algorithm.

[0193] In one embodiment, as shown in Figure 10 The target set determination module 903 comprises:

[0194] A first matching module 9031 is configured to perform target matching on the detection target set and the labeled target set according to the first cost matrix to obtain a first matching result.

[0195] The problem target determination module 9032 is configured to perform problem target extraction and classification on the detection target set and the labeled target set based on the first matching result and the first cost matrix, to obtain at least one problem target set of the to-be-tested frame.

[0196] As shown in FIG. 9, in one embodiment, the problem target determination module 9032 includes: Figure 11

[0197] The first acquisition module 90321 is configured to acquire a first unmatched target set and a first matched target pair set from the detection target set and the labeled target set according to the first matching result and the first cost matrix, wherein any matched target pair in the first matched target pair set includes a detection target and a matched labeled target, and an intersection-over-union between any two matched target pairs is greater than 0.

[0198] The first addition module 90322 is configured to, for each unmatched target in the first unmatched target set, add the unmatched target into a problem target set of a missed detection category if the unmatched target belongs to the labeled target set, or add the unmatched target into a problem target set of a false detection category if the unmatched target belongs to the detection target set.

[0199] The problem target sub-determination module 90323 is configured to perform problem target extraction and classification on the first matched target pair set based on the first cost matrix and the detection category of the targets in the first matched target pair set, to obtain M problem target sets, where M is a positive integer.

[0200] In one embodiment, performing problem target extraction and classification on the first matched target pair set based on the first cost matrix and the detection category of the targets in the first matched target pair set to obtain M problem target sets includes any of the following:

[0201] In a case where the intersection-over-union between the first matched target pairs in the first cost matrix is greater than or equal to a first threshold value, and the detection category of the detection target and the labeled target in the first matched target pair does not match, the detection target in the first matched target pair is added into a problem target set of a detection category error category, the first matched target pair being any matched target pair in the first matched target pair set.

[0202] In a case where the intersection-over-union between the first matched target pairs in the first cost matrix is less than the first threshold value, and the detection category of the detection target and the labeled target in the first matched target pair matches, the detection target in the first matched target pair is added into a problem target set of a position error category.

[0203] ​In a case that the intersection-over-union between the first matched target pair in the first cost matrix is less than the first threshold value, and the detection class between the detection target and the labeled target in the first matched target pair is not matched, the detection target in the first matched target pair is added to the problem target set of the detection class and position error class.

[0204] In one embodiment, the apparatus 900 further includes:

[0205] The second obtaining module is configured to obtain a second problem target set of the frame to be detected, the second problem target set being a problem target set corresponding to a second target detection algorithm, a version of the first target detection algorithm being different from a version of the second target detection algorithm.

[0206] The second determining module is configured to determine a second cost matrix based on the first problem target set and the second problem target set, any element in the second cost matrix being an intersection-over-union between a problem target in the first problem target set and a problem target in the second problem target set, the first problem target set being a problem target set corresponding to the first target detection algorithm, and the first problem target set including at least one problem target set.

[0207] The marking module is configured to mark problems in the first problem target set and the second problem target set according to the second cost matrix, the marked first problem target set and the marked second problem target set being used for updating the first target detection algorithm.

[0208] In one embodiment, the marking module includes:

[0209] The second matching module is configured to perform target matching on the first problem target set and the second problem target set according to the second cost matrix, to obtain a second matching result.

[0210] The problem marking module is configured to mark problems in the first problem target set and the second problem target set by using the second matching result and the second cost matrix.

[0211] In one embodiment, the problem marking module includes:

[0212] The third obtaining module is configured to obtain a second unmatched target set and a second matched target pair set from the first problem target set and the second problem target set according to the second matching result and the second cost matrix, any matched target pair in the second matched target pair set including a problem target in the first problem target set and a matched problem target in the second problem target set, and the intersection-over-union between any matched target pair in the second matched target pair set being greater than 0.

[0213] The first marking sub-module is configured to mark the unmatched target as a problem solved if the unmatched target belongs to the second problem target set, and mark the unmatched target as a new problem if the unmatched target belongs to the first problem set.

[0214] The second marking sub-module is configured to mark the second matched target pair set based on the second cost matrix.

[0215] In one embodiment, the marking of the second matched target pair set based on the second cost matrix comprises any one of the following:

[0216] In a case where the intersection-union ratio between the second matched target pair in the second cost matrix is greater than or equal to a second threshold, marking the problem target in the second matched target pair as a problem unsolved, the second matched target pair being any matched target pair in the second matched target pair set;

[0217] In a case where the intersection-union ratio between the second matched target pair in the second cost matrix is less than the second threshold, marking the first problem target in the second matched target pair as a new problem, and marking the second problem target in the second matched target pair as a problem solved, the first problem target belonging to the first problem target set, and the second problem target belonging to the second problem target set.

[0218] In one embodiment, the frame to be tested comprises N point cloud frames, N being a positive integer, any frame in the N point cloud frames corresponding to at least one problem target set;

[0219] The apparatus further comprises:

[0220] The first statistical module is configured to count the problem target sets of the N point cloud frames to obtain a target statistical result.

[0221] The target statistical result comprises at least one of the following:

[0222] The first statistical result comprises N first sub-statistical results, and any first sub-statistical result comprises a frame number and a problem target in a point cloud frame corresponding to the frame number.

[0223] The second statistical result comprises K second sub-statistical results, and any second sub-statistical result comprises a problem category and a total number of problem targets corresponding to the problem category, K being a positive integer.

[0224] The third statistical result comprises L third sub-statistical results, and any third sub-statistical result comprises a problem category, a frame number of each point cloud frame in the N point cloud frames, and a number of problem targets belonging to the problem category in a point cloud frame corresponding to each point cloud frame number, L being a positive integer.

[0225] In an embodiment, the target statistical result includes a third statistical result;

[0226] The device further includes:

[0227] The first filtering module is configured to filter the N point cloud frames according to at least one of a preset number of problems in a reference problem category and a first preset percentage and a target problem number of each point cloud frame in the N point cloud frames under the reference problem category, to obtain first target point cloud frames, the reference problem category being any one of problem categories in the third statistical result.

[0228] The first storage module is configured to store the first target point cloud frames, the drawing pictures of the first target point cloud frames, the detection target sets of the first target point cloud frames, and the labeled target sets of the first target point cloud frames.

[0229] In an embodiment, the frame to be tested includes N point cloud frames, N being a positive integer, and any frame in the N point cloud frames corresponding to at least one problem target set;

[0230] The device further includes:

[0231] The second statistical module is configured to count the labeled first problem target sets of the N point cloud frames and the labeled second problem target sets of the N point cloud frames, to obtain a reference statistical result, the reference statistical result including at least one of:

[0232] The fourth statistical result includes R fourth sub-statistical results, any one of the fourth sub-statistical results including a problem label result and a total number of targets corresponding to the problem label result in the N point cloud frames, R being a positive integer.

[0233] The fifth statistical result includes S fifth sub-statistical results, any one of the fifth sub-statistical results including a problem label result, a frame number of each point cloud frame in the N point cloud frames, and a number of targets labeled as the problem label result in the point cloud frame corresponding to each point cloud frame number, S being a positive integer.

[0234] In an embodiment, the reference statistical result includes the fifth statistical result;

[0235] The device further includes:

[0236] The second filtering module is configured to filter the N point cloud frames according to at least one of a preset number threshold of candidate problem label results and a second preset percentage and a target number of each point cloud frame in the N point cloud frames under the candidate problem label result, to obtain second target point cloud frames, the candidate problem label result being any one of the problem label results in the fifth statistical result.

[0237] The second storage module is configured to store the second target point cloud frame, the drawing picture of the second target point cloud frame, the detection target set of the second target point cloud frame, and the labeled target set of the second target point cloud frame.

[0238] The target detection device of each of the above embodiments is a device for implementing the target detection method of each of the above embodiments, and the technical features and technical effects are corresponding, which will not be repeated here.

[0239] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0240] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0241] The non-transitory computer readable storage medium of the embodiments of the present disclosure stores computer instructions for causing a computer to execute the target detection method provided by the present disclosure.

[0242] The computer program product of the embodiments of the present disclosure includes a computer program, and the computer program is used to make a computer execute the target detection method provided by the embodiments of the present disclosure.

[0243] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0244] As shown in Figure 12 The electronic device 1200 includes a computing unit 1201 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded into a random access memory (RAM) 1203 from a storage unit 1208. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0245] A plurality of components in the electronic device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a magnetic disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0246] The computing unit 1201 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs various methods and processes described above, such as the object detection method. For example, in some embodiments, the object detection method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded onto the RAM 1203 and executed by the computing unit 1201, one or more steps of the object detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform the object detection method by any other appropriate means, such as by means of firmware. The various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0247] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0248] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0249] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0250] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0251] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0252] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions disclosed in the present disclosure, and are not limited herein.

[0253] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, the method comprising: performing target detection on a to-be-detected frame by a first target detection algorithm to obtain a detection target set; determining a first cost matrix based on the detection target set and a labeled target set of the to-be-detected frame, any element in the first cost matrix being an intersection over union between a detection target in the detection target set and a labeled target in the labeled target set; performing problem target extraction and classification on the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the to-be-detected frame, wherein one problem target set corresponds to one problem category, and the at least one problem target set is used for updating the first target detection algorithm; obtaining a second problem target set of the to-be-detected frame, the second problem target set being a problem target set corresponding to a second target detection algorithm, a version of the first target detection algorithm being different from a version of the second target detection algorithm; determining a second cost matrix based on the first problem target set and the second problem target set, any element in the second cost matrix being an intersection over union between a problem target in the first problem target set and a problem target in the second problem target set, the first problem target set being a problem target set corresponding to the first target detection algorithm and including the at least one problem target set; performing problem labeling on the first problem target set and the second problem target set according to the second cost matrix, the labeled first problem target set and the labeled second problem target set being used for updating the first target detection algorithm.

2. The method of claim 1, wherein, The performing problem target extraction and classification on the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the to-be-detected frame comprises: performing target matching on the detection target set and the labeled target set according to the first cost matrix to obtain a first matching result; performing problem target extraction and classification on the detection target set and the labeled target set based on the first matching result and the first cost matrix to obtain at least one problem target set of the to-be-detected frame.

3. The method of claim 2, wherein, The performing problem target extraction and classification on the detection target set and the labeled target set based on the first matching result and the first cost matrix to obtain at least one problem target set of the to-be-detected frame comprises: obtaining a first unmatched target set and a first matched target pair set from the detection target set and the labeled target set according to the first matching result and the first cost matrix, wherein any matched target pair in the first matched target pair set includes a detection target and a matched labeled target, and an intersection over union between any matched target pair is greater than 0. adding the unmatched target to a problem target set of a missed detection category if the unmatched target belongs to the labeled target set, or adding the unmatched target to a problem target set of a false detection category if the unmatched target belongs to the detected target set; performing problem target extraction and classification on the first matched target pair set based on the first cost matrix and the detection categories of the targets in the first matched target pair set, to obtain M problem target sets, where M is a positive integer.

4. The method of claim 3, wherein, The performing problem target extraction and classification on the first matched target pair set based on the first cost matrix and the detection categories of the targets in the first matched target pair set, to obtain M problem target sets, includes any of the following: adding the detection target in the first matched target pair to a problem target set of a detection category error category if the intersection-over-union of the first matched target pair in the first cost matrix is greater than or equal to a first threshold value, and the detection categories of the detection target and the labeled target in the first matched target pair do not match, the first matched target pair being any matched target pair in the first matched target pair set; adding the detection target in the first matched target pair to a problem target set of a position error category if the intersection-over-union of the first matched target pair in the first cost matrix is less than the first threshold value, and the detection categories of the detection target and the labeled target in the first matched target pair match; adding the detection target in the first matched target pair to a problem target set of a detection category and position error category if the intersection-over-union of the first matched target pair in the first cost matrix is less than the first threshold value, and the detection categories of the detection target and the labeled target in the first matched target pair do not match.

5. The method of claim 1, wherein, The performing problem labeling on the first problem target set and the second problem target set according to the second cost matrix includes: performing target matching on the first problem target set and the second problem target set according to the second cost matrix, to obtain a second matching result; performing problem labeling on the first problem target set and the second problem target set by using the second matching result and the second cost matrix.

6. The method of claim 5, wherein, The performing problem labeling on the first problem target set and the second problem target set by using the second matching result and the second cost matrix includes: obtaining a second unmatched target set and a second matched target pair set from the first problem target set and the second problem target set according to the second matching result and the second cost matrix, where any matched target pair in the second matched target pair set includes a problem target in the first problem target set and a matched problem target in the second problem target set, and the intersection-over-union of any matched target pair in the second matched target pair set is greater than 0. For each unmatched target in the second unmatched target set, if the unmatched target belongs to the second problem target set, marking the unmatched target as a problem fixed, if the unmatched target belongs to the first problem set, marking the unmatched target as a problem newly appeared; Based on the second cost matrix, performing problem marking on the second matched target pair set.

7. The method of claim 6, wherein, The problem marking on the second matched target pair set based on the second cost matrix comprises any one of the following: If the intersection-union ratio between the second matched target pair in the second cost matrix is greater than or equal to a second threshold, marking the problem target in the second matched target pair as a problem not solved, the second matched target pair being any matched target pair in the second matched target pair set; If the intersection-union ratio between the second matched target pair in the second cost matrix is less than a second threshold, marking the first problem target in the second matched target pair as a problem newly appeared, and marking the second problem target in the second matched target pair as a problem fixed, the first problem target belonging to the first problem target set, and the second problem target belonging to the second problem target set.

8. The method of claim 1, wherein, The frame to be tested comprises N point cloud frames, N being a positive integer, and any frame in the N point cloud frames corresponding to at least one problem target set; The method further comprises: counting the problem target sets of the N point cloud frames to obtain a target statistical result; The target statistical result comprises at least one of the following: a first statistical result, the first statistical result comprising N first sub-statistical results, any first sub-statistical result comprising a frame number and a problem target in a point cloud frame corresponding to the frame number; a second statistical result, the second statistical result comprising K second sub-statistical results, any second sub-statistical result comprising a problem category and a total number of problem targets corresponding to the problem category, K being a positive integer; a third statistical result, the third statistical result comprising L third sub-statistical results, any third sub-statistical result comprising a problem category, a frame number of each point cloud frame in the N point cloud frames, and a number of problem targets belonging to the problem category in each point cloud frame corresponding to the frame number, L being a positive integer.

9. The method of claim 8, wherein, The target statistical result comprises the third statistical result; After the counting of the problem target sets of the N point cloud frames to obtain the target statistical result, the method further comprises: filtering the N point cloud frames by using at least one of a preset problem number of a reference problem category and a first preset percentage, and a number of problem targets of each point cloud frame in the N point cloud frames under the reference problem category, to obtain a first target point cloud frame, the reference problem category being any category in the problem categories in the third statistical result; storing the first target point cloud frame, a drawing picture of the first target point cloud frame, a detection target set of the first target point cloud frame, and a labeled target set of the first target point cloud frame.

10. The method of claim 1, wherein, The frame to be tested comprises N point cloud frames, N being a positive integer, and any frame in the N point cloud frames corresponding to at least one problem target set; The method further comprises: The method further comprises: The method further comprises:

12. A target detection device, the device comprising:

11. The method of claim 10, wherein, a target detection module configured to perform target detection on a frame to be detected by using a first target detection algorithm to obtain a detection target set; a first determination module configured to determine a first cost matrix based on the detection target set and a labeled target set of the frame to be detected, wherein any element in the first cost matrix is an intersection over union between a detection target in the detection target set and a labeled target in the labeled target set; a target set determination module configured to perform problem target extraction and classification on the detection target set and the labeled target set according to the first cost matrix to obtain at least one problem target set of the frame to be detected, wherein one problem target set corresponds to one problem category, and the at least one problem target set is used to update the first target detection algorithm; the device further comprises: a second acquisition module configured to acquire a second problem target set of the frame to be detected, wherein the second problem target set is a problem target set corresponding to a second target detection algorithm, and a version of the first target detection algorithm is different from a version of the second target detection algorithm. ​ ​ ​ ​ ​ The second determining module is configured to determine a second cost matrix based on the first problem target set and the second problem target set, wherein any element in the second cost matrix is an intersection-union ratio between a problem target in the first problem target set and a problem target in the second problem target set, and the first problem target set is a problem target set corresponding to the first target detection algorithm and includes the at least one problem target set; The marking module is configured to mark the first problem target set and the second problem target set according to the second cost matrix, and the marked first problem target set and the marked second problem target set are used for updating the first target detection algorithm.

13. The apparatus of claim 12, wherein, The target set determining module includes: The first matching module is configured to perform target matching on the detection target set and the annotation target set according to the first cost matrix to obtain a first matching result. The problem target determining module is configured to perform problem target extraction and classification on the detection target set and the annotation target set based on the first matching result and the first cost matrix to obtain at least one problem target set of the to-be-detected frame.

14. The apparatus of claim 13, wherein, The problem target determining module includes: The first obtaining module is configured to obtain a first unmatched target set and a first matched target pair set from the detection target set and the annotation target set according to the first matching result and the first cost matrix, wherein any matched target pair in the first matched target pair set includes a detection target and a matched annotation target, and an intersection-union ratio between any matched target pair is greater than 0. The first adding module is configured to, for each unmatched target in the first unmatched target set, add the unmatched target to a problem target set of a missed detection category if the unmatched target belongs to the annotation target set, or add the unmatched target to a problem target set of a false detection category if the unmatched target belongs to the detection target set. The problem target sub-determining module is configured to perform problem target extraction and classification on the first matched target pair set based on the first cost matrix and a detection category of a target in the first matched target pair set to obtain M problem target sets, where M is a positive integer.

15. The apparatus of claim 14, wherein, The problem target extraction and classification on the first matched target pair set based on the first cost matrix and the detection category of the target in the first matched target pair set to obtain M problem target sets include any of the following: In a case where an intersection-union ratio between a first matched target pair in the first cost matrix is greater than or equal to a first threshold value and a detection category of a detection target in the first matched target pair does not match a detection category of an annotation target in the first matched target pair, the detection target in the first matched target pair is added to a problem target set of a detection category error category, and the first matched target pair is any matched target pair in the first matched target pair set. add the detection target in the first matched target pair to a problem target set of a position error category, in a case that the intersection over union between the first matched target pair is less than a first threshold value and a detection category match between the detection target in the first matched target pair and the labeled target in the first cost matrix; add the detection target in the first matched target pair to a problem target set of a detection category and position error category, in a case that the intersection over union between the first matched target pair is less than a first threshold value and a detection category mismatch between the detection target in the first matched target pair and the labeled target in the first cost matrix.

16. The apparatus of claim 12, wherein, The marking module comprises: The second matching module is configured to perform target matching on the first problem target set and the second problem target set according to the second cost matrix, to obtain a second matching result. The problem marking module is configured to perform problem marking on the first problem target set and the second problem target set by using the second matching result and the second cost matrix.

17. The apparatus of claim 16, wherein, The problem marking module comprises: The third obtaining module is configured to obtain a second unmatched target set and a second matched target pair set from the first problem target set and the second problem target set according to the second matching result and the second cost matrix, wherein any matched target pair in the second matched target pair set comprises a problem target in the first problem target set and a matched problem target in the second problem target set, and the intersection over union between any matched target pair in the second matched target pair set is greater than 0. The first marking submodule is configured to, for each unmatched target in the second unmatched target set, mark the unmatched target as a problem newly appearing in a case that the unmatched target belongs to the first problem target set, and mark the unmatched target as a problem repaired in a case that the unmatched target belongs to the second problem target set. The second marking submodule is configured to perform problem marking on the second matched target pair set based on the second cost matrix.

18. The apparatus of claim 17, wherein, The problem marking on the second matched target pair set based on the second cost matrix comprises any of the following: mark a problem target in the second matched target pair as a problem unsolved in a case that the intersection over union between the second matched target pair is greater than or equal to a second threshold value, the second matched target pair being any matched target pair in the second matched target pair set; mark a first problem target in the second matched target pair as a problem newly appearing and mark a second problem target in the second matched target pair as a problem repaired in a case that the intersection over union between the second matched target pair is less than a second threshold value, the first problem target belonging to the first problem target set, and the second problem target belonging to the second problem target set.

19. The apparatus of claim 12, wherein, The to-be-tested frame comprises N point cloud frames, N being a positive integer, and any frame in the N point cloud frames corresponding to at least one problem target set; The device further comprises: The first statistical module is configured to count the problem target set of the N point cloud frames to obtain a target statistical result. The target statistical result includes at least one of the following: The first statistical result includes N first sub-statistical results, and any first sub-statistical result includes a frame number and a problem target in a point cloud frame corresponding to the frame number. The second statistical result includes K second sub-statistical results, and any second sub-statistical result includes a problem category and a total number of problem targets corresponding to the problem category, K being a positive integer. The third statistical result includes L third sub-statistical results, and any third sub-statistical result includes a problem category, a frame number of each point cloud frame in the N point cloud frames, and a number of problem targets belonging to the problem category in each point cloud frame corresponding to the frame number, L being a positive integer.

20. The apparatus of claim 19, wherein, The target statistical result includes the third statistical result. The device further includes: The first filtering module is configured to filter the N point cloud frames by using at least one of a preset number of problems of a reference problem category and a first preset percentage and a number of problem targets of each point cloud frame in the N point cloud frames under the reference problem category to obtain first target point cloud frames, the reference problem category being any category in the problem categories of the third statistical result. The first storage module is configured to store the first target point cloud frames, a drawing picture of the first target point cloud frames, a detection target set of the first target point cloud frames, and a labeled target set of the first target point cloud frames.

21. The apparatus of claim 12, wherein, The N point cloud frames include N point cloud frames, N being a positive integer, and any frame of the N point cloud frames corresponding to at least one problem target set. The device further includes: The second statistical module is configured to count the first problem target set and the second problem target set of the N point cloud frames after labeling to obtain a reference statistical result, the reference statistical result including at least one of the following: The fourth statistical result includes R fourth sub-statistical results, and any fourth sub-statistical result includes a problem labeling result and a total number of targets corresponding to the problem labeling result in the N point cloud frames, R being a positive integer. The fifth statistical result includes S fifth sub-statistical results, and any fifth sub-statistical result includes a problem labeling result, a frame number of each point cloud frame in the N point cloud frames, and a number of targets corresponding to each point cloud frame number in the point cloud frame for the problem labeling result, S being a positive integer.

22. The apparatus of claim 21, wherein, The reference statistical result includes the fifth statistical result. The device further includes: The second filtering module is configured to filter the N point cloud frames by using at least one of a preset number threshold of candidate problem labeling results and a second preset percentage and a number of targets of each point cloud frame in the N point cloud frames under the candidate problem labeling result to obtain second target point cloud frames, the candidate problem labeling result being any labeling result in the problem labeling results of the fifth statistical result. A second storage module, configured to store the second target point cloud frame, the drawing picture of the second target point cloud frame, the detection target set of the second target point cloud frame, and the labeled target set of the second target point cloud frame. 23.An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the target detection method according to any one of claims 1-11. 24.A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the target detection method according to any one of claims 1-11. 25.A computer program product comprising a computer program which, when executed by a processor, implements the target detection method according to any one of claims 1-11.

Images