Method and system for evaluating three-dimensional object detection algorithm, electronic device and medium

Abstract

The present application relates to the technical field of algorithm evaluation, and discloses a three-dimensional target detection algorithm evaluation method, system, electronic device and medium, the method obtains algorithm detection data corresponding to standard detection data through a to-be-evaluated algorithm, establishes a plurality of view coordinate systems, calculates the data overlap between the algorithm detection data and the standard detection data in different coordinate systems, determines the coordinate system accuracy of each coordinate system based on the data overlap, and then weightedly calculates the coordinate system accuracy of each view coordinate system to obtain an evaluation index. Different data overlaps between the standard detection data and the algorithm detection data are obtained through different view coordinate systems, the coordinate system accuracy under different view coordinate systems is determined, and finally the evaluation index is obtained through weighted calculation based on the coordinate system accuracy, which increases the range of evaluation parameters and improves the accuracy of the evaluation result.

Description

Technical Field

[0001] This invention relates to the field of algorithm evaluation technology, and in particular to an evaluation method, system, electronic device and medium for a three-dimensional target detection algorithm. Background Technology

[0002] Currently, with the continuous advancement of autonomous driving, object detection has become a highly focused area in the field, as the accuracy of object detection algorithms directly impacts the safety of autonomous vehicles. With the widespread use of LiDAR (Light Detection and Ranging) systems, using point cloud data collected by LiDAR to perform 3D object recognition, obtaining high-precision, high-depth information, and thus assisting autonomous driving has become a mainstream trend. Therefore, accurately evaluating a 3D object detection algorithm and using the evaluation results to guide its improvement has become a crucial prerequisite.

[0003] Existing algorithm evaluation methods typically involve manually annotating targets in point cloud data to obtain manually annotated bounding boxes, and then using a 3D object detection algorithm to detect targets in the point cloud data to obtain algorithmically annotated bounding boxes. If the overlap between the manually annotated bounding boxes and the algorithmically annotated bounding boxes is large, the evaluation result of the 3D object detection algorithm is considered satisfactory. However, evaluating the results of 3D object detection algorithms based solely on a single view involves overly simplistic evaluation parameters, resulting in only satisfactory or unsatisfactory evaluation results. This leads to low accuracy and fails to provide sufficient reference for improving 3D object detection algorithms. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, the present invention discloses an evaluation method, system, electronic device and medium for three-dimensional target detection algorithms, so as to improve the evaluation accuracy of three-dimensional target detection algorithms.

[0006] This invention discloses an evaluation method for a three-dimensional target detection algorithm, comprising: acquiring an algorithm to be evaluated, and acquiring multiple sample data and standard detection data corresponding to each sample data, wherein the algorithm to be evaluated is used to perform target detection on each sample data to obtain algorithm detection data corresponding to each sample data; establishing multiple view coordinate systems corresponding to preset view categories, determining any view coordinate system as the target coordinate system, and determining the data overlap between the algorithm detection data and the standard detection data according to the target coordinate system; determining positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and a preset overlap threshold, and determining the coordinate system accuracy of the target coordinate system based on the number of positive examples of sample recognition; and performing a weighted calculation of the coordinate system accuracy of each view coordinate system based on preset view weights to obtain an evaluation index for the algorithm to be evaluated.

[0007] Optionally, after determining any view coordinate system as the target coordinate system, before determining positive samples from the algorithm detection data based on the comparison result between the data overlap and the preset overlap threshold, the method further includes: acquiring multiple distance intervals; determining the center point of each of the algorithm detection data according to the target coordinate system, and determining the Euclidean distance between the center point and the origin of the target coordinate system as the center point distance; and determining the classification detection data corresponding to each distance interval from the algorithm detection data based on the matching result between each distance interval and each center point distance.

[0008] Optionally, after acquiring multiple sample data and standard detection data corresponding to each sample data, and before establishing view coordinate systems corresponding to multiple preset view categories, the method further includes: acquiring the standard target category and standard target orientation corresponding to the standard detection data; and acquiring at least one of the detection target category, detection target orientation, and detection confidence level corresponding to the algorithm detection data.

[0009] Optionally, determining positive examples of sample recognition for each view coordinate system from the algorithm detection data based on the comparison result between the data overlap and a preset overlap threshold includes: the number of overlap thresholds includes multiple ones; determining any overlap threshold as a target threshold, and pre-setting a first positive example condition, a second positive example condition, a third positive example condition, and a fourth positive example condition, wherein the first positive example condition includes that the classification detection data of the classification detection data is greater than or equal to the target threshold, the second positive example condition includes that the detection confidence of the classification detection data is greater than or equal to a preset confidence threshold, the third positive example condition includes that the detection target category of the classification detection data is the same as the corresponding standard target category, and the fourth positive example condition includes that the detection target orientation of the classification detection data is the same as the corresponding standard target orientation; determining at least a portion of the first positive example condition, the second positive example condition, the third positive example condition, and the fourth positive example condition as judgment conditions; if any classification detection data satisfies the judgment conditions, then the classification detection data is determined as a positive example of sample recognition for the target threshold.

[0010] Optionally, determining the coordinate system precision of the target coordinate system based on the number of positive examples identified by the samples includes: defining any distance interval as the target interval; performing quantity statistics on the classification detection data corresponding to the target interval to obtain the total amount of classification data corresponding to the target interval; performing quantity statistics on the positive examples identified by each overlap threshold to obtain the number of threshold positive examples for each overlap threshold; calculating the number of threshold positive examples for each overlap threshold based on the total amount of classification data to obtain the threshold precision of each overlap threshold; performing weighted calculation on the threshold precision of each overlap threshold based on the preset threshold weights corresponding to each overlap threshold to obtain the interval precision of the target interval; and performing weighted calculation on the interval precision of each distance interval based on the preset interval weights corresponding to each distance interval to obtain the coordinate system precision of the target coordinate system.

[0011] Optionally, the evaluation metric of the algorithm to be evaluated is determined by the following formula: In the formula, AP all AP is the evaluation metric for the algorithm to be evaluated. view(k) Let θ be the coordinate system accuracy of the k-th view coordinate system. view(k) Here, s represents the preset view weight for the k-th view coordinate system, and s represents the number of view coordinate systems. dis(j) Let β be the interval precision for the j-th distance interval. dis(j) Here, m represents the preset interval weight for the j-th distance interval, and m is the number of distance intervals. AP thr(i) Let α be the threshold accuracy for the i-th overlap threshold. thr(i)is the preset threshold weight for the i-th overlap threshold, and n is the number of overlap thresholds.

[0012] Optionally, determining the data overlap between the algorithm detection data and the standard detection data according to the target coordinate system includes: if the preset view category corresponding to the preset view category includes a 3D view, then the intersection of the algorithm detection data and the standard detection data is determined as a first coordinate point; the vertex of the algorithm detection data is determined as a first vertex, and the first vertex contained in the standard detection data is determined as a second coordinate point; the vertex of the standard detection data is determined as a second vertex, and the second vertex contained in the algorithm detection data is determined as a third coordinate point; triangulation calculation is performed on the first coordinate point, the second coordinate point, and the third coordinate point according to a preset triangulation algorithm to obtain a reference region; the data overlap between the algorithm detection data and the standard detection data is calculated based on the volume of the algorithm detection data, the volume of the standard detection data, and the volume of the reference region.

[0013] Optionally, the data overlap between the algorithm-detected data and the standard detection data is determined using the following formula: In the formula, IoU is the data overlap between the algorithm-detected data and the standard detection data, V1 is the volume of the algorithm-detected data, V2 is the volume of the standard detection data, and V I Let be the volume of the reference region.

[0014] This invention discloses an evaluation system for a three-dimensional target detection algorithm, comprising: an acquisition module for acquiring an algorithm to be evaluated, and acquiring multiple sample data and standard detection data corresponding to each sample data, wherein the algorithm to be evaluated is used to perform target detection on each sample data to obtain algorithm detection data corresponding to each sample data; an overlap determination module for establishing multiple view coordinate systems corresponding to preset view categories, determining any view coordinate system as the target coordinate system, and determining the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system; an accuracy determination module for determining positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and a preset overlap threshold, and determining the coordinate system accuracy of the target coordinate system based on the number of positive examples of sample recognition; and a calculation module for performing weighted calculation of the coordinate system accuracy of each view coordinate system based on preset view weights of each view coordinate system to obtain the evaluation index of the algorithm to be evaluated.

[0015] The present invention discloses an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the above-described method.

[0016] The present invention discloses a computer-readable storage medium having a computer program stored thereon: when the computer program is executed by a processor, it implements the above-described method.

[0017] The beneficial effects of this invention are:

[0018] By obtaining the algorithm detection data corresponding to the standard detection data from the algorithm to be evaluated, and establishing multiple view coordinate systems, the data overlap between the algorithm detection data and the standard detection data in different coordinate systems is calculated. Based on the data overlap, positive examples of sample recognition in the algorithm detection data under different coordinate systems are selected. Then, the coordinate system accuracy of each coordinate system is determined according to the number of positive examples. Finally, the coordinate system accuracy of each view coordinate system is weighted and calculated to obtain the evaluation index. In this way, compared with directly determining the evaluation result of the detection algorithm through the overlap of a single view, obtaining different data overlap between the standard detection data and the algorithm detection data through different view coordinate systems, and then determining the coordinate system accuracy under different view coordinate systems, and finally calculating the evaluation index based on the weighted coordinate system accuracy, the range of evaluation parameters is increased, thereby improving the accuracy of the evaluation results. Furthermore, the evaluation index is more suitable for judging the differences between different detection algorithms, thus providing a reference for the improvement of 3D target detection algorithms.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0021] Figure 1 This is a flowchart illustrating an evaluation method for a three-dimensional target detection algorithm in an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating an evaluation method for another three-dimensional target detection algorithm in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of an evaluation system for a three-dimensional target detection algorithm in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0025] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and sub-samples in the embodiments can be combined with each other.

[0026] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0027] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0029] Unless otherwise stated, the term "multiple" means two or more.

[0030] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0031] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0032] Combination Figure 1 As shown, this disclosure provides an evaluation method for a three-dimensional target detection algorithm, including:

[0033] Step S101: Obtain the algorithm to be evaluated, and obtain multiple sample data and the standard detection data corresponding to each sample data;

[0034] Among them, the algorithm to be evaluated is used to perform target detection on each sample data to obtain the algorithm detection data corresponding to each sample data;

[0035] Step S102: Establish multiple view coordinate systems corresponding to preset view categories, determine any view coordinate system as the target coordinate system, and determine the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system;

[0036] Step S103: Determine positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap degree and the preset overlap degree threshold, and determine the coordinate system accuracy of the target coordinate system based on the number of positive examples of sample recognition.

[0037] Step S104: The coordinate system accuracy of each view coordinate system is calculated by weighting the preset view weights of each view coordinate system to obtain the evaluation index of the algorithm to be evaluated.

[0038] The evaluation method for 3D target detection algorithms provided in this disclosure involves obtaining algorithm detection data corresponding to standard detection data from the algorithm to be evaluated, establishing multiple view coordinate systems, calculating the data overlap between the algorithm detection data and the standard detection data in different coordinate systems, filtering positive examples of sample recognition from the algorithm detection data in different coordinate systems based on the data overlap, determining the coordinate system accuracy of each coordinate system based on the number of positive examples, and then weighting the coordinate system accuracy of each view coordinate system to obtain the evaluation index. This approach, compared to directly determining the evaluation result of the detection algorithm through the overlap of a single view, increases the range of evaluation parameters by obtaining different data overlaps between the standard detection data and the algorithm detection data through different view coordinate systems, thereby determining the coordinate system accuracy in different view coordinate systems, and finally weighting the coordinate system accuracy to obtain the evaluation index. This improves the accuracy of the evaluation results and makes the evaluation index more suitable for judging the differences between different detection algorithms, thus providing a reference for improving 3D target detection algorithms.

[0039] Optionally, the preset view categories include two-dimensional views and three-dimensional views; the view coordinate system corresponding to the two-dimensional view includes the BEV bird's-eye view, which can be established through high-precision map (offline mapping) or through online mapping for view semantic segmentation and detection; the view coordinate system corresponding to the three-dimensional view includes a Cartesian coordinate system with the vehicle as the origin and including three coordinate axes.

[0040] Optionally, the preset view weight corresponding to the two-dimensional view is greater than the preset view weight corresponding to the three-dimensional view.

[0041] Optionally, determining the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system includes: if the preset view category corresponding to the preset view category includes a 3D view, then the intersection of the algorithm detection data and the standard detection data is determined as the first coordinate point; the vertex of the algorithm detection data is determined as the first vertex, and the first vertex contained in the standard detection data is determined as the second coordinate point; the vertex of the standard detection data is determined as the second vertex, and the second vertex contained in the algorithm detection data is determined as the third coordinate point; the first coordinate point, the second coordinate point, and the third coordinate point are triangulated according to a preset triangulation algorithm to obtain a reference region; and the data overlap between the algorithm detection data and the standard detection data is calculated based on the volume of the algorithm detection data, the volume of the standard detection data, and the volume of the reference region.

[0042] Optionally, triangulation algorithms are often used in surveying, including when observing the same three-dimensional point P(x,y,z) from different locations, and knowing the two-dimensional projection points X1(x1,y1) and X2(x2,y2) of the three-dimensional point observed at different locations, the depth information z of the three-dimensional point can be recovered using triangulation relationships.

[0043] Optionally, data overlap, or IoU (Intersection over Union), is a standard for measuring the accuracy of detecting corresponding objects in a specific dataset. Data overlap can be used to determine the positive samples (TP, true positive) and negative samples (FP, false positive) in the data detected by the algorithm, and then the accuracy (AP, Accuracy Precision) can be calculated using the precision formula AP = TP / (TP+FP).

[0044] Optionally, the data overlap between the algorithm-detected data and the standard detection data can be determined using the following formula:

[0045]

[0046] In the formula, IoU represents the data overlap between the algorithm detection data and the standard detection data, V1 represents the volume of the algorithm detection data, V2 represents the volume of the standard detection data, and V... I The volume of the reference region.

[0047] Optionally, after determining any view coordinate system as the target coordinate system, and before determining positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and the preset overlap threshold, the method further includes: obtaining multiple distance intervals; determining the center point of each algorithm detection data according to the target coordinate system, and determining the Euclidean distance between the center point and the origin of the target coordinate system as the center point distance; and determining the classification detection data corresponding to each distance interval from the algorithm detection data based on the matching result between each distance interval and each center point distance.

[0048] In some embodiments, the distance intervals include five types: 0-30m, 30-100m, 100-200m, 200-∞m, and 0-∞m. Meanwhile, the preset threshold values ​​of the distance intervals are negatively correlated with the values ​​of the distance intervals.

[0049] Optionally, after acquiring multiple sample data and the standard detection data corresponding to each sample data, and before establishing the view coordinate system corresponding to multiple preset view categories, the method further includes: acquiring the standard target category and standard target orientation corresponding to the standard detection data; and acquiring at least one of the detection target category, detection target orientation, and detection confidence corresponding to the algorithm detection data.

[0050] Optionally, based on the comparison result between the data overlap and a preset overlap threshold, positive examples for sample recognition corresponding to each view coordinate system are determined from the algorithm detection data. This includes: the number of overlap thresholds includes multiple thresholds; any overlap threshold is determined as the target threshold; a first positive example condition, a second positive example condition, a third positive example condition, and a fourth positive example condition are preset, wherein the first positive example condition includes that the classification detection data of the classification detection data is greater than or equal to the target threshold; the second positive example condition includes that the detection confidence of the classification detection data is greater than or equal to a preset confidence threshold; the third positive example condition includes that the detection target category of the classification detection data is the same as the corresponding standard target category; and the fourth positive example condition includes that the detection target orientation of the classification detection data is the same as the corresponding standard target orientation; at least a portion of the first, second, third, and fourth positive example conditions are determined as judgment conditions; if any classification detection data satisfies the judgment conditions, the classification detection data is determined as a positive example for sample recognition of the target threshold.

[0051] In some embodiments, the overlap threshold includes three values: 0.3, 0.5, and 0.7. Furthermore, the values ​​of the overlap thresholds are positively correlated with the corresponding preset threshold weights.

[0052] Optionally, determining the coordinate system precision of the target coordinate system based on the number of positive examples identified by the samples includes: defining any distance interval as the target interval; counting the number of classification detection data corresponding to the target interval to obtain the total amount of classification data corresponding to the target interval; counting the number of positive examples identified by the samples for each overlap threshold to obtain the number of threshold positive examples for each overlap threshold; calculating the number of threshold positive examples for each overlap threshold based on the total amount of classification data to obtain the threshold precision of each overlap threshold; weighting the threshold precision of each threshold according to the preset threshold weights corresponding to each overlap threshold to obtain the interval precision of the target interval; and weighting the interval precision of each distance interval according to the preset interval weights corresponding to each distance interval to obtain the coordinate system precision of the target coordinate system.

[0053] Optionally, the evaluation metrics for the algorithm to be evaluated can be determined using the following formula:

[0054]

[0055] In the formula, AP all AP is the evaluation metric for the algorithm to be evaluated. view(k) Let θ be the coordinate system accuracy of the k-th view coordinate system. view(k) Here, s represents the preset view weight for the k-th view coordinate system, and s represents the number of view coordinate systems. dis(j) Let β be the interval precision for the j-th distance interval. dis(j) Here, m represents the preset interval weight for the j-th distance interval, and m is the number of distance intervals. AP thr(i) Let α be the threshold accuracy for the i-th overlap threshold. thr(i) is the preset threshold weight for the i-th overlap threshold, and n is the number of overlap thresholds.

[0056] Combination Figure 2 As shown, this disclosure provides an evaluation method for a three-dimensional target detection algorithm, including:

[0057] Step S201: Obtain multiple data sets;

[0058] The data set includes sample data, standard detection data corresponding to the sample data, and algorithm detection data.

[0059] Among them, the algorithm to be evaluated is used to perform target detection on each sample data to obtain the algorithm detection data corresponding to each sample data;

[0060] Step S202: Establish multiple preset view coordinate systems corresponding to view categories, and determine each data group as the coordinate system data group corresponding to the view coordinate system;

[0061] Among them, the number of view coordinate systems A includes two, and the coordinate system data groups include coordinate system data group A1 and coordinate system group A2;

[0062] Step S203: Based on the matching results between each distance interval and each center point distance, determine the interval data group corresponding to each distance interval from each coordinate system data group;

[0063] Among them, the number of distance intervals B includes 3, and the interval data groups include interval data group B1, interval data group B2 and interval data group B3;

[0064] Step S204: Determine the data overlap between the algorithm detection data and the standard detection data in the corresponding view coordinate system;

[0065] Step S205: Determine the positive examples of sample recognition corresponding to each overlap threshold;

[0066] Among them, the overlap threshold C includes overlap threshold C1, overlap threshold C2 and overlap threshold C3;

[0067] Step S206: Determine the threshold accuracy corresponding to each overlap threshold;

[0068] Step S207: The threshold accuracy corresponding to each overlap threshold is weighted according to the preset weight to obtain the interval accuracy of the distance interval;

[0069] Step S208: The interval accuracy of each distance interval is calculated by weighting according to the preset weights to obtain the coordinate system accuracy of the view coordinate system.

[0070] Step S209: The coordinate system accuracy of each view coordinate system is weighted according to the preset weights to obtain the evaluation index of the algorithm to be evaluated.

[0071] The evaluation system for the 3D target detection algorithm provided in this disclosure obtains algorithm detection data corresponding to standard detection data through the algorithm to be evaluated, establishes multiple view coordinate systems, calculates the data overlap between the algorithm detection data and the standard detection data in different coordinate systems, filters positive examples of sample recognition in the algorithm detection data in different coordinate systems based on the data overlap, determines the coordinate system accuracy of each coordinate system based on the number of positive examples of sample recognition, and then calculates the weighted coordinate system accuracy of each view coordinate system to obtain the evaluation index. This system has the following advantages:

[0072] First, compared to directly determining the evaluation result of the detection algorithm through the overlap of a single view, different data overlaps between standard detection data and algorithm detection data are obtained through different view coordinate systems, thereby determining the coordinate system accuracy under different view coordinate systems. Finally, the evaluation index is obtained by weighted calculation based on the coordinate system accuracy, which increases the range of evaluation parameters and thus improves the accuracy of the evaluation results.

[0073] Secondly, compared to evaluation results that are only qualified or unqualified, evaluation indicators are more suitable for judging the differences between different detection algorithms, thus providing a reference for the improvement of 3D target detection algorithms;

[0074] Third, the evaluation parameters are classified by multiple distance intervals and multiple overlap thresholds, and evaluation indicators are calculated according to their respective weights, thereby improving the flexibility of the evaluation and making the evaluation results more accurate.

[0075] Combination Figure 3 As shown, this disclosure provides an evaluation system for a three-dimensional target detection algorithm, including an acquisition module 301, an overlap determination module 302, an accuracy determination module 303, and a calculation module 304. The acquisition module 301 acquires the algorithm to be evaluated, and acquires multiple sample data and standard detection data corresponding to each sample data. The algorithm to be evaluated performs target detection on each sample data to obtain algorithm detection data corresponding to each sample data. The overlap determination module 302 establishes multiple preset view coordinate systems corresponding to view categories, designates any view coordinate system as the target coordinate system, and determines the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system. The accuracy determination module 303 determines positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and a preset overlap threshold, and determines the coordinate system accuracy of the target coordinate system based on the number of positive examples. The calculation module 304 performs weighted calculation of the coordinate system accuracy of each view coordinate system based on preset view weights to obtain the evaluation index of the algorithm to be evaluated.

[0076] The evaluation system for the 3D target detection algorithm provided in this disclosure obtains algorithm detection data corresponding to standard detection data from the algorithm to be evaluated, establishes multiple view coordinate systems, calculates the data overlap between the algorithm detection data and the standard detection data in different coordinate systems, filters positive examples of sample recognition from the algorithm detection data in different coordinate systems based on the data overlap, determines the coordinate system accuracy of each coordinate system based on the number of positive examples, and then calculates the weighted coordinate system accuracy of each view coordinate system to obtain the evaluation index. In this way, compared to directly determining the evaluation result of the detection algorithm through the overlap of a single view, obtaining different data overlaps between the standard detection data and the algorithm detection data through different view coordinate systems, thereby determining the coordinate system accuracy in different view coordinate systems, and finally calculating the evaluation index based on the weighted coordinate system accuracy, increases the range of evaluation parameters, thereby improving the accuracy of the evaluation results. Furthermore, the evaluation index is more suitable for judging the differences between different detection algorithms, thus providing a reference for improving 3D target detection algorithms.

[0077] Figure 4 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 4 The computer system 400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0078] like Figure 4 As shown, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from Storage Unit 408 into Random Access Memory (RAM) 403. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0079] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0080] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs various functions defined in the system of this application.

[0081] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0082] This disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements any of the methods described in this embodiment.

[0083] The computer-readable storage medium in the embodiments of this disclosure will be understood by those skilled in the art: all or part of the steps of the above method embodiments can be implemented by hardware related to computer programs. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0084] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic device performs the various steps of the above method.

[0085] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0086] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. An evaluation method for a three-dimensional target detection algorithm, characterized in that, include: Obtain the algorithm to be evaluated, and obtain multiple sample data and standard detection data corresponding to each sample data. The algorithm to be evaluated is used to perform target detection on each sample data to obtain the algorithm detection data corresponding to each sample data. Establish multiple view coordinate systems corresponding to preset view categories, determine any view coordinate system as the target coordinate system, and determine the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system. Based on the comparison result between the data overlap and the preset overlap threshold, positive examples of sample recognition are determined from the algorithm detection data, and the coordinate system accuracy of the target coordinate system is determined based on the number of positive examples of sample recognition. The coordinate system accuracy of each view coordinate system is calculated by weighting the preset view weights of each view coordinate system to obtain the evaluation index of the algorithm to be evaluated. Determining the coordinate system accuracy of the target coordinate system based on the number of positive examples identified by the samples includes: defining any distance interval as the target interval; counting the number of classification detection data corresponding to the target interval to obtain the total amount of classification data corresponding to the target interval; counting the number of positive examples identified by each of the overlap thresholds to obtain the number of threshold positive examples for each overlap threshold; calculating the number of threshold positive examples for each overlap threshold based on the total amount of classification data to obtain the threshold accuracy for each overlap threshold; weighting each threshold accuracy according to a preset threshold weight corresponding to each overlap threshold to obtain the interval accuracy of the target interval; and weighting each interval accuracy according to a preset interval weight corresponding to each distance interval to obtain the coordinate system accuracy of the target coordinate system.

2. The method according to claim 1, characterized in that, After determining any view coordinate system as the target coordinate system, and before determining positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and the preset overlap threshold, the method further includes: Obtain multiple distance intervals; The center point of each of the algorithm detection data is determined according to the target coordinate system, and the Euclidean distance between the center point and the origin of the target coordinate system is determined as the center point distance; Based on the matching results between each distance interval and each center point, the classification detection data corresponding to each distance interval is determined from the algorithm detection data.

3. The method according to claim 2, characterized in that, After acquiring multiple sample data and the standard detection data corresponding to each sample data, and before establishing view coordinate systems corresponding to multiple preset view categories, the method further includes: Obtain the standard target category and standard target orientation corresponding to the standard detection data; Obtain at least one of the following: the target category, target orientation, and detection confidence level corresponding to the detection data of the algorithm.

4. The method according to claim 3, characterized in that, Based on the comparison between the data overlap and a preset overlap threshold, positive examples of sample recognition corresponding to each view coordinate system are determined from the algorithm-detected data, including: The number of overlap thresholds includes multiple types; Any overlap threshold is determined as the target threshold. A first positive example condition, a second positive example condition, a third positive example condition, and a fourth positive example condition are preset. The first positive example condition includes that the classification detection data is greater than or equal to the target threshold. The second positive example condition includes that the detection confidence of the classification detection data is greater than or equal to the preset confidence threshold. The third positive example condition includes that the detection target category of the classification detection data is the same as the corresponding standard target category. The fourth positive example condition includes that the detection target orientation of the classification detection data is the same as the corresponding standard target orientation. At least a portion of the first positive example condition, the second positive example condition, the third positive example condition, and the fourth positive example condition are determined as decision conditions; If any classification detection data satisfies the determination condition, then the classification detection data is determined as a positive example of sample recognition for the target threshold.

5. The method according to claim 1, characterized in that, The evaluation index of the algorithm to be evaluated is determined by the following formula: In the formula, AP all AP is the evaluation metric for the algorithm to be evaluated. view(k) Let θ be the coordinate system accuracy of the k-th view coordinate system. view(k) Here, s represents the preset view weight for the k-th view coordinate system, and s represents the number of view coordinate systems. dis(j) Let β be the interval precision for the j-th distance interval. dis(j) Here, m represents the preset interval weight for the j-th distance interval, and m is the number of distance intervals. AP thr(i) Let α be the threshold accuracy for the i-th overlap threshold. thr(i) is the preset threshold weight for the i-th overlap threshold, and n is the number of overlap thresholds.

6. The method according to any one of claims 1 to 5, characterized in that, Determining the data overlap between the algorithm-detected data and the standard detection data based on the target coordinate system includes: If the preset view category includes a 3D view, then the intersection of the algorithm detection data and the standard detection data is determined as the first coordinate point; The vertex detected by the algorithm is determined as the first vertex, and the first vertex contained in the standard detection data is determined as the second coordinate point; The vertex in the standard detection data is determined as the second vertex, and the second vertex contained in the algorithm detection data is determined as the third coordinate point; The first coordinate point, the second coordinate point, and the third coordinate point are triangulated according to a preset triangulation algorithm to obtain a reference region; The data overlap between the algorithm detection data and the standard detection data is calculated based on the volume of the algorithm detection data, the volume of the standard detection data, and the volume of the reference region.

7. The method according to claim 6, characterized in that, The data overlap between the algorithm-detected data and the standard detection data is determined using the following formula: In the formula, IoU is the data overlap between the algorithm-detected data and the standard detection data, V1 is the volume of the algorithm-detected data, V2 is the volume of the standard detection data, and V I Let be the volume of the reference region.

8. An evaluation system for a three-dimensional target detection algorithm, characterized in that, include: The acquisition module is used to acquire the algorithm to be evaluated, and to acquire multiple sample data and standard detection data corresponding to each sample data. The algorithm to be evaluated is used to perform target detection on each sample data to obtain the algorithm detection data corresponding to each sample data. The overlap determination module is used to establish view coordinate systems corresponding to multiple preset view categories, determine any view coordinate system as the target coordinate system, and determine the data overlap between the algorithm detection data and the standard detection data based on the target coordinate system. The accuracy determination module is used to determine positive examples of sample recognition from the algorithm detection data based on the comparison result between the data overlap and the preset overlap threshold, and to determine the coordinate system accuracy of the target coordinate system based on the number of positive examples of sample recognition. The calculation module is used to perform weighted calculation of the coordinate system accuracy of each view coordinate system based on the preset view weights of each view coordinate system, so as to obtain the evaluation index of the algorithm to be evaluated. The accuracy determination module determines the coordinate system accuracy of the target coordinate system based on the number of positive examples identified by the samples in the following manner: Any distance interval is defined as the target interval; the number of classification detection data corresponding to the target interval is counted to obtain the total amount of classification data corresponding to the target interval; the number of positive examples identified by the samples at each overlap threshold is counted to obtain the threshold positive example count for each overlap threshold; the threshold accuracy of each overlap threshold is calculated based on the total amount of classification data; the threshold accuracy is weighted according to the preset threshold weights corresponding to each overlap threshold to obtain the interval accuracy of the target interval; and the interval accuracy of each distance interval is weighted according to the preset interval weights corresponding to each distance interval to obtain the coordinate system accuracy of the target coordinate system.

9. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images