Method, device and equipment for determining detector detection performance, and storage medium

Abstract

A method, device and equipment for determining the performance of a detector and a storage medium are disclosed. The method comprises: obtaining a plurality of frames of to-be-detected data, inputting the plurality of frames of to-be-detected data into a target detector respectively, and obtaining obstacle detection information corresponding to each frame of to-be-detected data output by the detector; for each frame of to-be-detected data, obtaining standard obstacle information corresponding to the to-be-detected data, determining obstacle difference information based on the obstacle detection information and the standard obstacle information, and determining a preliminary matching result corresponding to the to-be-detected data based on the obstacle difference information; obtaining reference detection data corresponding to the to-be-detected data, determining a target matching result of the to-be-detected data based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the to-be-detected data; and determining a detection performance index of the detector based on the target matching results corresponding to the plurality of frames of to-be-detected data. The method improves the rationality of the detection result of the detector.

Description

Technical Field

[0001] This invention relates to the field of detection technology, and in particular to a method, apparatus, device, and storage medium for determining the detection performance of a detector. Background Technology

[0002] In the field of autonomous driving, the detection performance of onboard systems for targets around the vehicle is crucial to vehicle safety. In existing technologies, the research and testing of onboard environmental perception detectors are often based on a one-to-one matching comparison between detected targets and ground truth values ​​in a single frame, meaning that one frame of detection data corresponds to one frame of ground truth data.

[0003] However, autonomous driving systems employ tracking strategies downstream of the detectors. Specifically, techniques such as Kalman filtering can effectively compensate for occasional missed obstacle detections. Therefore, only the obstacle information list output after tracking by the tracker will influence subsequent vehicle path planning.

[0004] Therefore, performance testing methods based on single-frame detection not only easily lead to a waste of precision, but also the performance test results based on single-frame detection often fail to accurately reflect the detector's true detection performance in scene detection. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for determining the detection performance of a detector, in order to solve the technical problems of wasted precision and inability to accurately reflect the detector's real-world detection performance caused by performance detection methods based on single-frame detection.

[0006] According to one aspect of the present invention, a method for determining the detection performance of a detector is provided, the method comprising:

[0007] Acquire multiple frames of data to be detected, input the multiple frames of data to be detected into the target detector respectively, and obtain the obstacle detection information corresponding to each frame of data to be detected output by the detector;

[0008] For each frame of data to be detected, standard obstacle information corresponding to the data to be detected is obtained. Obstacle difference information is determined based on obstacle detection information and standard obstacle information. Preliminary matching result corresponding to the data to be detected is determined based on obstacle difference information.

[0009] Obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected. The reference detection data is the data to be detected that is adjacent to the data to be detected.

[0010] The detection performance index of the detector is determined based on the target matching results corresponding to multiple frames of data to be detected.

[0011] According to another aspect of the present invention, an apparatus for determining the detection performance of a detector is provided, the apparatus comprising:

[0012] The data acquisition module is used to acquire multiple frames of data to be detected, and input these frames into the target detector to obtain obstacle detection information corresponding to each frame of data output by the detector.

[0013] The preliminary matching result determination module is used to obtain standard obstacle information corresponding to each frame of data to be detected, determine obstacle difference information based on obstacle detection information and standard obstacle information, and determine the preliminary matching result corresponding to the data to be detected based on obstacle difference information.

[0014] The target matching result determination module is used to obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected. The reference detection data is the data to be detected that is adjacent to the data to be detected.

[0015] The detection performance index determination module is used to determine the detection performance index of the detector based on the target matching results corresponding to multiple frames of data to be detected.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory that is communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform a method for determining the detection performance of the detector according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute a method for determining the detection performance of a detector according to any embodiment of the present invention.

[0021] The technical solution of this invention first acquires multiple frames of data to be detected and inputs them into a target detector to obtain obstacle detection information corresponding to each frame of data, providing a data source for multi-frame detection. Then, for each frame of data to be detected, standard obstacle information corresponding to the data is acquired. Obstacle difference information is determined based on the obstacle detection information and the standard obstacle information, and a preliminary matching result is determined based on the obstacle difference information. Next, reference detection data corresponding to the data to be detected is acquired. The target matching result of the data to be detected is determined based on the preliminary matching result of the reference detection data and the preliminary matching result of the data to be detected, where the reference detection data is the data to be detected from neighboring frames. By referencing the detection results of neighboring frames, optimization and correction of the data to be detected are achieved. Finally, the detection performance index of the detector is determined based on the target matching results corresponding to multiple frames of data to be detected. This solves the technical problems of wasted precision and inaccurate reflection of the detector's real-world detection performance caused by single-frame detection-based performance testing methods. It has achieved beneficial results, such as improving the rationality of detector performance test results, effectively guiding the direction of detector performance improvement, and providing more accurate detection data for vehicle driving path planning.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a method for determining the detection performance of a detector according to Embodiment 1 of the present invention;

[0025] Figure 2 This is a logic block diagram for determining the detection performance of a detector according to Embodiment 2 of the present invention;

[0026] Figure 3 This is a logic block diagram for determining the detection performance of a detector according to Embodiment 2 of the present invention;

[0027] Figure 4 This is a schematic diagram of a device for determining the detection performance of a detector according to Embodiment 3 of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 This is a flowchart illustrating a method for determining detector detection performance according to Embodiment 1 of the present invention. This embodiment is applicable to situations where detector performance is being detected. The method can be executed by a device for determining detector detection performance, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110. Acquire multiple frames of data to be detected, input the multiple frames of data to be detected into the target detector respectively, and obtain the obstacle detection information corresponding to each frame of data to be detected output by the detector.

[0034] In this embodiment, the data to be detected can be a single frame of scene fragment imaging data acquired by environmental perception devices such as radar imaging systems. Specifically, the environmental perception device can transmit pulse signals to the surrounding environment and then receive pulse signals reflected by objects. The environmental perception device can then calculate the distance and azimuth information between the target object and the radar based on the time delay and phase information of the received reflected signals. Simultaneously, the environmental perception device can also calculate the size and shape of the target object based on the amplitude information of the reflected signals, thereby enabling imaging and recognition of the target object. After receiving multiple reflected signals, the system uses signal processing algorithms to process the reflected signals, thereby achieving the recognition of a scene including multiple target objects and the acquisition of scene fragment imaging data. Multiple frames of data to be detected can be multiple frames from the scene fragment imaging data. For example, the sensing frequency of the radar imaging system can be at least 10Hz (the interval between consecutive frames can be greater than 0.1s), and the multiple frames of data to be detected can be scene fragment data including at least 5 consecutive frames. The target detector can be a terminal, system, or performance detection module used for performance testing of the data to be detected. Obstacle detection information can be information related to the obstacle detection results output after detecting obstacles in each frame of the scene segment imaging data. For example, the detection information such as the position and shape of obstacles.

[0035] Specifically, at least five consecutive frames of scene data can be acquired using a radar imaging system as five data points to be detected, and these multiple frames can be input into a target detector. Then, based on the analysis and detection by the target detector, obstacle detection information corresponding to each frame of data output by the detector can be obtained.

[0036] S120. For each frame of data to be detected, obtain standard obstacle information corresponding to the data to be detected, determine obstacle difference information based on obstacle detection information and standard obstacle information, and determine preliminary matching results corresponding to the data to be detected based on obstacle difference information.

[0037] In this embodiment, the standard obstacle information can be the true value data information of obstacles obtained using a data acquisition device, system, or acquisition module. For example, the standard obstacle information can be the obstacle true value data information extracted from scene segments acquired by the IbeoReference System evaluation system. It is understood that the true value data acquisition device and the environmental perception device can complete spatial calibration and temporal synchronization beforehand. The standard obstacle information corresponding to the data to be detected can be the objective true value data corresponding to each frame of data from multiple frames of data to be detected, acquired by the true value data acquisition device after spatial calibration and temporal synchronization. The obstacle difference information can be the data difference information in dimensions such as the shape and position of obstacles between the obstacle detection information and the standard obstacle information. The preliminary matching result can be the preliminary matching result determined based on the obstacle difference information of one frame from multiple frames of data to be detected and the standard obstacle information. According to the magnitude of the difference in the obstacle difference information, the preliminary matching result can be a correct match (positive class) with a small difference or a wrong match (negative class) with a large difference.

[0038] Specifically, for each frame of data to be detected, standard obstacle information corresponding to each frame can be obtained based on the ground truth data acquisition device. This standard obstacle information is the ground truth data of obstacles acquired by the ground truth data acquisition device after spatial calibration and temporal synchronization with the environmental perception device. Then, one frame from multiple frames of data to be detected can be selected, and obstacle detection information, along with the standard obstacle information extracted from the corresponding frame of ground truth data, can be extracted based on this data. Finally, obstacle difference information can be determined based on differences in obstacle shape and position, and a preliminary matching result corresponding to the data to be detected can be determined based on this obstacle difference information.

[0039] S130. Obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected, wherein the reference detection data is the data to be detected that is adjacent to the data to be detected.

[0040] In this embodiment, the reference detection data can be the detection data corresponding to the frames immediately before and after one of the frames to be detected from multiple frames of detection data. The target matching result can be the matching result obtained by comprehensively judging all the preliminary matching results corresponding to the multiple frames of detection data. Since single-frame detection cannot eliminate the influence of measurement noise on the single-frame detection result, the target matching result determined based on the preliminary matching results corresponding to the reference detection data and the preliminary matching results corresponding to the detection data can effectively improve the rationality of the detection result. The target matching result can be a correct match (positive class) or an incorrect match (negative class) obtained after correcting the preliminary matching result.

[0041] For example, the preliminary matching result corresponding to the data to be detected can be the matching result determined based on the third frame of the data to be detected out of five consecutive frames. The preliminary matching result corresponding to the reference detection data can be the preliminary matching result corresponding to the two frames of the data to be detected before and after the data to be detected. The preliminary matching result corresponding to the data to be detected can be corrected based on the preliminary matching result corresponding to the two frames of the data to be detected before and after the data to be detected, to obtain the target matching result corresponding to the data to be detected.

[0042] S140. Determine the detector's detection performance indicators based on the target matching results corresponding to multiple frames of data to be detected.

[0043] In this embodiment, the detection performance index of the detector is determined based on the target matching results corresponding to multiple frames of data to be detected. This can be achieved by evaluating the detector's performance by plotting PR curves and ROC curves, based on the statistical results of the true positive, false positive, false negative, and true negative classes corresponding to multiple target matching results. Specifically, the true positive class can be TP (True Positive): the initial matching result is positive, and the target matching result is positive; the false positive class can be FP (False Positive): the initial matching result is positive, and the target matching result is negative; the false negative class can be FN (False Negative): the initial matching result is negative, and the target matching result is positive; and the true negative class can be TN (True Negative): the initial matching result is negative, and the target matching result is negative.

[0044] Optionally, obstacle detection information refers to the obstacle detection location, and standard obstacle information refers to the standard obstacle location. Obstacle difference information is determined based on the obstacle detection information and the standard obstacle information, and a preliminary matching result corresponding to the data to be detected is determined based on the obstacle difference information, including: calculating the deviation distance between the obstacle detection location and the standard obstacle location, and determining the preliminary matching result corresponding to the data to be detected based on the deviation distance and a preset distance threshold.

[0045] In this embodiment, the obstacle detection location can be the relative spatial location information between the obstacle and the environmental sensing device, such as the distance between the obstacle and the environmental sensing device. The obstacle standard location can be the relative spatial location information between the obstacle and the ground truth data acquisition device, such as the distance between the obstacle and the ground truth data acquisition device. The preset distance threshold can be a distance limit value preset according to the actual situation. For example, when the deviation distance is greater than the preset distance threshold, the preliminary matching result can be determined as a matching error (negative class); when the deviation distance is less than or equal to the preset distance threshold, the preliminary matching result can be determined as a matching correct (positive class).

[0046] Optionally, obstacle detection information refers to the obstacle detection area, and standard obstacle information refers to the obstacle standard area. Obstacle difference information is determined based on the obstacle detection information and the standard obstacle information, and a preliminary matching result corresponding to the data to be detected is determined based on the obstacle difference information, including: calculating the area intersection-union ratio between the obstacle detection area and the obstacle standard area, and determining the preliminary matching result corresponding to the data to be detected based on the area intersection-union ratio and a preset intersection-union ratio threshold.

[0047] In this embodiment, the obstacle detection area can be the geometric information of the obstacle region determined in the obstacle detection information, such as the shape of the obstacle. The obstacle standard area can be the geometric information of the obstacle region acquired by the ground truth data acquisition device. The preset intersection-over-union (IoU) threshold can be a pre-set IoU limit value based on actual conditions, such as 0.6. It is understood that the larger the IoU between the obstacle detection area and the obstacle standard area, the closer the detected data is to the ground truth data.

[0048] For example, when the intersection-union ratio (IU) between the obstacle detection area and the standard obstacle area is greater than a preset IU threshold of 0.6, the preliminary matching result can be determined as a correct match (positive class). When the IU between the obstacle detection area and the standard obstacle area is less than or equal to the preset IU threshold of 0.6, the preliminary matching result can be determined as a mismatch (negative class).

[0049] Optionally, determining the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected includes at least one of the following operations: when the number of frames of the reference detection data that is different from the preliminary matching result corresponding to the data to be detected reaches a preset number of frames, the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected; when the reference detection data is adjacent to the data to be detected and is different from the preliminary matching result corresponding to the data to be detected, if the obstacle difference information corresponding to the preliminary matching result of the data to be detected meets a first preset condition, the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected; when the obstacle difference information corresponding to the preliminary matching result of the reference detection data meets a second preset condition, the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected.

[0050] In this embodiment, the preset number of frames can be a number of consecutive frames preset according to the actual situation, such as 3 consecutive frames.

[0051] Specifically, when the preliminary matching result of three consecutive frames of reference detection data in a scene segment is a correct match (positive class), while the preliminary matching result of the data to be detected is an incorrect match (negative class), the correct match (positive class) can be used as the target matching result for the data to be detected. Conversely, when the preliminary matching result of three consecutive frames of reference detection data in a scene segment is an incorrect match (negative class), while the preliminary matching result of the data to be detected is a correct match (positive class), the incorrect match (negative class) can be used as the target matching result for the data to be detected.

[0052] The first preset condition can be a range of difference information values ​​preset according to the actual situation. For example, the first preset condition can be that the obstacle difference information is greater than a preset deviation distance threshold or a preset intersection-union ratio threshold. Alternatively, the first preset condition can be that the obstacle difference information is greater than a preset deviation distance threshold or a preset intersection-union ratio threshold.

[0053] Specifically, if the reference detection data is adjacent to the data to be detected and the preliminary matching result corresponding to the data to be detected is different, and if the obstacle difference information corresponding to the preliminary matching result of the data to be detected meets a specific difference information value range, then the preliminary matching result of the reference detection data can be used as the target matching result of the data to be detected.

[0054] For example, if the preliminary matching result of the data to be detected in two consecutive frames immediately adjacent to and before or after the data to be detected is a correct match (positive class), while the preliminary matching result of the data to be detected is a wrong match (negative class), and the intersection-union ratio (IUGR) of the obstacle difference information corresponding to the preliminary matching result of the data to be detected is greater than 0.3, then the preliminary matching result of the reference detection data that is a correct match (positive class) can be used as the target matching result of the data to be detected. Conversely, if the preliminary matching result of the data to be detected in two consecutive frames immediately adjacent to and before or after the data to be detected is a wrong match (negative class), while the preliminary matching result of the data to be detected is a correct match (positive class), and the IUGR of the obstacle difference information corresponding to the preliminary matching result of the data to be detected is less than 0.3, then the preliminary matching result of the reference detection data that is a wrong match (negative class) can be used as the target matching result of the data to be detected.

[0055] The second preset condition can be a range of difference information values ​​pre-set according to the actual situation. For example, the second preset condition can be that the obstacle difference information is greater than a preset deviation distance threshold or a preset intersection-union ratio (IU / R) threshold, or the second preset condition can be that the obstacle difference information is greater than a preset deviation distance threshold or a preset IU / R threshold. For example, the second preset condition can be that the IU / R of the obstacle difference information corresponding to the preliminary matching result of the data to be detected is greater than 0.9.

[0056] Specifically, if the obstacle difference information corresponding to the preliminary matching result of the reference detection data satisfies the second preset condition that the intersection-union ratio is greater than 0.9, then the preliminary matching result of the reference detection data can be used as the target matching result of the data to be detected if the preliminary matching result of the reference detection data is correct (positive class).

[0057] Optionally, obtaining reference detection data corresponding to the data to be detected includes at least one of the following operations: obtaining a first preset number of frames of data to be detected that are adjacent to the acquisition time of the data to be detected and located before and / or after the data to be detected, as reference detection data corresponding to the data to be detected; obtaining a second preset number of frames of data to be detected that are consecutive in acquisition time among multiple frames of the data to be detected.

[0058] In this embodiment, the first preset number of frames can be a certain number of frames of data to be detected that are adjacent to the acquisition time of the data to be detected and located before and / or after the data to be detected. For example, it can be one frame before or one frame after the acquisition time of the data to be detected. The second preset number of frames can be a number of frames preset according to the actual situation, for example, it can be two frames. Obtaining the second preset number of frames of data to be detected that are consecutive in acquisition time from multiple frames of data to be detected can be obtaining two consecutive frames of data to be detected before or after the data to be detected in a scene segment.

[0059] Optionally, determining the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected includes at least one of the following operations: when the preliminary matching results corresponding to the data to be detected in the previous frame and the data to be detected in the next frame are both different from the preliminary matching result of the data to be detected, the preliminary matching result corresponding to the data to be detected in the previous frame or the data to be detected in the next frame is taken as the target matching result of the data to be detected; when the preliminary matching results of the data to be detected in a second preset number of consecutive frames adjacent to the data to be detected and located before or after the data to be detected are all different from the preliminary matching result of the data to be detected. If the preliminary matching results of the reference detection data are different from those of the target data, and the obstacle difference information corresponding to the preliminary matching result of the target data is within a preset first information value range, then the preliminary matching result of the reference detection data is taken as the target matching result of the target data; if the preliminary matching results of the target data in the previous frame or the next frame are different from the preliminary matching results of the target data, and the obstacle difference information corresponding to the preliminary matching results of the target data in the previous frame or the next frame is within a preset second information value range, then the preliminary matching result of the reference detection data is taken as the target matching result of the target data.

[0060] In this embodiment, the preset first and second information value ranges can be value ranges of difference information pre-set according to actual conditions. For example, the first and second information value ranges can be value ranges of deviation distance or value ranges of regional intersection-union ratio.

[0061] Specifically, if the preliminary matching results of the data to be detected in both the preceding and following frames are different from the preliminary matching result of the data to be detected, the preliminary matching result of either the preceding or following frame can be used as the target matching result of the data to be detected. For example, if the preliminary matching result of the data to be detected is a correct match (positive class) while the preliminary matching results of the preceding and following frames are incorrect matches (negative class), the correct match (positive class) can be used as the target matching result of the data to be detected; conversely, if the preliminary matching result of the data to be detected is an incorrect match (negative class) while the preliminary matching results of the preceding and following frames are correct matches (positive class), the incorrect match (negative class) can be used as the target matching result of the data to be detected.

[0062] If the preliminary matching result of the data to be detected in two consecutive frames immediately adjacent to and before or after the data to be detected is a correct match (positive class), while the preliminary matching result of the data to be detected is a wrong match (negative class), and if the obstacle difference information corresponding to the preliminary matching result of the data to be detected is greater than a preset deviation distance threshold or a preset intersection-union ratio threshold, then the preliminary matching result of the reference detection data can be used as the target matching result of the data to be detected if the preliminary matching result of the data to be detected is a correct match (positive class). Conversely, if the preliminary matching result of the data to be detected in two consecutive frames immediately adjacent to and before or after the data to be detected is a wrong match (negative class), while the preliminary matching result of the data to be detected is a correct match (positive class), and if the obstacle difference information corresponding to the preliminary matching result of the data to be detected is less than a preset deviation distance threshold or a preset intersection-union ratio threshold, then the preliminary matching result of the reference detection data can be used as the target matching result of the data to be detected if the preliminary matching result of the reference detection data is a wrong match (negative class).

[0063] If the preliminary matching results for the data to be detected in the preceding or following frame are both correct (positive), while the preliminary matching results for the data to be detected are incorrect (negative), and the obstacle difference information corresponding to the preliminary matching results of the preceding or following frame is greater than a preset deviation distance threshold or a preset intersection-union ratio (IU / IU) threshold, then the preliminary matching result of the reference detection data can be considered correct (positive) and used as the target matching result for the data to be detected. Conversely, if the preliminary matching results for the data to be detected in the preceding or following frame are both incorrect (negative), while the preliminary matching results for the data to be detected are correct (positive), and the obstacle difference information corresponding to the preliminary matching results of the preceding or following frame is less than a preset deviation distance threshold or a preset IU / IU / IU threshold, then the preliminary matching result of the reference detection data can be considered incorrect (negative) and used as the target matching result for the data to be detected.

[0064] Optionally, the detection performance index of the detector is determined based on the target matching results corresponding to multiple frames of data to be detected, including: constructing a receiver operating feature curve based on the target matching results corresponding to multiple frames of data to be detected, and determining the detection performance index of the detector based on the area under the receiver operating feature curve.

[0065] In this embodiment, the horizontal axis of the receiver operating characteristic curve (ROC) can be represented by the False Positive Rate (FPR), which represents the percentage of data with negative target matching results that initially match positive (false positives), where FPR = FP / (FP+TN). The vertical axis of the ROC curve can be represented by the True Positive Rate (TPR), which represents the percentage of data with positive target matching results that initially match positive (true positives), where TPR = TP / (TP+FN). The detector's detection performance index can be determined by the area under the ROC curve (AUC). Since a larger TPR indicates a larger percentage of data with positive target matching results initially match positive, the detector's performance can be considered superior. Therefore, a larger AUC on the ROC curve indicates a higher detector performance index.

[0066] The technical solution of this embodiment acquires multiple frames of data to be detected and inputs them into a target detector to obtain obstacle detection information corresponding to each frame of data output by the detector, providing a data source for multi-frame detection. Then, for each frame of data to be detected, standard obstacle information corresponding to the data to be detected is acquired. Obstacle difference information is determined based on the obstacle detection information and the standard obstacle information, and a preliminary matching result corresponding to the data to be detected is determined based on the obstacle difference information. Next, reference detection data corresponding to the data to be detected is acquired. The target matching result of the data to be detected is determined based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected, where the reference detection data is the data to be detected from neighboring frames. By referencing the detection results of neighboring frames, optimization and correction of the data to be detected are achieved. Finally, the ROC curve is determined based on the target matching results corresponding to multiple frames of data to be detected, and the detection performance index of the detector is determined based on the area under the ROC curve (AUC). This solves the technical problems of precision waste and inaccurate reflection of the detector's real-scene detection performance caused by performance detection methods based on single-frame detection. It has achieved beneficial results, such as improving the rationality of detector performance test results, effectively guiding the direction of detector performance improvement, and providing more accurate detection data for vehicle driving path planning.

[0067] Example 2

[0068] Figure 2-3This is a logical block diagram of a method for determining detector detection performance provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment uses the Intersection over Union (IoU) ratio between the obstacle detection area and the standard obstacle area as an example to specifically illustrate the method for determining the statistical results of the true positive, false positive, false negative, and true negative classes corresponding to the target matching result. An IoU greater than 0.6 is considered a correct match, and less than or equal to 0.6 is considered a mismatch. The preceding and following frames of the data to be detected can be reference detection data when the first preset number of frames is 1 frame in the aforementioned embodiments; three consecutive frames can be the data to be detected in consecutive frames of the second preset number when the second preset number is 2 in the aforementioned embodiments. Three consecutive frames can be reference detection data in the aforementioned embodiments where the data to be detected is in the same scene segment and the acquisition time is adjacent to a preset number of frames when the consecutive preset number of frames is 3. For detailed implementation methods, please refer to the description of this embodiment. Technical features that are the same as or similar to those in the aforementioned embodiments will not be repeated here.

[0069] like Figure 2 As shown, the logic block diagram includes:

[0070] 1. Acquire multiple frames of data to be detected, and input these frames into the target detector to obtain obstacle detection information.

[0071] 2. For each frame of data to be detected, determine the preliminary matching result corresponding to the data to be detected, and obtain the reference detection data corresponding to the data to be detected.

[0072] When the data to be detected is determined to be a mismatch,

[0073] 3. If the data to be detected is correctly matched in both the preceding and following frames, the data to be detected is considered to be correctly matched (FN);

[0074] 4. If three consecutive frames match correctly, the other frames are considered to be correctly matched (FN);

[0075] 5. If the two consecutive frames immediately preceding or the two frames immediately following are correctly matched, the IoU of the data to be detected is considered to be a correct match (FN) if it is greater than 0.3.

[0076] 6. When the value of the immediately preceding or following frame is greater than 0.9 (extremely high matching degree), the data to be detected is considered to be a correct match (FN);

[0077] 7. If the above conditions are not met, it is considered a mismatch (TN).

[0078] 8. Determine the detector's detection performance indicators based on the target matching results corresponding to multiple frames of data to be detected.

[0079] like Figure 3As shown, the logic block diagram includes:

[0080] 1. Acquire multiple frames of data to be detected, and input these frames into the target detector to obtain obstacle detection information.

[0081] 2. For each frame of data to be detected, determine the preliminary matching result corresponding to the data to be detected, and obtain the reference detection data corresponding to the data to be detected.

[0082] When the data to be detected is determined to be a correct match

[0083] 3. If both the preceding and following frames of the data to be detected are incorrect matches, the data to be detected is considered a match error (FP).

[0084] 4. If there are three consecutive frames with incorrect matches, the other frames are considered to be incorrect matches (FP);

[0085] 5. If there is a matching error in two consecutive frames immediately preceding or two frames immediately following, the IoU of the data to be detected is less than or equal to 0.3, which is considered a matching error (FP).

[0086] 6. If the value of the immediately preceding or following frame is less than 0.1, the data to be detected is considered a match error (FP);

[0087] 7. If the above conditions are not met, the match is considered correct (TP).

[0088] 8. Determine the detector's detection performance indicators based on the target matching results corresponding to multiple frames of data to be detected.

[0089] The technical solution of this embodiment determines the target matching result of the data to be detected based on the preliminary matching results corresponding to the reference detection data and the preliminary matching results corresponding to the data to be detected, and obtains the statistical results of the true positive class (TP), false positive class (FP), false negative class (FN), and true negative class (TN) corresponding to the target matching results. This embodiment transforms the target detection problem into a classification problem by continuously judging the matching results, providing data basis for drawing PR curves and ROC curves and calculating the area under the curves to determine the detector's detection performance indicators. This achieves the beneficial effects of improving the rationality of detector performance detection results, effectively guiding the direction of detector performance improvement, and providing more accurate detection data for vehicle path planning.

[0090] Example 3

[0091] Figure 4 This is a schematic diagram of a device for determining the detection performance of a detector according to Embodiment 3 of the present invention. Figure 4As shown, the device includes: a data acquisition module 410, a preliminary matching result determination module 420, a target matching result determination module 430, and a detection performance index determination module 440.

[0092] The detection data acquisition module 410 is used to acquire multiple frames of detection data, input the multiple frames of detection data into the target detector respectively, and obtain the obstacle detection information corresponding to each frame of detection data output by the detector.

[0093] The preliminary matching result determination module 420 is used to obtain standard obstacle information corresponding to each frame of data to be detected, determine obstacle difference information based on obstacle detection information and standard obstacle information, and determine the preliminary matching result corresponding to the data to be detected based on obstacle difference information.

[0094] The target matching result determination module 430 is used to obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected, wherein the reference detection data is the data to be detected that is adjacent to the data to be detected;

[0095] The detection performance index determination module 440 is used to determine the detection performance index of the detector based on the target matching results corresponding to multiple frames of data to be detected.

[0096] Based on the above technical solution, optionally, obstacle detection information is obstacle detection location, and standard obstacle information is obstacle standard location; the preliminary matching result determination module 420 is specifically used to calculate the deviation distance between the obstacle detection location and the obstacle standard location, and determine the preliminary matching result corresponding to the data to be detected based on the deviation distance and a preset distance threshold.

[0097] Based on the above technical solution, optionally, obstacle detection information is obstacle detection area, and standard obstacle information is obstacle standard area; the preliminary matching result determination module 420 is specifically used to calculate the area intersection-union ratio between the obstacle detection area and the obstacle standard area, and determine the preliminary matching result corresponding to the data to be detected based on the area intersection-union ratio and the preset intersection-union ratio threshold.

[0098] Based on the above technical solution, optionally, the preliminary matching result determination module 420 is specifically used to perform at least one of the following operations:

[0099] If the number of frames of reference detection data that are different from the preliminary matching result of the data to be detected reaches a preset number, the preliminary matching result of the reference detection data will be used as the target matching result of the data to be detected.

[0100] If the reference detection data is adjacent to the data to be detected and the preliminary matching result corresponding to the data to be detected is different, if the obstacle difference information corresponding to the preliminary matching result of the data to be detected meets the first preset condition, then the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected.

[0101] If the obstacle difference information corresponding to the preliminary matching result of the reference detection data meets the second preset condition, the preliminary matching result of the reference detection data will be used as the target matching result of the data to be detected.

[0102] Based on the above technical solution, optionally, the preliminary matching result determination module 420 is specifically used to perform at least one of the following operations:

[0103] Acquire the first preset number of frames of data to be detected that are adjacent to the acquisition time of the data to be detected and located before and / or after the data to be detected, and use them as reference detection data corresponding to the data to be detected.

[0104] Acquire the second preset number of frames of data to be detected that are consecutive in acquisition time from multiple frames of data to be detected.

[0105] Based on the above technical solution, optionally, the target matching result determination module 430 is specifically used to perform at least one of the following operations:

[0106] If the preliminary matching results corresponding to the data to be detected in the previous frame and the data to be detected in the next frame are different from the preliminary matching results of the data to be detected, the preliminary matching result corresponding to the data to be detected in the previous frame or the data to be detected in the next frame shall be taken as the target matching result of the data to be detected.

[0107] If the preliminary matching results of the data to be detected in the next second preset number of frames that are adjacent to the data to be detected and are located before or after the data to be detected are all different from the preliminary matching results of the data to be detected, if the obstacle difference information corresponding to the preliminary matching results of the data to be detected is within the preset first information value range, then the preliminary matching results of the reference detection data will be used as the target matching results of the data to be detected.

[0108] If the preliminary matching result corresponding to the data to be detected in the previous frame or the next frame is different from the preliminary matching result of the data to be detected, and the obstacle difference information corresponding to the preliminary matching result of the data to be detected in the previous frame or the next frame is within the preset second information value range, then the preliminary matching result of the reference detection data will be used as the target matching result of the data to be detected.

[0109] Based on the above technical solution, optionally, the detection performance index determination module 440 is used to construct the receiver operation feature curve based on the target matching results corresponding to multiple frames of data to be detected, and to determine the detection performance index of the detector based on the area under the receiver operation feature curve.

[0110] The detector detection performance determination device provided in the embodiments of the present invention can execute the detector detection performance determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0111] Example 4

[0112] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for determining detector detection performance.

[0116] In some embodiments, the method for determining detector detection performance may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for determining detector detection performance described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the method for determining detector detection performance by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0122] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0123] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of determining the performance of a detector, characterized by, include: Acquire multiple frames of data to be detected, input the multiple frames of data to be detected into the target detector respectively, and obtain the obstacle detection information corresponding to each frame of data to be detected output by the detector; For each frame of data to be detected, standard obstacle information corresponding to the data to be detected is obtained. Obstacle difference information is determined based on the obstacle detection information and the standard obstacle information. A preliminary matching result corresponding to the data to be detected is determined based on the obstacle difference information. The standard obstacle information is the ground truth data information of obstacles obtained using a ground truth data acquisition device. Obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected, wherein the reference detection data is the data to be detected that is adjacent to the data to be detected; The detection performance index of the detector is determined based on the target matching results corresponding to multiple frames of the data to be detected; The step of determining the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected includes at least one of the following operations: If the number of frames of the reference detection data that are different from the preliminary matching result corresponding to the data to be detected reaches a preset number, the preliminary matching result of the reference detection data shall be taken as the target matching result of the data to be detected. If the reference detection data is adjacent to the data to be detected and the preliminary matching result corresponding to the data to be detected is different, and if the obstacle difference information corresponding to the preliminary matching result of the data to be detected meets the first preset condition, then the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected. If the obstacle difference information corresponding to the preliminary matching result of the reference detection data meets the second preset condition, the preliminary matching result of the reference detection data shall be used as the target matching result of the data to be detected.

2. The method of claim 1, wherein, The obstacle detection information is the obstacle detection location, and the standard obstacle information is the standard obstacle location; The step of determining obstacle difference information based on the obstacle detection information and the standard obstacle information, and determining a preliminary matching result corresponding to the data to be detected based on the obstacle difference information, includes: Calculate the deviation distance between the obstacle detection position and the obstacle standard position, and determine the preliminary matching result corresponding to the data to be detected based on the deviation distance and a preset distance threshold.

3. The method of claim 1, wherein, The obstacle detection information is the obstacle detection area, and the standard obstacle information is the obstacle standard area; The step of determining obstacle difference information based on the obstacle detection information and the standard obstacle information, and determining a preliminary matching result corresponding to the data to be detected based on the obstacle difference information, includes: Calculate the region crossover ratio (CBR) between the obstacle detection region and the obstacle standard region, and determine the preliminary matching result corresponding to the data to be detected based on the region CBR and a preset CBR threshold.

4. The method of claim 1, wherein, The acquisition of reference detection data corresponding to the data to be detected includes at least one of the following operations: Acquire a first preset number of frames of data to be detected that are adjacent to the acquisition time of the data to be detected and located before and / or after the data to be detected, and use them as reference detection data corresponding to the data to be detected. Acquire a second preset number of frames of data to be detected that are consecutive in acquisition time from the multiple frames of data to be detected.

5. The method of claim 4, wherein, Determining the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected includes at least one of the following operations: If the preliminary matching results corresponding to the data to be detected in the previous frame and the data to be detected in the next frame are not the same as the preliminary matching results of the data to be detected, the preliminary matching result corresponding to the data to be detected in the previous frame or the data to be detected in the next frame shall be taken as the target matching result of the data to be detected. If the preliminary matching results of the data to be detected in the next second preset number of consecutive frames that are adjacent to the data to be detected and located before or after the data to be detected are all different from the preliminary matching result of the data to be detected, and if the obstacle difference information corresponding to the preliminary matching result of the data to be detected is within a preset first information value range, then the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected. If the preliminary matching result corresponding to the data to be detected in the previous frame or the next frame is different from the preliminary matching result of the data to be detected, and the obstacle difference information corresponding to the preliminary matching result of the data to be detected in the previous frame or the next frame is within a preset second information value range, then the preliminary matching result of the reference detection data is taken as the target matching result of the data to be detected.

6. The method of claim 1, wherein, The step of determining the detector's detection performance metrics based on the target matching results corresponding to multiple frames of the data to be detected includes: Receiver operation feature curves are constructed based on the target matching results corresponding to multiple frames of the data to be detected, and the detection performance index of the detector is determined based on the area under the receiver operation feature curves.

7. An apparatus for determining the detection performance of a detector, the apparatus implementing the method for determining the detection performance of a detector as described in any one of claims 1-6, characterized in that, include: The detection data acquisition module is used to acquire multiple frames of detection data, input the multiple frames of detection data into the target detector respectively, and obtain the obstacle detection information corresponding to each frame of detection data output by the detector. The preliminary matching result determination module is used to obtain standard obstacle information corresponding to the data to be detected for each frame of the data to be detected, determine obstacle difference information based on the obstacle detection information and the standard obstacle information, and determine the preliminary matching result corresponding to the data to be detected based on the obstacle difference information. The target matching result determination module is used to obtain reference detection data corresponding to the data to be detected, and determine the target matching result of the data to be detected based on the preliminary matching result corresponding to the reference detection data and the preliminary matching result corresponding to the data to be detected, wherein the reference detection data is the data to be detected that is adjacent to the data to be detected; The detection performance index determination module is used to determine the detection performance index of the detector based on the target matching results corresponding to multiple frames of the data to be detected.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for determining the detector detection performance of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for determining the detection performance of the detector as described in any one of claims 1-6.

Images