A three-dimensional target perception and evaluation method and system

Abstract

This invention discloses a method and system for evaluating 3D target perception. When a test vehicle is driving on an open road, it acquires raw point cloud data from a LiDAR sensor synchronized with a reference time, 3D target perception data from the LiDAR sensor, and 3D target perception data from the sensor under test. Frames are extracted from the raw LiDAR point cloud data and the 3D target perception data at preset fixed intervals to obtain a ground truth dataset. The 3D target perception data from the sensor under test is converted to a new format to obtain a test dataset. The ground truth frames from the ground truth dataset are used to evaluate the test frames from the test dataset to obtain the evaluation result. This invention, based on a LiDAR sensor, enables data acquisition and evaluation of open road scenarios, thus expanding the testing scenarios and providing substantial data support for 3D target perception evaluation, thereby improving the reliability of various evaluation indicators for 3D target perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, more particularly, to a three-dimensional target perception evaluation method and system. BACKGROUND

[0002] Three-dimensional target perception is an important perception function in intelligent driving environment perception, therefore, evaluating three-dimensional target perception is of great significance for the development of perception algorithms, the perception performance and position accuracy test and evaluation of mass-produced vehicle sensors.

[0003] At present, cameras and millimeter wave sensors are mainly used for three-dimensional target perception. The existing method for evaluating the performance of three-dimensional target perception mainly includes: collecting data in a closed field, manually labeling each frame of three-dimensional target box of the reference data used for evaluation in the collected data to obtain more accurate 3DBBox, and taking the 3DBBox as the true value to obtain various evaluation indexes of the three-dimensional target according to the true value.

[0004] However, since the existing evaluation scheme collects data and evaluates in a closed field, the test scene is single and cannot test the open road scene, thereby resulting in that the three-dimensional target perception evaluation lacks a large amount of data support, and the credibility of various evaluation indexes of the three-dimensional target perception is not high. SUMMARY

[0005] Therefore, the present application discloses a three-dimensional target perception evaluation method and system to realize data collection and evaluation of the open road scene, so that the test scene is no longer single, a large amount of data support can be provided for the three-dimensional target perception evaluation, and the credibility of various evaluation indexes of the three-dimensional target perception is improved.

[0006] A three-dimensional target perception evaluation method, comprising:

[0007] When a test vehicle is driving on an open road, laser radar original point cloud data collected by a laser radar sensor synchronized with a reference time, laser radar three-dimensional target perception data, and three-dimensional target perception data of a sensor to be evaluated collected by a sensor to be evaluated are acquired, wherein the laser radar sensor and the sensor to be evaluated are pre-calibrated in space and synchronized in time;

[0008] The laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a true value data set;

[0009] The three-dimensional target perception data of the sensor to be evaluated is format-converted to obtain a to-be-tested data set;

[0010] Based on the timestamp file of the ground truth dataset and the timestamp file of the to-be-tested dataset, the most recent time of each ground truth frame and the corresponding to-be-tested value frame are matched, and each ground truth frame is used to evaluate the corresponding to-be-tested value frame according to a preset evaluation index, so as to obtain an evaluation result.

[0011] Optionally, the process of the spatial calibration of the laser radar sensor and the evaluated sensor includes:

[0012] The laser radar coordinate system and the evaluated sensor coordinate system are both calibrated to the vehicle coordinate system, and a corresponding homogeneous transformation matrix is obtained, so as to complete the spatial calibration between the laser radar sensor and the evaluated sensor.

[0013] Optionally, the process of the time synchronization of the laser radar sensor and the evaluated sensor includes:

[0014] The laser radar sensor and the evaluated sensor are time-synchronized by using the same device for time service.

[0015] Optionally, the laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a ground truth dataset, including:

[0016] The laser radar original point cloud data and the laser radar three-dimensional target perception data are synchronously frame-extracted at the preset fixed interval to obtain a pre-labeled laser radar dataset, wherein the pre-labeled laser radar dataset includes frame-extracted point cloud files, frame-extracted three-dimensional target label files, and a timestamp file containing frame timestamps, and the laser radar point cloud data files and the three-dimensional target label files of each frame are one-to-one corresponding in time sequence.

[0017] The pre-labeled laser radar dataset is checked and corrected to obtain the ground truth dataset.

[0018] Optionally, the pre-labeled laser radar dataset is checked and corrected to obtain the ground truth dataset, including:

[0019] The pre-labeled laser radar dataset is input into a labeling software for interactive addition, deletion, modification, and query to obtain the ground truth dataset.

[0020] Optionally, the to-be-tested dataset obtained by performing format conversion on the evaluated sensor three-dimensional target perception data satisfies the following conditions:

[0021] The format of the to-be-tested dataset is consistent with that of the ground truth target list.

[0022] The ID of the to-be-tested dataset is globally unique.

[0023] The reference coordinate system of the to-be-tested data set is consistent with the true value reference coordinate system;

[0024] The timestamp reference of each frame of data in the to-be-tested data set is consistent with the true value data.

[0025] Optionally, based on the timestamp file of the true value data set and the timestamp file of the to-be-tested data set, the most recent time of each true value frame and the corresponding to-be-tested value frame is matched, and each true value frame is used to evaluate the corresponding to-be-tested value frame to obtain an evaluation result, including:

[0026] Based on the timestamp file of the true value data set and the timestamp file of the to-be-tested data set, the most recent time of each true value frame and the corresponding to-be-tested value frame is matched, and each true value frame is used to evaluate the corresponding to-be-tested value frame to obtain an evaluation result, including:

[0027] Each target true value frame and the corresponding to-be-tested value frame are spatially correlated to obtain each ID correlation pair;

[0028] Based on each ID correlation pair, three-dimensional target perception related attribute information is extracted, and the preset evaluation index is statistically obtained based on the three-dimensional target perception related attribute information;

[0029] The preset evaluation index is obtained based on the preset evaluation index.

[0030] Optionally, the preset evaluation index includes a three-dimensional target perception detection performance index and a tracking performance index.

[0031] Optionally, it further includes:

[0032] The evaluation result is output in the form of an evaluation report, wherein the evaluation report includes a three-dimensional target perception overall evaluation result and each frame of intermediate calculation data.

[0033] A three-dimensional target perception evaluation system, comprising:

[0034] A synchronous data acquisition module is configured to acquire laser radar original point cloud data, laser radar three-dimensional target perception data collected by a laser radar sensor, and three-dimensional target perception data of a sensor under test collected by a sensor under test when a test vehicle is driving on an open road, wherein the laser radar sensor and the sensor under test are pre-calibrated and time-synchronized;

[0035] A true value data set establishment module is configured to frame the laser radar original point cloud data and the laser radar three-dimensional target perception data according to a preset fixed interval to obtain a true value data set;

[0036] A to-be-tested data set establishing module is configured to perform format conversion on the three-dimensional target perception data of the sensor under test to obtain a to-be-tested data set;

[0037] An evaluation module is configured to match each true value frame and a corresponding to-be-tested value frame based on the time stamp file of the true value data set and the time stamp file of the to-be-tested data set, and perform evaluation on the corresponding to-be-tested value frame according to a preset evaluation index by using each true value frame to obtain an evaluation result.

[0038] From the above technical solution, it can be seen that the present application discloses a three-dimensional target perception evaluation method and system. When a test vehicle is driving on an open road, laser radar original point cloud data collected by a laser radar sensor, laser radar three-dimensional target perception data, and three-dimensional target perception data of a sensor under test collected by a sensor under test are obtained. The laser radar sensor and the sensor under test are pre-calibrated in space and synchronized in time. The laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a true value data set. The three-dimensional target perception data of the sensor under test is format-converted to obtain a to-be-tested data set. Each true value frame and a corresponding to-be-tested value frame are matched based on the time stamp file of the true value data set and the time stamp file of the to-be-tested data set. The corresponding to-be-tested value frame is evaluated according to a preset evaluation index by using each true value frame to obtain an evaluation result. It can be seen that the present application realizes data collection and evaluation of an open road scene based on a laser radar sensor, so that the test scene is no longer single, a large amount of data support can be provided for three-dimensional target perception evaluation, and the credibility of each evaluation index of three-dimensional target perception is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the disclosed drawings.

[0040] Figure 1 A three-dimensional target perception evaluation method flowchart disclosed by the embodiments of the present application;

[0041] Figure 2 A true value data set determination method flowchart disclosed by the embodiments of the present application;

[0042] Figure 3 A three-dimensional target perception evaluation method flowchart disclosed by the embodiments of the present application;

[0043] Figure 4A structural schematic diagram of a three-dimensional target perception evaluation system disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0045] The embodiments of the present application disclose a three-dimensional target perception evaluation method and system. When a test vehicle is driving on an open road, laser radar original point cloud data collected by a laser radar sensor synchronized with a reference time, laser radar three-dimensional target perception data and three-dimensional target perception data of a sensor to be evaluated collected by the sensor to be evaluated are acquired. The laser radar sensor and the sensor to be evaluated are pre-calibrated in space and pre-synchronized in time. The laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a true value data set. The three-dimensional target perception data of the sensor to be evaluated is format-converted to obtain a to-be-tested data set. Based on a timestamp file of the true value data set and a timestamp file of the to-be-tested data set, each true value frame and a corresponding to-be-tested value frame closest in time are matched, and each true value frame is used to evaluate the corresponding to-be-tested value frame according to a preset evaluation index to obtain an evaluation result. It can be seen that, according to the present application, data collection and evaluation of an open road scene are realized based on a laser radar sensor, so that the test scene is no longer single, a large amount of data support can be provided for three-dimensional target perception evaluation, and the credibility of each evaluation index of three-dimensional target perception is improved.

[0046] Reference Figure 1 The embodiments of the present application disclose a three-dimensional target perception evaluation method flowchart, and the method comprises the following steps.

[0047] In step S101, when a test vehicle is driving on an open road, laser radar original point cloud data collected by a laser radar sensor synchronized with a reference time, laser radar three-dimensional target perception data and three-dimensional target perception data of a sensor to be evaluated collected by the sensor to be evaluated are acquired.

[0048] Preferably, the laser radar sensor and the sensor to be evaluated are pre-calibrated in space and pre-synchronized in time.

[0049] In actual application, the sensor to be evaluated can be a millimeter wave radar, a camera and a low-line laser radar.

[0050] Specifically, the process of calibrating the laser radar sensor and the sensor to be evaluated in space can comprise the following steps.

[0051] The laser radar coordinate system and the measured evaluation sensor coordinate system are both calibrated to the vehicle coordinate system to obtain respective corresponding homogeneous transformation matrices, and the spatial calibration between the laser radar sensor and the measured evaluation sensor is completed, which can be used for subsequent conversion of the true value data set and the to-be-measured data set to the same coordinate system.

[0052] The process of time synchronization of the laser radar sensor and the measured evaluation sensor can include:

[0053] The laser radar sensor and the measured evaluation sensor are time-synchronized by using the same device for time synchronization.

[0054] The present application can make the collected laser radar original point cloud data, laser radar three-dimensional target perception data and measured evaluation sensor three-dimensional target perception data based on the same time reference by using the same device for time synchronization of the laser radar sensor and the measured evaluation sensor, so as to perform timestamp matching based on the true value timestamp and the to-be-measured value timestamp of each frame subsequently.

[0055] In actual application, when the test vehicle is driving on an open road, the laser radar perception function of the test vehicle is turned on, and the laser radar original point cloud data and the laser radar three-dimensional target perception data of different scenes are collected online by the laser radar sensor.

[0056] Preferably, a vehicle-mounted industrial computer or an NTP (Network Time Protocol) server can be used for time synchronization.

[0057] It should be noted that the sensors of a general mass-produced vehicle directly output target lists in a vehicle bus protocol without outputting original data, and therefore, the measured evaluation sensor three-dimensional target perception data in the present embodiment can be target list data.

[0058] Step S102, frame extraction is performed on the laser radar original point cloud data and the laser radar three-dimensional target perception data according to a preset fixed interval to obtain a true value data set;

[0059] The value of the preset fixed interval is determined according to actual needs, such as 5s, which is not limited in the present application.

[0060] In actual application, frame extraction can be performed on all scene data in the laser radar original point cloud data and the laser radar three-dimensional target perception data, or frame extraction can be performed on a specific scene segment.

[0061] Step S103, format conversion is performed on the measured evaluation sensor three-dimensional target perception data to obtain a to-be-measured data set;

[0062] Since the sensors being evaluated typically output target list information using the vehicle bus protocol, and the target IDs are not globally unique, it is necessary to convert the format of the 3D target perception data of the sensors being evaluated to obtain the dataset to be tested.

[0063] Step S104: Based on the timestamp files of the truth dataset and the timestamp files of the test dataset, match the most recent truth frames and their corresponding test frames, and use each truth frame to evaluate the corresponding test frames according to the preset evaluation indicators to obtain the evaluation results.

[0064] The preset evaluation indicators include: 3D target perception and detection performance indicators and tracking performance indicators.

[0065] Common performance metrics for 3D target perception and detection include: precision (P), recall (R), AP value, MAP value, position deviation, heading angle deviation, and velocity deviation.

[0066] Common tracking performance metrics include: Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), and ID switch count.

[0067] In summary, this invention discloses a method for evaluating three-dimensional target perception. When a test vehicle is driving on an open road, it acquires raw point cloud data from a LiDAR sensor synchronized with a reference time, three-dimensional target perception data from the LiDAR sensor, and three-dimensional target perception data from the sensor under test. The LiDAR sensor and the sensor under test are pre-calibrated spatially and synchronized in time. Frames are extracted from the raw point cloud data and the three-dimensional target perception data at preset fixed intervals to obtain a ground truth dataset. The three-dimensional target perception data from the sensor under test is converted to a test dataset. Based on the timestamp files of the ground truth dataset and the test dataset, the most recent ground truth frames and their corresponding test frames are matched. The corresponding test frames are then evaluated using preset evaluation indicators to obtain the evaluation results. Therefore, this invention enables data acquisition and evaluation of open road scenarios based on LiDAR sensors, thus expanding the testing scenarios and providing substantial data support for three-dimensional target perception evaluation, thereby improving the reliability of various evaluation indicators for three-dimensional target perception.

[0068] In addition, the detection accuracy of the laser radar sensor is higher than that of the camera and the millimeter wave sensor, and by spatial calibration and time synchronization of the laser radar sensor and the evaluation sensor, the laser radar sensor and the evaluation sensor are based on the same time reference, so that the accuracy of the evaluation of the true value data set obtained based on the laser radar sensor to the to-be-evaluated data set obtained based on the evaluation sensor is ensured.

[0069] To further optimize the above embodiment, referring to Figure 2 The embodiment of the present application discloses a true value data set determination method flowchart, that is, step S102 can specifically include:

[0070] Step S201, synchronously frame the laser radar original point cloud data and the laser radar three-dimensional target perception data according to a preset fixed interval to obtain a pre-labeled laser radar data set;

[0071] The value of the preset fixed interval is determined according to actual needs, for example, 5s, which is not limited in the present application.

[0072] The pre-labeled laser radar data set includes: each frame of point cloud file, each frame of three-dimensional target label file and time stamp file containing each frame of time stamp, and each frame of laser radar point cloud data file, three-dimensional target label file is one-to-one corresponding in time sequence.

[0073] Step S202, correcting the pre-labeled laser radar data set to obtain a true value data set.

[0074] Specifically, the pre-labeled laser radar data set is input into a labeling software for interactive addition, deletion, modification and query to obtain a true value data set.

[0075] The interactive addition, deletion, modification and query specifically include: addition, deletion, modification and query.

[0076] When collecting the laser radar original point cloud data and the laser radar three-dimensional target perception data online, the laser radar three-dimensional target perception data in the scene has been perceived and output, but the perception result inevitably has some false detection or missed detection, so the perception result can be used as the pre-labeled laser radar data set, and after checking and correcting to check, modify, add and delete, it is saved as the final true value data set.

[0077] In practical applications, pre-annotated LiDAR datasets can be input into LiDAR annotation software. The software can display the raw point cloud data of the LiDAR, the 3DBBox (Bounding Box) of the LiDAR 3D target perception data, and attribute information for each frame in the dataset. Attribute information includes: ID (globally unique), type, 3D size of the 3DBBox, position, heading angle, velocity, etc. Furthermore, it allows viewing the 3DBBox and point cloud envelope of the LiDAR 3D target perception data, as well as its attribute information. Users can also modify all attribute information and add or delete LiDAR 3D target perception data.

[0078] In summary, the lidar sensor in this invention can scan the three-dimensional point cloud of the target (i.e., the original lidar point cloud data), thereby facilitating the visualization of the three-dimensional target point cloud and the perceived 3DBBox, making it convenient for manual viewing, and allowing comparison of the pose of the 3DBBox of the ground truth dataset and the dataset to be tested with the pose of the real point cloud.

[0079] The process of establishing the true value dataset involves querying, modifying, adding, and deleting data based on the algorithm perception results as a pre-labeled LiDAR dataset, thereby greatly improving the labeling efficiency and quality. Based on this, the present invention can evaluate open road scene data and present the evaluation report results.

[0080] In step S103, the dataset to be tested obtained by converting the format of the 3D target perception data of the sensor being evaluated meets the following conditions:

[0081] (1) The format of the dataset to be tested is consistent with the list of true values.

[0082] (2) The IDs of the dataset to be tested are globally unique. In practical applications, a preset strategy can be used to ensure that the IDs of the obtained dataset to be tested are globally unique. The preset strategy can be determined according to actual needs.

[0083] (3) The reference coordinate system of the dataset to be tested is consistent with the true reference coordinate system. In practical applications, a spatial calibration matrix can be used to ensure that the reference coordinate system of the dataset to be tested is consistent with the true reference coordinate system.

[0084] (4) The timestamp reference of each frame of data in the dataset to be tested is consistent with the true data.

[0085] To further optimize the above embodiments, see [link to relevant documentation]. Figure 3 The flowchart of a method for determining evaluation results disclosed in this embodiment of the invention, namely step S104, includes:

[0086] Step S301: Based on the timestamp files of the truth dataset and the timestamp files of the dataset to be tested, match the most recent truth frames and their corresponding frames to be tested, and perform target interpolation on each truth frame to obtain the target truth frame.

[0087] Because the frequencies of the lidar sensor and the sensor under test may be inconsistent, the timestamps of the matched frame pointers and the measured value frames will have some interpolation. When the target velocity acquired by the sensor under test is large, the time error will cause a large spatial error. Therefore, for each ground truth data, its position data is updated by interpolating its velocity and timestamp to obtain the target ground truth frame for subsequent processing.

[0088] Step S302: Spatial association is performed on each target ground truth frame and the corresponding test value frame to obtain each ID association pair;

[0089] In practical applications, the Hungarian matching algorithm can be used to spatially associate each truth frame with its corresponding test frame to obtain each ID association pair.

[0090] Step S303: Extract the three-dimensional target perception-related attribute information based on each ID association pair, and obtain the preset evaluation index based on the three-dimensional target perception-related attribute information.

[0091] Step S304: Obtain the evaluation results based on the preset evaluation indicators.

[0092] In practical applications, the evaluation results can be output and displayed in the form of an evaluation report, which includes: the overall evaluation results of 3D target perception and the intermediate calculation data for each frame.

[0093] The overall evaluation results of 3D target perception are used to view the overall perception performance. Intermediate computational data for each frame, such as the evaluation results and target association status, can be used by developers to view and filter frames with unsatisfactory detection performance, and to examine specific data in detail, analyze and troubleshoot problems, which is helpful for the iterative development of the algorithm. It also allows developers to analyze algorithm failure scenarios based on the evaluation report and conduct iterative algorithm development.

[0094] In practical applications, in addition to displaying the evaluation report, for user convenience, the ground truth point cloud data, ground truth 3DBBox and attributes, and test value 3DBBox and attributes can also be displayed frame by frame.

[0095] Corresponding to the above method embodiments, the present invention also discloses a three-dimensional target perception and evaluation system.

[0096] See Figure 4 The present invention discloses a structural schematic diagram of a three-dimensional target perception and evaluation system, which includes:

[0097] The synchronous data acquisition module 401 is used to acquire the raw point cloud data of the lidar sensor, the lidar three-dimensional target perception data, and the three-dimensional target perception data of the sensor under test collected by the lidar sensor synchronized with the reference time when the test vehicle is driving on an open road.

[0098] Among them, the lidar sensor and the sensor under test were pre-calibrated spatially and synchronized in time.

[0099] In practical applications, an on-board industrial control computer can be used to synchronize the time of the lidar sensor and the sensor under test.

[0100] In practical applications, the sensors being evaluated can be: millimeter-wave radar, cameras, and low-line-count lidar.

[0101] The truth dataset creation module 402 is used to extract frames from the original point cloud data and the three-dimensional target perception data of the lidar according to a preset fixed interval to obtain the truth dataset.

[0102] When collecting raw point cloud data and 3D target perception data from LiDAR online, the 3D target perception data of LiDAR in the scene has already been output for perception. However, some false detections or missed detections are inevitable in the perception results. Therefore, the perception results can be used as a pre-labeled LiDAR dataset. After verification, the dataset can be checked, modified, added, and deleted, and then saved as the final ground truth dataset.

[0103] The test dataset creation module 403 is used to convert the format of the three-dimensional target perception data of the sensor being evaluated to obtain the test dataset.

[0104] The evaluation module 404 is used to match the timestamp files of the truth dataset and the timestamp files of the test dataset to obtain the most recent truth frames and the corresponding test frames, and to evaluate the corresponding test frames according to preset evaluation indicators using the truth frames to obtain the evaluation results.

[0105] The evaluation module 404 uses the truth dataset to evaluate the dataset under test according to preset evaluation indicators. The process mainly includes the following steps:

[0106] 1) Data loading

[0107] In practical applications, before the evaluation module 404 evaluates the dataset to be tested according to preset evaluation indicators using the true value dataset, it needs to load data into both the true value dataset and the dataset to be tested.

[0108] 2) Timestamp matching and truth target interpolation

[0109] Specifically, based on the timestamp files of the truth dataset and the timestamp files of the test dataset, the most recent truth frames and their corresponding test frames are matched, and target interpolation is performed on each truth frame to obtain the target truth frame.

[0110] 3) Spatial association

[0111] Specifically, spatial association is performed between each truth frame and its corresponding test frame to obtain each ID association pair.

[0112] 4) Calculation of preset evaluation indicators

[0113] Specifically, based on the association of each ID, relevant attribute information for 3D target perception is extracted, and preset evaluation indicators are obtained based on the statistical analysis of the relevant attribute information for 3D target perception.

[0114] 5) Evaluation Report

[0115] Specifically, the evaluation results are obtained based on preset evaluation indicators, and the evaluation results are output and displayed in the form of an evaluation report.

[0116] In summary, this invention discloses a 3D target perception evaluation system. When a test vehicle is driving on an open road, it acquires raw point cloud data from a LiDAR sensor synchronized with a reference time, 3D target perception data from the LiDAR sensor, and 3D target perception data from the sensor under test. The LiDAR sensor and the sensor under test are pre-calibrated and synchronized in time. Frames are extracted from the raw point cloud data and 3D target perception data at preset fixed intervals to obtain a ground truth dataset. The 3D target perception data from the sensor under test is converted to a test dataset. Based on the timestamp files of the ground truth dataset and the test dataset, the most recent ground truth frames and their corresponding test frames are matched. The corresponding test frames are then evaluated using preset evaluation indicators to obtain the evaluation results. Therefore, this invention, based on a LiDAR sensor, enables data acquisition and evaluation of open road scenarios, thus expanding the testing scenarios and providing substantial data support for 3D target perception evaluation, thereby improving the reliability of various evaluation indicators for 3D target perception.

[0117] In addition, compared with cameras and millimeter-wave sensors, lidar sensors have higher detection accuracy. By spatially calibrating and synchronizing the lidar sensor and the sensor under test, the lidar sensor and the sensor under test are based on the same time reference, thereby ensuring the accuracy of the evaluation of the test dataset based on the test dataset obtained from the lidar sensor.

[0118] It should be noted that the specific working principles of each component in the system embodiment can be found in the corresponding sections of the method embodiment, and will not be repeated here.

[0119] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0121] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for assessing three-dimensional target perception, the method comprising: The method comprises the following steps: When the test vehicle is driving on an open road, laser radar original point cloud data collected by a laser radar sensor synchronized with a reference time, laser radar three-dimensional target perception data, and three-dimensional target perception data collected by a sensor under test are acquired, wherein the laser radar sensor and the sensor under test are pre-calibrated and time-synchronized in space; The laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a ground truth data set; The three-dimensional target perception data collected by the sensor under test is format-converted to obtain a data set under test; Based on the timestamp files of the ground truth data set and the data set under test, the most recent true value frame and the corresponding test value frame are matched, and the test value frame corresponding to each true value frame is evaluated according to a preset evaluation index to obtain an evaluation result, which comprises: Based on the timestamp files of the ground truth data set and the data set under test, the most recent true value frame and the corresponding test value frame are matched, and the target true value frame is obtained by target interpolation of each true value frame; The target true value frame and the corresponding test value frame are spatially correlated to obtain each ID correlation pair; Based on each ID correlation pair, three-dimensional target perception related attribute information is extracted, and the preset evaluation index is statistically obtained based on the three-dimensional target perception related attribute information; The evaluation result is obtained based on the preset evaluation index, and the preset evaluation index comprises a three-dimensional target perception detection performance index and a tracking performance index.

2. The three-dimensional target perception assessment method of claim 1, wherein, The process of calibrating the laser radar sensor and the sensor under test in space comprises: The laser radar coordinate system and the sensor under test coordinate system are calibrated to the vehicle coordinate system to obtain their respective homogeneous transformation matrices, and the spatial calibration between the laser radar sensor and the sensor under test is completed.

3. The three-dimensional target perception assessment method of claim 1, wherein, The process of time-synchronizing the laser radar sensor and the sensor under test comprises: The laser radar sensor and the sensor under test are time-synchronized by using the same device.

4. The three-dimensional target perception assessment method of claim 1, wherein, The laser radar original point cloud data and the laser radar three-dimensional target perception data are frame-extracted at a preset fixed interval to obtain a ground truth data set, which comprises: The laser radar original point cloud data and the laser radar three-dimensional target perception data are synchronously frame-extracted at the preset fixed interval to obtain a pre-labeled laser radar data set, wherein the pre-labeled laser radar data set comprises frame point cloud files, frame three-dimensional target label files, and timestamp files containing frame timestamps, and the laser radar point cloud data files and the three-dimensional target label files of each frame are time-sequentially and one-to-one corresponding; The pre-labeled laser radar data set is checked and corrected to obtain the ground truth data set.

5. The three-dimensional target perception assessment method of claim 4, wherein, The pre-labeled laser radar data set is checked and corrected to obtain the ground truth data set, which comprises: The pre-labeled lidar dataset is input into a labeling software for interactive addition, deletion, modification and query to obtain the ground truth dataset.

6. The three-dimensional target perception assessment method of claim 1, wherein, The measured dataset obtained by format conversion of the three-dimensional target perception data of the sensor under test satisfies the following conditions: The format of the measured dataset is consistent with the ground truth target list; The ID of the measured dataset is globally unique; The reference coordinate system of the measured dataset is consistent with the ground truth reference coordinate system; The timestamp reference of each frame of data in the measured dataset is consistent with the ground truth data.

7. The three-dimensional target perception assessment method of claim 1, wherein, Further comprising: The evaluation result is output in the form of an evaluation report, wherein the evaluation report includes a three-dimensional target perception overall evaluation result and each frame of intermediate calculation data.

8. A three-dimensional target perception assessment system, comprising: Comprising: A synchronous data acquisition module, configured to acquire laser radar original point cloud data collected by a laser radar sensor, laser radar three-dimensional target perception data, and three-dimensional target perception data of a sensor under test collected by the sensor under test when a test vehicle is driving on an open road, wherein the laser radar sensor and the sensor under test are pre-calibrated in space and time-synchronized; A ground truth dataset establishment module, configured to frame the laser radar original point cloud data and the laser radar three-dimensional target perception data according to a preset fixed interval to obtain a ground truth dataset; A measured dataset establishment module, configured to convert the three-dimensional target perception data of the sensor under test to obtain a measured dataset; An evaluation module, configured to match each ground truth frame and a corresponding measured value frame that are closest in time based on a timestamp file of the ground truth dataset and a timestamp file of the measured dataset, and evaluate the corresponding measured value frame according to a preset evaluation index using each ground truth frame to obtain an evaluation result; The evaluation module is specifically configured to: Match each ground truth frame and a corresponding measured value frame that are closest in time based on a timestamp file of the ground truth dataset and a timestamp file of the measured dataset, and perform target interpolation on each ground truth frame to obtain a target ground truth frame; Perform spatial correlation on each target ground truth frame and the corresponding measured value frame to obtain each ID correlation pair; Extract three-dimensional target perception related attribute information based on each ID correlation pair, and statistically obtain the preset evaluation index based on the three-dimensional target perception related attribute information; Obtain the evaluation result based on the preset evaluation index, wherein the preset evaluation index includes a three-dimensional target perception detection performance index and a tracking performance index.

Images