Detection performance evaluation method and system for vehicle-mounted laser radar

Abstract

The invention relates to the technical field of vehicle-mounted equipment evaluation, and discloses a detection performance evaluation method and system for a vehicle-mounted laser radar, and the method comprises the following steps: 1, configuring a test vehicle and a test site; 2, installing a test auxiliary mechanism on the test vehicle, and deploying the tested radar; 3, the automatic driving robot controls the test vehicle to run according to a preset test path and a test working condition; 4, in the running process of the test vehicle, collecting point cloud data of the tested radar and vehicle running data of the to-be-tested vehicle at different time nodes; step 5, performing preliminary calibration according to the collected point cloud data, and then extracting target object information in the point cloud data; and in combination with standard driving data recorded by the RT equipment and image data recorded by the forward camera module, the point cloud quality and the target sensing performance of the tested radar are evaluated. The real performance of the laser radar can be objectively evaluated, the evaluation accuracy is high, and the evaluation result is reliable.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted equipment evaluation, and in particular to a detection performance evaluation method and system for a vehicle-mounted laser radar. Background Art

[0002] As an important component of the perception system of intelligent driving vehicles, LiDAR can not only provide high-precision perception information, but also help vehicles understand complex driving environments, thereby achieving safe and efficient autonomous driving. In order to ensure the accuracy of perception and its reliability and performance advantages in various environments, performance evaluation of LiDAR is essential.

[0003] The evaluation of LiDAR hardware in China is currently mainly limited to the hardware performance testing stage in the laboratory environment. Although this testing method can provide a certain technical parameter evaluation, it cannot fully reflect the performance of LiDAR in the real driving environment. This is because most of the existing testing standards focus on component-level testing, that is, only focusing on the performance of the LiDAR itself, while ignoring its comprehensive performance when working with other vehicle systems in actual application scenarios. In component-level testing, a radar darkroom environment is usually used to evaluate the basic performance of LiDAR, such as detection distance, angular resolution, measurement accuracy, and anti-interference ability. These tests can provide a preliminary technical parameter benchmark for LiDAR, but they do not fully consider the various challenges that LiDAR may encounter in the complex and changeable real road environment. For example, how does LiDAR perform in rainy and snowy weather or under strong direct sunlight? How much will its detection accuracy be affected under different lighting conditions? These issues are often not given enough attention in current testing standards.

[0004] In addition, the existing testing schemes have not fully considered the specific performance of LiDAR after it is installed on the car. In fact, once LiDAR is integrated into the autonomous driving system, it needs to work with other sensors (such as cameras, millimeter-wave radars, etc.) to ensure that the vehicle can drive safely in various complex traffic environments. This requires LiDAR to have not only good hardware performance, but also strong target recognition capabilities and the ability to adapt to different environmental conditions. Summary of the invention

[0005] The present invention aims to provide a detection performance evaluation method and system for a vehicle-mounted laser radar, which can objectively evaluate the true performance of the laser radar with high evaluation accuracy and reliable evaluation results.

[0006] To achieve the above objectives, the present invention provides the following basic solutions.

[0007] Solution 1

[0008] A method for evaluating the detection performance of a vehicle-mounted laser radar comprises the following steps:

[0009] Step 1: Configure the test vehicle and test site;

[0010] Step 2: Install a test auxiliary mechanism on the test vehicle and deploy the radar under test; the test auxiliary mechanism includes an RT device, a forward camera module and an autonomous driving robot;

[0011] Step 3, importing a preset test path and test conditions into the autonomous driving robot, and the autonomous driving robot controls the test vehicle to run according to the preset test path and test conditions;

[0012] Step 4, during the operation of the test vehicle, collecting the point cloud data of the tested radar and the vehicle driving data of the tested vehicle at different time nodes;

[0013] Step 5: Perform preliminary calibration based on the collected point cloud data, and then extract the target object information in the point cloud data through the AI ​​algorithm; evaluate the point cloud quality and target perception performance of the radar under test in combination with the standard driving data recorded by the RT device and the image data recorded by the forward camera module; and, when evaluating the target perception performance, use a preset perception model to assist in the evaluation; the preset perception model is pre-trained using multi-source radar data.

[0014] Solution 2

[0015] A vehicle-mounted laser radar detection performance evaluation system, used to execute a vehicle-mounted laser radar detection performance evaluation method as described in Scheme 1; comprising a test auxiliary mechanism, a data acquisition module, a point cloud processing module and an evaluation module;

[0016] The test auxiliary mechanism includes an RT device, a forward camera module and an autonomous driving robot; the data acquisition module establishes communication connections with the test auxiliary mechanism, the radar under test and the test vehicle respectively to collect standard driving data, image data, point cloud data and vehicle driving data; the point cloud processing module is used to process the point cloud data and add annotations to the point cloud data, and perform point cloud data comparison based on the image data; a preset perception model is set in the evaluation module; the evaluation module is used to evaluate the point cloud quality and target perception performance of the laser radar; and when evaluating the target perception performance, the preset perception model is used to assist the evaluation; the preset perception model is pre-trained with multi-source radar data.

[0017] The working principle and advantages of the present invention are:

[0018] First, in the evaluation method of this solution, the radar under test is installed on the test vehicle for testing, which fully considers the specific performance of the laser radar after being installed on the vehicle, can provide a real operating environment for the radar under test, actually simulate the state of the radar installed on the vehicle, and build more realistic vibration interference and road environment conditions, so that the evaluation results are more realistic and reliable. In addition, during the test process, the use of an autonomous driving robot to control the test vehicle can ensure the accuracy and repeatability of the test process and avoid possible control errors in actual vehicle testing. The RT device is used to collect vehicle driving data in real time, which can effectively record the position of the vehicle; and the forward camera module records image data, which can record the real information of the forward target in real time, providing a reliable reference for the subsequent point cloud quality and target perception performance evaluation. During the evaluation process, by comparing the information between the radar under test and the RT device, the relative position, speed, angle and other information of the radar under test on the test vehicle can be accurately confirmed to facilitate the verification of the point cloud data. By comparing the information between the radar under test and the forward camera module, target object recognition can be performed, and then the point cloud quality and target object recognition under actual vehicle working conditions can be analyzed. Performance evaluation can be completed based on the point cloud quality and target object recognition, and the evaluation factors are comprehensive and objective.

[0019] Second, when evaluating the target perception performance, this solution uses a preset perception model for auxiliary evaluation, which can efficiently complete the target perception performance analysis. In particular, the preset perception model is pre-trained with multi-source radar data (that is, point cloud data of open roads from multiple different radar models are used to build the algorithm's preliminary data training model), which is more fair to the algorithm model training, has no bias towards a certain radar, and can objectively evaluate the true performance of the lidar. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a method flow of a detection performance evaluation method and system embodiment of a vehicle-mounted laser radar of the present invention;

[0021] Figure 2 A schematic diagram of the structure of a test vehicle in an embodiment of a detection performance evaluation method and system for a vehicle-mounted laser radar of the present invention;

[0022] Figure 3 A schematic diagram of the test bench structure of a detection performance evaluation method and system embodiment of a vehicle-mounted laser radar of the present invention. DETAILED DESCRIPTION

[0023] The following is a further detailed description through specific implementation methods:

[0024] The embodiment is basically as shown in the attached Figure 1 As shown: A detection performance evaluation method for a vehicle-mounted laser radar includes the following steps:

[0025] Step 1: Configure the test vehicle and test site.

[0026] Specifically, configuring the test vehicle includes: preparing the test vehicle to ensure that the vehicle weight and tire pressure meet the design requirements, and the oil level or power is at a rated state. After the vehicle preparation is completed, the test vehicle information is recorded.

[0027] The configuration of the test site includes: configuring the test site according to the test scenario requirements in the evaluation requirements. The test scenario requirements may include: indoor scenes (such as parking lots), outdoor scenes (such as closed sites), fast-moving scenes (such as high-speed roads), fast-rotating scenes (such as continuous curves), moving object scenes (such as driving scenes with following vehicles / pedestrians crossing events), occlusion scenes (such as driving scenes under rain / fog / haze weather conditions), and light and dark change scenes (such as driving scenes including tunnel entry and exit events). The configuration of the test site is shown in Table 1:

[0028] Table 1 Test site configuration example

[0029]

[0030] Step 2: Install the test auxiliary mechanism on the test vehicle and deploy the radar under test, such as Figure 2 As shown; the test auxiliary mechanism includes RT equipment, a forward camera module and an autonomous driving robot.

[0031] When installing the test auxiliary mechanism, the forward camera module is installed in the middle of the front windshield of the test vehicle, and the RT device and the autonomous driving robot are installed in the vehicle. The autonomous driving robot includes an inertial navigation system, a steering robot for controlling the steering of the test vehicle, and a combined brake and throttle robot for controlling the travel of the test vehicle. The RT device refers to the vehicle data acquisition device.

[0032] In this step, after the test auxiliary mechanism is installed, the target object and the test vehicle are deployed in the test site; the longitudinal distance between the test vehicle and the target object is zeroed to ensure that the relative position of the target object and the test vehicle is accurate and the fluctuation range of the longitudinal distance is within ±0.2%; and the speed of the test vehicle and the autonomous driving robot is calibrated to ensure that the speed measurement of the vehicle is consistent with the speed information received by the autonomous driving robot.

[0033] When deploying the radar under test, the radar under test is first installed and fixed on the test bench, and then the test bench is fixed on the top of the test vehicle, and the vehicle circuit deployment and position calibration of the radar under test are completed, and the relative coordinate system position difference between the radar under test and the test vehicle is confirmed to reduce the test error caused by the installation. In this embodiment, the test bench used is as follows Figure 3 As shown, in actual applications, the structure of the test bench can be appropriately adjusted according to the installation requirements of the radar being tested.

[0034] Step 3: Import the preset test path and test conditions into the autonomous driving robot, place the test vehicle in the corresponding test scene, and set the coordinate origin at the center of the road where the test starts. The autonomous driving robot controls the test vehicle to run according to the preset test path and test conditions.

[0035] An autonomous driving robot is used here to control the test vehicle, which can ensure the accuracy and repeatability of the test process and effectively record the vehicle's position.

[0036] Step 4: During the operation of the test vehicle, the point cloud data of the tested radar and the vehicle driving data of the tested vehicle at different time points are collected.

[0037] Step 5: Perform preliminary calibration based on the collected point cloud data, and then extract the target information in the point cloud data through the AI ​​algorithm. Here, the extracted target information includes the target type, target position, target angle relative to the radar, the straight-line distance between the target and the radar, and the distance range within which the target is recognized by the radar.

[0038] The point cloud quality and target perception performance of the tested radar are evaluated by combining the standard driving data recorded by the RT device and the image data recorded by the forward camera module. In addition, when evaluating the target perception performance, a preset perception model is used to assist the evaluation. The preset perception model is pre-trained with multi-source radar data.

[0039] The standard driving data recorded by the RT device includes: the position of the test vehicle, the speed of the test vehicle, and the pitch angle of the test vehicle. This standard driving data can be used as a true value reference in subsequent point cloud annotation for comparison with the test data.

[0040] The multi-source radar data includes: point cloud data collected by different types of laser radars with a single-frame beam of 128 lines and a wavelength of 905nm, and laser radars with a single-frame beam of 128 lines and a wavelength of 1550nm; and the collection scenes of point cloud data cover two types of roads, highways and urban roads, and cover at least four weather conditions. In this embodiment, a total of 5 different types of laser radars are selected; the weather conditions covered include sunny days, rainy days, foggy days, and nights. And the number of valid frames of point cloud data collected by each laser radar must be no less than 10,000 frames.

[0041] When using multi-source radar data for training, the following steps are included:

[0042] The multi-source radar data is cleaned and labeled to form a data set, and the data set is divided into a training set and a validation set. The labels include: target type, target position, target angle relative to the radar, straight-line distance between the target and the radar, and the distance range within which the target is identified by the radar.

[0043] The training set is divided into N groups of training subsets according to the radar model; the N groups of training subsets are used to train the preset perception model, which is built based on the YOLO network. And N specific models are obtained accordingly; and the training set is used to train the preset perception model to obtain the basic model.

[0044] The validation set is used to validate the basic model and N specific models respectively; and the model with the best performance is selected as the final preset perception model.

[0045] Specifically, in this embodiment, when evaluating the point cloud quality and target perception performance of the tested radar, the reference evaluation criteria are shown in Table 2 and Table 3. The evaluation data has two dimensions, one is the training data score when the radar training model is used, and the other is the data score when the radar is tested on a real vehicle. The data score ratio under each dimension is based on Table 2, and the evaluation level is standardized according to Table 3.

[0046] Table 2 Evaluation score ratio

[0047]

[0048] Table 3 Examples of scores and ratings

[0049]

[0050] In addition, when evaluating the point cloud quality and target perception performance of the tested radar, the following steps are performed:

[0051] (1) After the training of the preset perception model is completed, the training score result of the preset perception model for a single radar under test (the point cloud data of the radar under test is used as the model input, and the radar training score result is used as the model output) is used as the first dimension evaluation basis, and the scores corresponding to different evaluation items (target missed recognition rate, single target data accuracy-size, single target data accuracy-angle, and the number of point clouds in a single frame of a long-distance target) are obtained according to the analysis of the training score results; wherein, during the analysis, the standard driving data recorded by the RT device and the image data recorded by the forward camera module (including the reference target data identified based on the image data, such as the number, size, type, etc.) are used as the comparison benchmark.

[0052] (2) After the actual vehicle test is completed, the target object information extracted from the collected point cloud data is used as the basis for the second dimension evaluation, and the scores corresponding to different evaluation items (target object missed recognition rate, single target object data accuracy-size, single target object data accuracy-angle, and the number of point clouds in a single frame of a long-distance target object) are obtained based on the target object information analysis; in the analysis, the standard driving data recorded by the RT device and the image data recorded by the forward camera module (including the reference target object data identified based on the image data, such as the number, size, type, etc.) are also used as the comparison benchmark.

[0053] (3) Summarize the evaluation scores in the above two steps and convert them into score rates. Refer to Table 3 to obtain the comprehensive evaluation results of the tested radar.

[0054] This embodiment also provides a vehicle-mounted laser radar detection performance evaluation system, which is used to execute a vehicle-mounted laser radar detection performance evaluation method as described in Scheme 1; it includes a test auxiliary mechanism, a data acquisition module, a point cloud processing module and an evaluation module;

[0055] The test auxiliary mechanism includes an RT device, a forward camera module and an autonomous driving robot; the data acquisition module establishes communication connections with the test auxiliary mechanism, the radar under test and the test vehicle respectively to collect standard driving data, image data, point cloud data and vehicle driving data; the point cloud processing module is used to process the point cloud data and add annotations to the point cloud data, and perform point cloud data comparison based on the image data; a preset perception model is set in the evaluation module; the evaluation module is used to evaluate the point cloud quality and target perception performance of the laser radar; and when evaluating the target perception performance, the preset perception model is used to assist the evaluation; the preset perception model is pre-trained with multi-source radar data.

[0056] The detection performance evaluation method and system of a vehicle-mounted laser radar provided in this embodiment can objectively evaluate the true performance of the laser radar, with a high evaluation accuracy and reliable evaluation results. In particular, first, this solution breaks through the limitations of component-level testing in conventional laser radar testing, and carries out laser radar testing on a real vehicle for testing, which is more realistic. In addition, this solution overcomes the problems of low repeatability and poor reliability that may exist in real vehicle testing (that is, the consistency of vehicle control is difficult to maintain in conventional vehicle testing). The use of an automatic driving robot to control the test vehicle can ensure the accuracy and repeatability of the test process and avoid possible control errors in real vehicle testing. Second, this solution specially designs a general perception algorithm, which does not have a preference for a certain radar, and can objectively evaluate the true performance of the laser radar from a third-party perspective. Third, this solution designs a two-layer evaluation standard, which can evaluate the detection performance of the laser radar from different angles, and the comprehensive evaluation results are more reliable.

[0057] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field are aware of all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement the scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for the ordinary technicians in the relevant field to implement this application. It should be pointed out that for the technicians in this field, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for evaluating the detection performance of a vehicle-mounted laser radar, characterized in that: The following steps are involved: Step 1: Configure the test vehicle and test site; Step 2: Install a test auxiliary mechanism on the test vehicle and deploy the radar under test; the test auxiliary mechanism includes an RT device, a forward camera module and an autonomous driving robot; Step 3, importing a preset test path and test conditions into the autonomous driving robot, and the autonomous driving robot controls the test vehicle to run according to the preset test path and test conditions; Step 4, during the operation of the test vehicle, collecting the point cloud data of the tested radar and the vehicle driving data of the tested vehicle at different time nodes; Step 5: Perform preliminary calibration based on the collected point cloud data, and then extract the target object information in the point cloud data through the AI ​​algorithm; evaluate the point cloud quality and target perception performance of the radar under test in combination with the standard driving data recorded by the RT device and the image data recorded by the forward camera module; and, when evaluating the target perception performance, use a preset perception model to assist in the evaluation; the preset perception model is pre-trained using multi-source radar data.

2. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1 is characterized in that: The autonomous driving robot includes an inertial navigation system, a steering robot for controlling the steering of a test vehicle, and a combined brake-throttle robot for controlling the travel of the test vehicle.

3. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1 is characterized in that: In step 2, a target object and a test vehicle are deployed in a test field; and the longitudinal distance between the test vehicle and the target object is zeroed to ensure that the relative positions of the target object and the test vehicle are accurate and that the fluctuation range of the longitudinal distance is within ±0.2%; and the speed of the test vehicle and the autonomous driving robot is calibrated to ensure that the speed measurement of the vehicle is consistent with the speed information received by the autonomous driving robot.

4. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1, characterized in that: In step 2, when deploying the radar under test, first install and fix the radar under test on the test bench, then fix the test bench on the top of the test vehicle, and complete the vehicle circuit deployment and position calibration of the radar under test, and confirm the relative coordinate system position difference between the radar under test and the test vehicle.

5. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1, characterized in that: Standard driving data recorded by the RT equipment include: position, speed and pitch angle of the test vehicle.

6. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1, characterized in that: The multi-source radar data includes: point cloud data collected by different models of laser radars with a single-frame beam of 128 lines and a wavelength of 905nm, and a laser radar with a single-frame beam of 128 lines and a wavelength of 1550nm; and the collection scenarios of the point cloud data cover two types of roads, highways and urban roads, and cover at least four weather conditions.

7. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1, characterized in that: When using multi-source radar data for training, the following steps are included: Perform data cleaning on the multi-source radar data and add labels to the cleaned multi-source radar data to form a data set; and divide the data set into a training set and a validation set; The training set is divided into N groups of training subsets according to the radar model; the preset perception models are trained using the N groups of training subsets respectively, and N specific models are obtained correspondingly; and the preset perception model is trained using the training set to obtain a basic model; The validation set is used to validate the basic model and N specific models respectively; and the model with the best performance is selected as the final preset perception model.

8. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 1, characterized in that: The preset perception model is built based on the YOLO network.

9. The detection performance evaluation method of a vehicle-mounted laser radar according to claim 7, characterized in that: The label includes: target type, target position, target angle relative to the radar, straight-line distance between the target and the radar, and the distance range within which the target is identified by the radar.

10. A detection performance evaluation system for a vehicle-mounted laser radar, characterized in that: Used to execute a detection performance evaluation method for a vehicle-mounted laser radar as described in any one of claims 1 to 9; It includes test auxiliary mechanism, data acquisition module, point cloud processing module and evaluation module; The test auxiliary mechanism includes an RT device, a forward camera module and an autonomous driving robot; the data acquisition module establishes communication connections with the test auxiliary mechanism, the radar under test and the test vehicle respectively to collect standard driving data, image data, point cloud data and vehicle driving data; the point cloud processing module is used to process the point cloud data and add annotations to the point cloud data, and perform point cloud data comparison based on the image data; a preset perception model is set in the evaluation module; the evaluation module is used to evaluate the point cloud quality and target perception performance of the laser radar; and when evaluating the target perception performance, the preset perception model is used to assist in the evaluation; The preset perception model is pre-trained using multi-source radar data.

Images