A method and system for evaluating fidelity detection based on radar sensor model

Abstract

The present invention discloses an evaluation method and system for fidelity detection based on a radar sensor model, wherein the method comprises the following steps: obtaining real radar point cloud data of a test vehicle under real driving, and measuring and recording the real trajectory of the test vehicle at the same time; obtaining simulated radar point cloud data of the test vehicle under simulated driving; using a point cloud tracking algorithm to track the simulated radar point cloud data and generate a tracking trajectory; performing consistency calculation on the tracking trajectory and the real trajectory, quantifying the calculation result, and using the quantified result as the fidelity of the radar sensor model. The present invention uses a perception algorithm such as point cloud target tracking to evaluate the difference between the simulated data generated by the radar sensor model and the real data, thereby indirectly evaluating the fidelity of the radar sensor model.

Description

Technical Field

[0001] The present invention belongs to the field of radar sensor model fidelity evaluation methods, and in particular, relates to an evaluation method and system based on radar sensor model fidelity detection. Background Art

[0002] During the testing phase of autonomous vehicles, radars need to be tested for a variety of performance and indicators. If all field tests are used, it will consume a lot of manpower, material resources, and financial resources and will be easily affected by weather conditions. Therefore, radar models based on simulation are gradually being used. However, it is necessary to quantify the difference between simulation and reality and verify whether the fidelity of the adopted radar sensor model meets the intended use. At present, there is no reliable method to measure the fidelity of radar sensor models, nor is there an appropriate measurement. Traditional measurement methods mainly include evaluation methods at the raw data level before generating point clouds, but they can only detect simple scenes and basic functions; the detection level evaluation methods for generating point clouds mainly have the problems of qualitative evaluation and manual formulation of evaluation indicators. This method proposes to further perceive point cloud data and use the performance effect of the perception algorithm as an evaluation method for the sensor model. This method can evaluate the fidelity of the radar model at the perception level by verifying whether the radar model meets the intended use. Summary of the invention

[0003] In order to determine the reliability of the radar sensor model, it is necessary to detect whether the gap between simulation and reality is suitable for the application purpose. In addition to evaluating the generated radar data, the quality of the data must also be checked for adaptability to the intended use to meet the different needs of different applications. There are many radar sensor models, and the working principle is to generate a virtual scene of the environment from the perspective of the radar sensor based on the data of the real test drive to obtain a simulated radar point cloud. However, the problem of verifying and quantitatively evaluating the overall fidelity of the radar sensor model has not yet been solved. Based on this, the present invention proposes an evaluation method and system based on fidelity detection of radar sensor models.

[0004] On the one hand, to achieve the above-mentioned object, the present invention provides an evaluation method for fidelity detection based on a radar sensor model, comprising the following steps:

[0005] Acquire real radar point cloud data of the test car under real driving, and simultaneously measure and record the real trajectory of the test car;

[0006] Acquiring simulated radar point cloud data of the test vehicle under simulated driving;

[0007] Using a point cloud tracking algorithm, tracking the simulated radar point cloud data to generate a tracking trajectory;

[0008] The tracking trajectory and the real trajectory are calculated for consistency, the calculation result is quantified, and the quantified result is used as the fidelity of the radar sensor model.

[0009] Optionally, the method for calculating the consistency of the tracking trajectory and the true trajectory includes: quantitative calculation of root mean square error, optimal sub-mode allocation distance and IoU indicator.

[0010] Optionally, the calculation formula of the root mean square error is:

[0011] Among them, x k is the true value of the target at time k, x k ～ is the target estimate, M is the number of simulations, and i represents the measurement number.

[0012] Optionally, the calculation formula of the optimal sub-mode allocation distance is:

[0013] Among them, p is the distance sensitivity parameter, c is the level adjustment number, is an estimate.

[0014] Optionally, the IoU indicator is quantitatively calculated by dividing the intersection area between the estimated shape of the tracking trajectory and the predicted shape of the actual trajectory of the test vehicle by the union area of ​​the two shapes.

[0015] Optionally, the method for quantifying the calculation results is: mapping the calculation results in a 0-1 space by a z-score method, adjusting the coefficients of each consistency calculation method, and adding the mapped calculation results to obtain the fidelity of the radar sensor model.

[0016] On the other hand, to achieve the above-mentioned purpose, the present invention provides an evaluation system for fidelity detection based on a radar sensor model, comprising:

[0017] Real data generation module, simulated data generation module, tracking module and perception module;

[0018] The real data generation module is used to obtain the real radar point cloud data of the test car under real driving, and at the same time measure and record the real trajectory of the test car;

[0019] The simulation data generation module is used to obtain the simulated radar point cloud data of the test vehicle under simulated driving;

[0020] The tracking module is used to track the simulated radar point cloud data based on a point cloud tracking algorithm to generate a tracking trajectory;

[0021] The perception module is used to calculate the consistency between the tracking trajectory and the real trajectory of the test vehicle, quantify the calculation result, and use the quantified result as the fidelity of the radar sensor model.

[0022] Optionally, the perception module includes: a first calculation unit, a second calculation unit, a third calculation unit and a quantization unit;

[0023] The first calculation unit is used to calculate the root mean square error between the tracking trajectory and the true trajectory;

[0024] The second calculation unit is used to calculate the optimal sub-mode allocation distance between the tracking trajectory and the real trajectory;

[0025] The third computing unit is used to evaluate the performance of the bounding box prediction of the tracking trajectory and the real trajectory based on the intersection area between the estimated shape of the tracking trajectory and the predicted shape of the real trajectory of the test vehicle divided by the union area of ​​the two shapes;

[0026] The quantization unit is used to map the calculation results into a 0-1 space by using a z-score method, and to add the mapped calculation results to obtain the fidelity of the radar sensor model by adjusting the coefficients of each consistency calculation method.

[0027] The technical effects of the present invention are:

[0028] The present invention discloses an evaluation method and system for fidelity detection based on a radar sensor model. After the perception layer uses the data, the fidelity of the radar model is evaluated through multiple metrics. When the perception layer is selected as the tracking algorithm, the quality of the simulated data is quantitatively evaluated by comparing the consistency of the real trajectory and the tracking trajectory. The structure of the present invention is very flexible and can be adjusted according to actual needs, such as different perception methods, and can be applied to many types of radar sensor models. It can efficiently evaluate the effect of the radar model and has high economic and social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0030] Figure 1 This is a schematic flow chart of an evaluation method for fidelity detection based on a radar sensor model in Embodiment 1 of the present invention;

[0031] Figure 2 This is a schematic diagram of IoU interpretation of the evaluation method for fidelity detection based on a radar sensor model in Embodiment 1 of the present invention;

[0032] Figure 3 Schematic diagram of the structure of an evaluation system for fidelity detection based on a radar sensor model in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Embodiment 1

[0036] The flow chart of the evaluation method of fidelity detection based on radar sensor model is as follows: Figure 1 The overall process of the present invention is as follows:

[0037] Acquire real radar point cloud data of the test car under real driving, and simultaneously measure and record the real trajectory of the test car; acquire simulated radar point cloud data of the test car under simulated driving; use a point cloud tracking algorithm to track the simulated radar point cloud data to generate a tracking trajectory; perform consistency calculation on the tracking trajectory and the real trajectory, quantify the calculation result, and use the quantified result as the fidelity of the radar sensor model.

[0038] Specifically, the present invention evaluates the fidelity of the radar sensor model from the perception level. The perception level evaluation is to evaluate the output of the perception module after inputting the real and simulated radar data into the perception module developed for the real radar data, and then indirectly evaluate the radar sensor model. The algorithm in the perception module should be generally applicable and not developed for a specific algorithm. It can be considered as a black box, so the present invention has strong flexibility. At the same time, the evaluation metric of each perception module is determined by the corresponding perception algorithm. At present, the research on the detection of the effect of the perception algorithm is relatively mature. The present invention mainly tracks the simulated data through the common object tracking (Object Tracking) perception algorithm. The perception algorithm mainly introduced in the present invention is object tracking (Object Tracking), and the tracking trajectory is obtained by tracking the simulated data. The authenticity of the radar sensor model can be evaluated by comparing the consistency between the tracking trajectory and the real trajectory. The performance indicators of the accuracy of the tracking algorithm are:

[0039] 1. Root mean square error: x k is the true value of the target at time k, x k~ is the target estimate, M is the number of simulations, and i represents the measurement number.

[0040]

[0041] 2. Optimal subpattern assignment (OSPA), which is a commonly used method to measure the consistency of the overall performance of target tracking systems. p is the distance sensitivity parameter, c is the level adjustment number, is an estimate.

[0042]

[0043]

[0044]

[0045] 3. The third metric is used to evaluate the performance of bounding box prediction, using the intersection over union (IoU) as the third metric. Figure 2 As shown, the IoU is defined as the area of ​​the intersection between the estimated shape based on simulated radar data and the predicted shape based on real radar data, divided by the area of ​​the union of the two shapes.

[0046] The specific process is as follows: Step 1: Select a test scenario, such as a lane change scenario of the vehicle in front. Step 2: Perform real driving and record measurements. Step 3: Simulate driving and generate simulated point cloud data. Step 4: Use the point cloud tracking algorithm to track the simulated point cloud, generate a tracking trajectory, and quantitatively calculate it through three metrics, namely, mean square error RMSE, optimal sub-mode allocation distance OSPA, and IoU indicators. Step 5: Standardize the measurement results. Each measurement result is mapped in the 0-1 space using the z-score method. Adjust the importance coefficient of each measurement indicator as needed. Add them together to get the fidelity result.

[0047] The present invention indirectly compares the fidelity of radar models by comparing the accuracy indicators of the above three common tracking algorithms. That is, tracking is performed through simulated data, the tracking trajectory after tracking is compared with the real trajectory, the above three indicators are calculated, and the calculation results are quantified as fidelity through z-score standardization.

[0048] Embodiment 2

[0049] The schematic diagram of the evaluation system structure based on the fidelity detection of the radar sensor model is as follows: Figure 3 As shown,

[0050] An evaluation system for fidelity detection based on a radar sensor model, comprising: a real data generation module, a simulation data generation module, a tracking module and an evaluation module;

[0051] The real data generation module is used to obtain the real radar point cloud data of the test car under real driving, and at the same time measure and record the real trajectory of the test car;

[0052] The simulation data generation module is used to obtain the simulated radar point cloud data of the test vehicle under simulated driving;

[0053] The tracking module is used to track the simulated radar point cloud data based on the point cloud tracking algorithm and generate the tracking trajectory;

[0054] The evaluation module is used to calculate the consistency between the tracking trajectory and the actual trajectory of the test car, quantify the calculation results, and use the quantified results as the fidelity of the radar sensor model.

[0055] Specifically, the evaluation module includes: a first calculation unit, a second calculation unit, a third calculation unit and a quantification unit;

[0056] The first calculation unit is used to calculate the root mean square error between the tracking trajectory and the true trajectory;

[0057] The second calculation unit is used to calculate the optimal sub-mode allocation distance between the tracking trajectory and the real trajectory;

[0058] The third computing unit is used to evaluate the performance of bounding box prediction of the tracking trajectory and the true trajectory based on the intersection area between the estimated shape of the tracking trajectory and the predicted shape of the true trajectory of the test vehicle divided by the union area of ​​the two shapes;

[0059] The quantization unit is used to map the calculation results into the 0-1 space through the z-score method, and to obtain the fidelity of the radar sensor model by adding the mapped calculation results by adjusting the coefficients of each consistency calculation method.

[0060] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for evaluating fidelity detection based on a radar sensor model, characterized in that: The following steps are involved: Acquire real radar point cloud data of the test car under real driving, and simultaneously measure and record the real trajectory of the test car; Acquiring simulated radar point cloud data of the test vehicle under simulated driving; Using a point cloud tracking algorithm, tracking the simulated radar point cloud data to generate a tracking trajectory; The tracking trajectory and the real trajectory are calculated for consistency, the calculation result is quantified, and the quantified result is used as the fidelity of the radar sensor model.

2. The method for evaluating fidelity detection based on a radar sensor model according to claim 1, characterized in that: The method for calculating the consistency between the tracking trajectory and the true trajectory includes: root mean square error, optimal sub-mode allocation distance and IoU indicator quantitative calculation.

3. The method for evaluating fidelity detection based on a radar sensor model according to claim 2, characterized in that: The calculation formula of the root mean square error is: Among them, x k is the true value of the target at time k, x k ～ is the target estimate, M is the number of simulations, and i represents the measurement number.

4. The method for evaluating fidelity detection based on a radar sensor model according to claim 2, characterized in that: The calculation formula of the optimal sub-mode allocation distance is: Among them, p is the distance sensitivity parameter, c is the level adjustment number, is an estimate.

5. The method for evaluating fidelity detection based on a radar sensor model according to claim 2, characterized in that: The IoU indicator is quantitatively calculated by dividing the intersection area between the estimated shape of the tracking trajectory and the predicted shape of the actual trajectory of the test vehicle by the union area of ​​the two shapes.

6. The method for evaluating fidelity detection based on a radar sensor model according to claim 2, characterized in that: The method for quantifying the calculation results is as follows: mapping the calculation results in the 0-1 space by the z-score method, adjusting the coefficients of each consistency calculation method, and adding the mapped calculation results to obtain the fidelity of the radar sensor model.

7. An evaluation system for fidelity detection based on a radar sensor model, characterized in that: include: Real data generation module, simulated data generation module, tracking module and perception module; The real data generation module is used to obtain the real radar point cloud data of the test car under real driving, and at the same time measure and record the real trajectory of the test car; The simulation data generation module is used to obtain the simulated radar point cloud data of the test vehicle under simulated driving; The tracking module is used to track the simulated radar point cloud data based on a point cloud tracking algorithm to generate a tracking trajectory; The perception module is used to calculate the consistency between the tracking trajectory and the real trajectory of the test vehicle, quantify the calculation result, and use the quantified result as the fidelity of the radar sensor model.

8. The evaluation system for fidelity detection based on radar sensor model according to claim 7, characterized in that: The perception module includes: a first calculation unit, a second calculation unit, a third calculation unit and a quantization unit; The first calculation unit is used to calculate the root mean square error between the tracking trajectory and the true trajectory; The second calculation unit is used to calculate the optimal sub-mode allocation distance between the tracking trajectory and the real trajectory; The third computing unit is used to evaluate the performance of the bounding box prediction of the tracking trajectory and the real trajectory based on the intersection area between the estimated shape of the tracking trajectory and the predicted shape of the real trajectory of the test vehicle divided by the union area of ​​the two shapes; The quantization unit is used to map the calculation results into a 0-1 space by using a z-score method, and to add the mapped calculation results to obtain the fidelity of the radar sensor model by adjusting the coefficients of each consistency calculation method.

Images