A data fusion evaluation method and system based on high-fidelity sensor simulation

Abstract

The present invention discloses an evaluation method and system for data fusion based on high-fidelity sensor simulation, the steps include: building a real scene and deploying a high-fidelity virtual sensor model; acquiring and parsing the data of each sensor; synchronizing the parsed sensor data in time and space in turn; preprocessing the synchronized sensor data, and then fusing the preprocessed sensor data, screening targets according to the vehicle driving area, and outputting target-level data; performing anomaly inspection on the target-level data; extracting abnormal target-level data and simultaneously saving all the original data at the corresponding moment; reading the time of each frame of abnormal target-level data and backtracking the original data at that moment, identifying the abnormal type and outputting a table. The method and system of the present invention are more accurate, can evaluate sensors quickly and in large quantities, and provide clear error time points and preliminary judgment criteria for subsequent manual precision analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned driving perception algorithm development and performance verification, and in particular to an evaluation method and system for data fusion based on high-fidelity sensor simulation. Background Art

[0002] Currently, the data analysis methods available for verification of autonomous driving fusion algorithms are basically systems with true values. For example, the true value test system based on LiDAR provides multi-target, real-time true value data comparison and analysis. This system is currently the only true value test system that can be used for real vehicle road data. However, the equipment and use costs of this system are too high, and it is impossible to quickly evaluate sensors. Summary of the invention

[0003] The purpose of the present invention is to provide an evaluation method based on data fusion of high-fidelity sensor simulation, which has higher accuracy, can evaluate sensors quickly and in large quantities, and provide clear error time points and preliminary criteria for subsequent manual precision analysis, greatly accelerating the rough screening of massive sensor data and performance investigation.

[0004] To achieve the above object, the present invention provides an evaluation method for data fusion based on high-fidelity sensor simulation, the steps comprising:

[0005] Build real-world scenarios and deploy high-fidelity virtual sensor models;

[0006] Acquire and analyze the data of each sensor during the simulation test;

[0007] Time synchronization of the parsed sensor data;

[0008] Spatial synchronization of the parsed sensor data;

[0009] Preprocess the synchronized sensor data, fuse the preprocessed sensor data, screen targets according to the vehicle driving area, and output multiple targets that have a close impact on driving safety and driving performance as target-level data;

[0010] Perform anomaly checks on target-level data;

[0011] Extract abnormal target-level data and simultaneously save all the original data at the corresponding moment of the abnormal target-level data; read the time of each frame of abnormal target-level data and trace back the original data at that moment; identify the abnormal type and output a table, the table content includes the abnormal type, attribute, time, abnormal error rate and missed detection rate of the abnormal target-level data.

[0012] Furthermore, after outputting a plurality of targets that have a close impact on driving safety and driving performance as target-level data, the following steps are further performed:

[0013] Project the target-level data for display.

[0014] Furthermore, the fusing of the sensor data after data preprocessing includes performing target data fusion and lane line data fusion on the sensor data after data preprocessing.

[0015] Furthermore, the abnormality check includes: transverse and longitudinal distance continuity and mutation slope check, transverse and longitudinal speed continuity and mutation slope check, transverse and longitudinal acceleration continuity and mutation slope check, target ID continuity check, target missed detection check, and time difference interval check.

[0016] Furthermore, the horizontal and vertical distance / speed / acceleration continuity and mutation slope inspection specifically performs the following steps:

[0017] Determine whether the rate of change of the transverse and longitudinal distance / speed / acceleration is less than or equal to the first transverse and longitudinal distance threshold / first transverse and longitudinal speed threshold / first transverse and longitudinal acceleration threshold. If so, the continuity is satisfied and the output is normal; otherwise, the output is abnormal mutation.

[0018] Furthermore, the time difference interval check specifically performs the following steps:

[0019] Determine whether the time interval between two adjacent frames of data output by the sensor exceeds the preset time interval. If so, the data is blocked and the transmission is abnormal; otherwise, the data is normal.

[0020] Further, the target ID continuity check specifically performs the following steps:

[0021] Using the binary tree algorithm, traverse to find the sensor target ID with consistent target attributes, put the multiple outputs found into the data container, and use the time difference to calculate the time interval. If the time interval is greater than the sensor sending cycle, it is determined to be an independent data source. The number of independent data sources is counted and spliced ​​by time to determine whether the sensor target ID changes in the spliced ​​data. If so, the sensor target ID jumps; otherwise, the sensor target ID satisfies continuity.

[0022] Furthermore, the target missed detection check specifically performs the following steps:

[0023] The target-level data is used to use the target data matching algorithm to reversely trace the data source, find out the sensor data corresponding to the target-level data, count the valid frames output by each sensor, divide the valid frames output by the number of frames of the target fusion result, and get the continuous detection rate of each sensor target. Subtract the continuous detection rate from 1 to get the target missed detection rate of each sensor.

[0024] The present invention also provides an evaluation system for data fusion based on high-fidelity sensor simulation, which is configured to execute the steps of the evaluation method for data fusion based on high-fidelity sensor simulation, including:

[0025] Modeling module, used to build real-world scenarios and deploy high-fidelity virtual sensor models;

[0026] The data receiving and parsing module is used to obtain and parse the data of each sensor during the simulation test;

[0027] A time synchronization module is used to synchronize the sensor data;

[0028] A spatial synchronization module, used for spatial synchronization of sensor data;

[0029] The perception processing module is used to pre-process, fuse and filter the data from each sensor;

[0030] The evaluation module is used to check the data after perception processing for abnormalities;

[0031] Data statistics module, used to identify abnormal types, calculate abnormal error rates and output tables;

[0032] The modeling module, data receiving and analyzing module, time synchronization module, space synchronization module, perception processing module, evaluation module and data statistics module are connected in sequence.

[0033] Furthermore, it also includes a visualization module for projecting and displaying target-level data.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The data fusion evaluation method and system based on high-fidelity sensor simulation of the present invention realizes rapid data reading through a universal data acquisition interface, and after time-space synchronization, performs rapid and large-scale evaluation of sensors according to prior knowledge such as the data characteristics and laws of each sensor itself and the multi-sensor fusion results. At the same time, data visualization can be provided to analyze the data quality and rationality that are of close concern in the development process of autonomous driving, and provide clear error time points and preliminary judgment criteria for subsequent manual precision analysis, greatly accelerating the rough screening and performance investigation of massive sensor data; the accuracy of the tabular data output by the method and system is theoretically higher than that of a single sensor, so its data can be used as a benchmark for analysis; it provides a high-fidelity virtual sensor interface and fusion algorithm simulation capability that cannot be provided by existing simulation platforms, and can quickly respond to modifications and online debugging of the fusion algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the evaluation method of data fusion based on high-fidelity sensor simulation of the present invention;

[0037] Figure 2 It is a schematic diagram of the evaluation system of the present invention based on data fusion of high-fidelity sensor simulation.

[0038] In the figure:

[0039] a1-modeling module, 1-data receiving and parsing module; 2-time synchronization module; 3-space synchronization module; 4-perception processing module; 5-evaluation module; 6-data statistics module; 7-visualization module. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0041] See also Figure 1 As shown, this embodiment discloses a data fusion evaluation method based on high-fidelity sensor simulation, the steps comprising:

[0042] Build a real scene and deploy a high-fidelity virtual sensor model; the high-fidelity virtual sensor model is consistent with the real sensor. In this embodiment, the CarMaker software is used to build a real scene and deploy a high-fidelity virtual sensor model.

[0043] During the simulation test, the data of each sensor is obtained and parsed; the data of each sensor is parsed into a readable and analyzable data format. Since the types, models, manufacturers and communication protocols of sensors are different, a general data reading module is built, which can quickly read various types of data with only a small amount of adjustment. Using the concept of C++ polymorphism, the abstract sensor receiving AbstractSensorRecevier parent class object is designed to manage the data reception of all types of sensors, which contains the pure virtual function recevie. According to the communication protocol of Ethernet, CAN or CANFd, three receiving subclass objects of Ethernet data Socket_Sensor_Receiver, Can data Can_Sensor_Recevier, and Canfd data Canfd_Sensor_Recevier are designed. Each of them contains a virtual function recevie, which realizes the required data transmission method according to the requirements of each communication protocol; on the basis of realizing the normal reception of the original data of the sensor, the abstract parsing abstact_parser parent class is designed to uniformly manage the parsing of all original data. According to the CAN dbc or signal list provided by sensors of various manufacturers and models, specific data analysis is implemented to convert the raw sensor data into a readable and usable data format. The data is uniformly saved in the RawSensorMsg class of sensor raw data predefined within the program. The transformation of external sensors or their communication protocols only requires changing the parsing code, which achieves isolation and independence of data transmission and has good granularity. Sensor data includes front radar data, surround view camera data, and front camera data.

[0044] Time synchronization is performed on the parsed sensor data; time synchronization is to unify all sensor output data under the same time coordinate axis to achieve high data accuracy. Time synchronization uses the system absolute time as the time axis, selects the output time of a certain sensor as the reference time according to the actual situation, and uses the vector container and map container in C++ to establish a sequential query mechanism, and compares the difference between each frame of data of the remaining sensors and the reference time in turn, and finds the frame with the smallest time interval as the time after synchronization.

[0045] The analyzed sensor data are spatially synchronized; the front bumper center point of the vehicle is taken as the origin, and the vehicle coordinate system with negative left and positive right, positive front and negative rear is taken as the reference coordinate system for all sensors. All sensor data are unified to the front bumper center of the vehicle, and all subsequent analysis and graphical display data are based on this point to achieve the unification of data measurement benchmarks.

[0046] The synchronized sensor data is preprocessed, and then the sensor data is fused. The target is screened according to the vehicle driving area, and multiple targets that have a close impact on driving safety and driving performance are output as target-level data; the fusion adopts the fusion code of the mass production stage of level 3 autonomous driving to achieve multi-sensor data fusion, and then output target-level data with relatively high credibility. The output accuracy of this module is theoretically higher than that of a single sensor, so its data can be used as a benchmark for analysis.

[0047] Perform anomaly checks on target-level data;

[0048] Extract abnormal target-level data and save all the original data at the corresponding moment of the abnormal target-level data at the same time; read the time of each frame of abnormal target-level data and trace back the original data at that moment; identify the abnormal type and output a table, the table content includes the abnormal type, attribute, time, abnormal error rate and missed detection rate of the abnormal target-level data. Specifically, the data statistics module extracts all the abnormal target-level data recorded in the evaluation module and stores them in the error data structure, and saves all the original data at that moment. Read the time of each frame of error data, trace back in the original data, determine the error type, and finally output a table containing all the attributes of the error point, error type, error time, abnormal error rate, and missed detection rate.

[0049] In this embodiment, after outputting multiple targets that have a close impact on driving safety and driving performance as target-level data, the following steps are further performed: the target-level data is projected and displayed. The projection display facilitates analysts to find and reproduce required scene data from a large number of scenes in subsequent work.

[0050] In this embodiment, the fusing of the sensor data after data preprocessing includes performing target data fusion and lane line data fusion on the sensor data after data preprocessing.

[0051] In this embodiment, the abnormality check includes: transverse and longitudinal distance continuity, sudden slope check, transverse and longitudinal speed continuity, sudden slope check, transverse and longitudinal acceleration continuity, sudden slope check, target ID continuity check, target missed detection check, and time difference interval check. Among them, the target ID continuity check is the identification number assigned to a target by the sensor after it identifies and stably tracks it.

[0052] In this embodiment, the continuity and mutation slope check of the transverse and longitudinal distance / speed / acceleration is specifically performed by the following steps: determining whether the change rate of the transverse and longitudinal distance / speed / acceleration is less than or equal to the first transverse and longitudinal distance threshold / first transverse and longitudinal speed threshold / first transverse and longitudinal acceleration threshold. If so, the continuity is satisfied and the output is normal; otherwise, an abnormal mutation is output. Since most of the target vehicles are in a continuously changing motion trajectory when driving on the road, their transverse and longitudinal positions, speeds and even accelerations rarely jump significantly, and most of the data jumps of the same target are caused by sensor detection errors. Therefore, according to the statistical data obtained by collecting sensor data in the early stage, the slope of the change rate of the transverse and longitudinal distance is generally within 10m / s, and the change rate of the transverse and longitudinal speed is divided into two categories according to the sensor output characteristics: image type and radar Doppler effect type. The image type output data has poor speed estimation accuracy based on pixel points, so the threshold is relatively large and is generally above 20. The Doppler effect is very sensitive to the rate of change of speed, so the speed detection accuracy is relatively high, and its threshold is generally within 15. If the slope is greater than the threshold, it is judged as an abnormal output mutation. The continuity and mutation slope of the target's horizontal and vertical distances, speed, and acceleration are determined accordingly.

[0053] In this embodiment, the time difference interval check specifically performs the following steps: determine whether the time interval between two adjacent frames of data output by the sensor exceeds the preset time interval. If it exceeds, the data is blocked and the transmission is abnormal; otherwise, the data is normal. According to the output cycle of the sensor, its output frequency is generally 30-50ms, so the interval of the output must be within this time range. The frame data exceeding the time interval can be determined as data blocking and transmission abnormality, thereby completing the time interval judgment. The value range of the preset time interval is 30-50ms.

[0054] In this embodiment, the target ID continuity check specifically performs the following steps: using a binary tree algorithm, traverse to find the sensor target ID with consistent target attributes, put the multiple outputs found into a data container, use the time difference to calculate the time interval, if the time interval is greater than the sensor sending cycle, it is determined to be an independent data source, count the number of independent data sources obtained and splice them by time, and determine whether the sensor target ID changes in the spliced ​​data. If so, the sensor target ID jumps; otherwise, the sensor target ID meets the continuity. The more jumps, the worse the sensor output quality. Counting the number of independent data sources obtained and splicing them by time is the detection result of the sensor for the same sensor target ID.

[0055] In the normal working process of autonomous driving, the data output by each sensor is used to obtain target-level data through association and fusion algorithms. In this embodiment, the target missed detection check specifically performs the following steps: the target-level data is used to use the target data matching algorithm to reversely trace the data source, find out the sensor data corresponding to the target-level data, count the valid frames output by each sensor, divide the valid frames output by the target fusion result by the number of frames, and obtain the target continuous detection rate of each sensor, and subtract the continuous detection rate from 1 to obtain the target missed detection rate of each sensor.

[0056] See also Figure 2 As shown, a data fusion evaluation system based on high-fidelity sensor simulation is configured to execute the steps of the above-mentioned data fusion evaluation method based on high-fidelity sensor simulation, including:

[0057] Modeling module a1, used to build real scenarios and deploy high-fidelity virtual sensor models;

[0058] The data receiving and parsing module 1 is used to obtain and parse the data of each sensor during the simulation test. The data receiving and parsing module uses the concept of C++ polymorphism to design the abstract sensor receiving AbstractSensorRecevier parent class object to manage the data reception of all types of sensors, which includes a pure virtual function recevie. According to the communication protocol of Ethernet, CAN or CANFd, three receiving subclass objects of Ethernet data Socket_Sensor_Receiver, Can data Can_Sensor_Recevier, and Canfd data Canfd_Sensor_Recevier are designed. Each of them contains a virtual function recevie, which implements the required data transmission method according to the requirements of each communication protocol. On the basis of achieving normal reception of sensor raw data, the abstract parsing abstact_parser parent class is designed to uniformly manage the parsing of all raw data. According to the CAN dbc or signal list provided by sensors of various manufacturers and models, specific data analysis is implemented to convert the sensor raw data into a readable and usable data format. The data is uniformly saved in the RawSensorMsg class of sensor raw data predefined inside the program. The transformation of external sensors or their communication protocols only requires changing the parsing code, thus achieving isolation and independence of data transmission with good granularity.

[0059] Time synchronization module 2, used for time synchronization of sensor data;

[0060] A spatial synchronization module 3, used for spatial synchronization of sensor data;

[0061] Perception processing module 4 is used to preprocess, fuse and filter the data of each sensor; processing module 4 is the fusion code for the mass production stage of level 3 autonomous driving, which realizes the fusion of multi-sensor data. Based on data reception and analysis, this part of the code uses the perception module to preprocess the data according to the characteristics of each sensor data, eliminate unreasonable data, and then perform objfusion target data fusion and lanefusion lane line data fusion to achieve the "strengths and weaknesses" of sensor output characteristics. Finally, target_selector target screening is performed according to the vehicle driving area, and 0-18 targets that have a close impact on driving safety and driving performance are finally output as the final output result. This result is target-level data with relatively high credibility, and the accuracy of the output is theoretically higher than that of a single sensor.

[0062] Evaluation module 5, used to check the data after perception processing for abnormalities;

[0063] Data statistics module 6, used to identify the abnormal type, calculate the abnormal error rate and output the table;

[0064] The modeling module a1, data receiving and parsing module 1, time synchronization module 2, space synchronization module 3, perception processing module 4, evaluation module 5 and data statistics module 6 are connected in sequence. Specifically, the data statistics module 6 extracts all the abnormal target-level data recorded in the evaluation module and stores them in the error data structure, while saving all the original data at that moment. The error data is passed to the performance evaluation module, which reads the time of each frame of error data, and traces back in the original data to determine the error type. The final output table contains all the attributes of the error point, error type, error time, abnormal error rate, and missed detection rate. The system uses the common sensor output characteristics of level 3 autonomous driving, accesses the non-true value evaluation system, and combines the characteristics and restrictions of each sensor data to quickly retrieve and judge the accuracy and rationality of the sensor output data, providing a preliminary quantitative standard for subsequent manual analysis of data, greatly accelerating the rough screening of massive sensor data and performance investigation.

[0065] In this embodiment, a visualization module 7 for projecting and displaying target-level data is also included. The visualization module 7 is divided into two types: online data display developed using Qt, and offline analysis and playback developed using matlab. The visualization program based on C++ developed by Qt is mainly used to display sensor data in real time and dynamically and intuitively observe the sensor output effect. The module stores the data saved in the RawSensorMsg class into the data structure defined in Visulization_Message for video projection. The starVisulizetionThread starts the visualization thread to package the data and send it to the Qt end for reception. The Qt end uses the window parent class mianWidgt to receive it, and calls its parent class virtual function show for unified management of the display. ShowFrontImage is used to realize polymorphic display of the front target, and showRearImage is used to realize polymorphic display of the rear target; matlab reads the parsed sensor data, and displays the recorded image frame by frame according to the actual sensor FOV limit.

[0066] The data fusion evaluation method and system based on high-fidelity sensor simulation of the present invention realizes rapid data reading through a universal data acquisition interface, and after time-space synchronization, performs rapid and large-scale evaluation of sensors according to prior knowledge such as the data characteristics and laws of each sensor itself and the multi-sensor fusion results. At the same time, data visualization can be provided to analyze the data quality and rationality that are of close concern in the development process of autonomous driving, and provide clear error time points and preliminary judgment criteria for subsequent manual precision analysis, greatly accelerating the rough screening and performance investigation of massive sensor data; the accuracy of the tabular data output by the method and system is theoretically higher than that of a single sensor, so its data can be used as a benchmark for analysis; it provides a high-fidelity virtual sensor interface and fusion algorithm simulation capability that cannot be provided by existing simulation platforms, and can quickly respond to modifications and online debugging of the fusion algorithm.

[0067] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A data fusion evaluation method based on high-fidelity sensor simulation, characterized in that the steps include: Build real-world scenarios and deploy high-fidelity virtual sensor models; Acquire and analyze the data of each sensor during the simulation test; Time synchronization of the parsed sensor data; Spatial synchronization of the parsed sensor data; Preprocess the synchronized sensor data, fuse the preprocessed sensor data, screen targets according to the vehicle driving area, and output multiple targets that have a close impact on driving safety and driving performance as target-level data; Perform an abnormality check on the target-level data; the abnormality check includes: horizontal and vertical distance continuity, sudden change slope check, horizontal and vertical speed continuity, sudden change slope check, horizontal and vertical acceleration continuity, sudden change slope check, target ID continuity check, target missed detection check, and time difference interval check; Extract abnormal target-level data and simultaneously save all the original data at the corresponding moment of the abnormal target-level data; read the time of each frame of abnormal target-level data and trace back the original data at that moment; identify the abnormal type and output a table, the table content includes the abnormal type, attribute, time, abnormal error rate and missed detection rate of the abnormal target-level data.

2. The data fusion evaluation method based on high-fidelity sensor simulation according to claim 1 is characterized in that: After outputting the multiple targets that have a close impact on driving safety and driving performance as target-level data, the following steps are further performed: Project and display target-level data.

3. The evaluation method for data fusion based on high-fidelity sensor simulation according to claim 1 or 2, characterized in that: The fusing of the sensor data after data preprocessing includes performing target data fusion and lane line data fusion on the sensor data after data preprocessing.

4. The data fusion evaluation method based on high-fidelity sensor simulation according to claim 3 is characterized in that: The horizontal and vertical distance / speed / acceleration continuity and mutation slope inspection specifically performs the following steps: Determine whether the change rate of the transverse and longitudinal distance / speed / acceleration is less than or equal to the first transverse and longitudinal distance threshold / first transverse and longitudinal speed threshold / first transverse and longitudinal acceleration threshold. If so, the continuity is satisfied and the output is normal; Otherwise, an abnormal mutation is output.

5. The data fusion evaluation method based on high-fidelity sensor simulation according to claim 4 is characterized in that: The time difference interval check specifically performs the following steps: Determine whether the time interval between two adjacent frames of data output by the sensor exceeds the preset time interval. If so, the data is blocked and the transmission is abnormal; otherwise, the data is normal.

6. The data fusion evaluation method based on high-fidelity sensor simulation according to claim 4 or 5, characterized in that: The target ID continuity check specifically performs the following steps: Using the binary tree algorithm, traverse to find the sensor target ID with consistent target attributes, put the multiple outputs found into the data container, and use the time difference to calculate the time interval. If the time interval is greater than the sensor sending cycle, it is determined to be an independent data source. The number of independent data sources is counted and spliced ​​by time to determine whether the sensor target ID changes in the spliced ​​data. If so, the sensor target ID jumps; otherwise, the sensor target ID satisfies continuity.

7. The data fusion evaluation method based on high-fidelity sensor simulation according to claim 6 is characterized in that: The target missed detection check specifically performs the following steps: The target-level data is used to use the target data matching algorithm to reversely trace the data source, find out the sensor data corresponding to the target-level data, count the valid frames output by each sensor, divide the valid frames output by the number of frames of the target fusion result, and get the continuous detection rate of each sensor target. Subtract the continuous detection rate from 1 to get the target missed detection rate of each sensor.

8. A data fusion evaluation system based on high-fidelity sensor simulation, characterized in that: The steps configured to execute the evaluation method of data fusion based on high-fidelity sensor simulation as described in any one of claims 1 to 7 include: Modeling module (a1), used to build real scenarios and deploy high-fidelity virtual sensor models; A data receiving and parsing module (1) is used to obtain and parse the data of each sensor during the simulation test process; A time synchronization module (2), used for performing time synchronization on sensor data; A spatial synchronization module (3), used for spatial synchronization of sensor data; A sensing processing module (4) is used to pre-process, fuse and filter the data of each sensor; Evaluation module (5), used to check the data after perception processing for abnormalities; A data statistics module (6), used to identify the abnormal type, calculate the abnormal error rate and output a table; The modeling module (a1), the data receiving and analyzing module (1), the time synchronization module (2), the space synchronization module (3), the perception processing module (4), the evaluation module (5) and the data statistics module (6) are connected in sequence.

9. The data fusion evaluation system based on high-fidelity sensor simulation according to claim 8, characterized in that: The invention also includes a visualization module for projecting and displaying the target-level data.

Images