Automatic driving multi-sensor data acquisition and processing system

Abstract

An automatic driving multi-sensor data acquisition and processing system comprises a sensor data analysis module, a data preprocessing module, a visualization module and a data storage module which are realized in the form of ROS nodes, wherein the sensor data analysis module is used for analyzing original data acquired by each sensor on a vehicle and converting the original data into corresponding data modals; the data preprocessing module carries out time and space synchronization preprocessing on an analysis result; the visualization module visualizes the preprocessed laser radar and millimeter wave radar data into point cloud and visualizes the preprocessed camera data into images; and the data storage module stores the preprocessed data as an HDF5 file according to a format preset by each sensor. According to the invention, nodes and communication mechanisms of robot operating system control software (ROS notation) are fully utilized to allocate computing resources of a host, and real-time processing of sensor data is realized.

Description

Technical Field

[0001] The present invention relates to a technology in the field of autonomous driving, and specifically to an autonomous driving multi-sensor data acquisition and processing system. Background Art

[0002] The realization of autonomous driving is inseparable from reliable environmental perception capabilities, and the development of related algorithms relies on high-quality autonomous driving datasets. Existing open-source autonomous driving sensor datasets fail to truly cover all scenarios due to cost constraints. For example, tunnel and snowy scenes have high requirements for sensors and autonomous driving algorithms, and the sensor configuration is not comprehensive enough. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes an autonomous driving multi-sensor data acquisition and processing system, which fully utilizes the nodes and communication mechanism of the robot operating system control software (ROS noetic) to allocate the computing resources of the host and realize real-time processing of sensor data.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an automatic driving multi-sensor data acquisition and processing system, comprising: a sensor data analysis module, a data preprocessing module, a visualization module and a data storage module implemented in the form of a ROS node, wherein: the sensor data analysis module analyzes the raw data collected by each sensor on the vehicle and converts it into a corresponding data modality; the data preprocessing module performs time synchronization and online calibration on the analysis result; the visualization module visualizes the preprocessed laser radar and millimeter wave radar data as a point cloud, and visualizes the preprocessed camera data as an image; the data storage module stores the preprocessed data as an HDF5 file according to the format preset by each sensor.

[0006] The implementation in the form of ROS nodes means that each implemented node corresponds to a thread, the sensor data analysis module, data preprocessing module, visualization module and data storage module in the system are implemented by a single or multiple nodes, each node corresponds to a thread, and executes a specific function or task independently, and the communication and data transmission between modules are realized by the topic mechanism of ROS. Technical Effects

[0007] The present invention is based on the online real-time calibration of sensor external parameters based on entropy. According to the mode of sensor data, different statistical methods are used to estimate its distribution, and different entropies are used as cost functions to evaluate the randomness of its distribution. The sensor external parameters are optimized by the optimization algorithm to minimize the cost function, thereby realizing the online calibration of sensor external parameters. Compared with the prior art, the present invention can ensure the timely update of sensor external parameters and prevent the failure of sensor external parameters due to aging of fixed structural parts, severe turbulence and other problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the system of the present invention;

[0009] Figure 2 It is a schematic diagram of the sensor calibration unit;

[0010] Figure 3 This is a schematic diagram of the calibration results of millimeter wave radar and laser radar;

[0011] Figure 4 This is a schematic diagram of the calibration results of the laser radar and camera;

[0012] Figure 5 This is a schematic diagram of the HDF5 file structure. DETAILED DESCRIPTION

[0013] like Figure 1 As shown, this embodiment relates to an autonomous driving multi-sensor data acquisition and processing system, which includes: a sensor data analysis module, a data preprocessing module, a visualization module and a data storage module implemented in the form of a ROS node.

[0014] The sensor data parsing module includes: a 4D millimeter wave radar driving unit, a laser radar driving unit, a camera driving unit and a GNSS driving unit, wherein: the 4D millimeter wave radar driving unit parses the raw data of the 4D millimeter wave radar on the vehicle into two groups of channel ObjectLists and DetectionLists each containing multiple fields, and then publishes the data as a custom ROS message; the laser radar driving unit parses the raw data of the laser radar on the vehicle into a point cloud, and publishes it as a PCLPointCloud2 message containing four fields of x, y, z, and intensity; the camera driving unit obtains real-time YUV format images; the GNSS driving unit decodes the current latitude and longitude coordinates and the current timestamp of the vehicle according to the model received by the external antenna.

[0015] The data publishing means that the 4D millimeter-wave radar sends data to the same multicast address using the UDP protocol. The 4D millimeter-wave radar driving unit receives and parses the original data by subscribing to the multicast address, and distinguishes different 4D millimeter-wave radars by the original IP of the data, and creates a topic for each 4D millimeter-wave radar.

[0016] The data preprocessing module includes: a time synchronization unit and a sensor calibration unit, wherein: the time synchronization unit synchronizes the timestamps between various sensors; the sensor calibration unit calibrates and updates the internal and external parameters of the sensor in the offline and online stages respectively.

[0017] The synchronization refers to: periodically receiving and saving all sensor data, and assigning the GNSS time to all data received by the time synchronization module within a preset time interval.

[0018] like Figure 2 As shown, the sensor calibration unit includes: an offline calibration subunit and an online calibration subunit, wherein: the offline calibration subunit calibrates the internal and external parameters of each sensor when the sensor is installed, and the online calibration subunit, after receiving the time-synchronized data and the initial sensor external parameters, for sensors whose data modality is a spatial point cloud, namely, millimeter-wave radar and lidar sensors, respectively ① uses a mixed Gaussian model to estimate the distribution of the point cloud in three-dimensional space, namely and calculate the Rényi entropy of the distribution ② Use the kernel density of the edge and joint histogram of the laser radar point cloud intensity and camera image grayscale to estimate the distribution of the laser radar point cloud and camera image, and use the entropy of the edge distribution and joint distribution to calculate the mutual information of the point cloud and image M(X,Y)=H(X)+H(Y)-H(X,Y), and then complete the online calibration of the external parameters through the optimization algorithm, where: x represents the distribution of the measured point cloud in the real world, the subscript L represents the laser radar, R represents the millimeter wave radar, and N L and N R Respectively represent the number of points in the laser radar and millimeter wave radar point clouds, represents the set of points in all lidar point clouds, represents the set of points in all millimeter-wave radar point clouds, Indicates that the measured point cloud and After that, the posterior distribution of the true value of the point cloud is Indicates that the mean is z L (i), the covariance matrix is ​​P L Gaussian distribution of , H(x) represents the Rényi entropy of the random variable x, X and Y represent the histogram kernel density estimates of the lidar intensity and camera grayscale respectively, and M(X,Y) represents the mutual information of the random variables X and Y.

[0019] The optimization algorithm includes: optimizing the external parameters between the laser radar and the millimeter wave radar so that the Rényi entropy of all point clouds is minimized, then optimizing the external parameters between the laser radar and the camera so that the above mutual information is minimized, and finally completing the online calibration of the external parameters.

[0020] like Figure 3 As shown in the figure, it is the calibration result of millimeter wave radar and laser radar; Figure 4 As shown, the calibration results of the lidar and camera.

[0021] The internal and external parameters include: the internal parameters of the camera, the transformation matrix between the camera coordinate system and the laser radar coordinate system, the transformation matrix between the laser radar coordinate systems, and the transformation matrix between the millimeter wave radar and the laser radar, to obtain the initial position of the sensor on the test vehicle.

[0022] After specific practical experiments, on a vehicle equipped with 8 4D millimeter-wave radars, 3 solid-state laser radars and 8 industrial cameras, the reading and parsing of the 8 4D millimeter-wave radar data were completed by one node after initialization, and different sensors were distinguished by different ROS topic names when publishing messages. The amount of data from laser radars and industrial cameras is large, so one node only processes the data of one laser radar.

[0023] The bandwidth of the data processed by the system of the present invention reaches 5375.45Mbps, of which 8 4D millimeter wave radars account for 323.38Mbps, 3 laser radars account for 1156.11Mbps, and 8 cameras account for 3895.96Mbps, achieving real-time data processing capabilities. In order to verify the real-time performance of the software, each data is timestamped by the ROS system when the data processing is completed, and compared with the timestamp of the sensor, such as Figure 2 As shown, it is the comparison result of millimeter wave radar timestamp, where the initial time of the two timestamps is set to 0, and it can be seen that the two times are basically consistent.

[0024] like Figure 4 As shown in the figure, it is the HDF5 file structure saved by the storage module. Each sensor corresponds to a group. The point cloud or target of a single lidar and millimeter-wave radar is saved as a Dataset, and each frame image of the camera corresponds to a Dataset.

[0025] Compared with the prior art, the present invention realizes real-time analysis and processing of multi-sensor data through ROS nodes, solves the problems of insufficient types and quantity of sensors, poor robustness in harsh environments, and online automatic calibration of sensors in the prior art, realizes synchronous data processing of multiple sensors such as laser radar, millimeter-wave radar, and camera, and improves the system's perception of complex environments. Data is efficiently stored through the HDF5 file structure, and the real-time performance of data processing is improved through the multi-node mechanism of ROS, so that the system can process 5375.45Mbps of data.

[0026] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. An autonomous driving multi-sensor data acquisition and processing system, characterized in that: include: The sensor data analysis module, data preprocessing module, visualization module and data storage module are implemented in the form of ROS nodes, wherein: the sensor data analysis module analyzes the raw data collected by each sensor on the vehicle and converts it into the corresponding data modality; the data preprocessing module preprocesses the analysis results in time and space synchronization; the visualization module visualizes the preprocessed lidar and millimeter wave radar data as point clouds, and visualizes the preprocessed camera data as images; the data storage module stores the preprocessed data as HDF5 files according to the preset formats of each sensor.

2. The autonomous driving multi-sensor data acquisition and processing system according to claim 1, characterized in that: The sensor data parsing module includes: a 4D millimeter wave radar driving unit, a laser radar driving unit, a camera driving unit and a GNSS driving unit, wherein: the 4D millimeter wave radar driving unit parses the raw data of the 4D millimeter wave radar on the vehicle into two groups of channel ObjectLists and DetectionLists each containing multiple fields, and then publishes the data as a custom ROS message; the laser radar driving unit parses the raw data of the laser radar on the vehicle into a point cloud, and publishes it as a PCLPointCloud2 message containing four fields of x, y, z, and intensity; the camera driving unit obtains real-time YUV format images; the GNSS driving unit decodes the current latitude and longitude coordinates and the current timestamp of the vehicle according to the model received by the external antenna.

3. The autonomous driving multi-sensor data acquisition and processing system according to claim 2, characterized in that: The data publishing means that the 4D millimeter-wave radar sends data to the same multicast address using the UDP protocol. The 4D millimeter-wave radar driving unit receives and parses the original data by subscribing to the multicast address, and distinguishes different 4D millimeter-wave radars by the original IP of the data, and creates a topic for each 4D millimeter-wave radar.

4. The autonomous driving multi-sensor data acquisition and processing system according to claim 1, characterized in that: The data preprocessing module includes: a time synchronization unit and a sensor calibration unit, wherein: the time synchronization unit synchronizes the timestamps between various sensors; the sensor calibration unit calibrates and updates the internal and external parameters of the sensor in the offline and online stages respectively.

5. The autonomous driving multi-sensor data acquisition and processing system according to claim 4 is characterized in that: The synchronization refers to: periodically receiving and saving all sensor data, and assigning the GNSS time to all data received by the time synchronization module within a preset time interval.

6. The autonomous driving multi-sensor data acquisition and processing system according to claim 1, characterized in that: The sensor calibration unit comprises: an offline calibration subunit and an online calibration subunit, wherein: the offline calibration subunit calibrates the internal and external parameters of each sensor when the sensor is installed, and the online calibration subunit, after receiving the time-synchronized data and the initial sensor external parameters, for sensors whose data modality is a spatial point cloud, namely, millimeter-wave radar and lidar sensors, respectively ① uses a mixed Gaussian model to estimate the distribution of the point cloud in three-dimensional space, namely and calculate the Rényi entropy of the distribution ② Use the kernel density of the edge and joint histogram of the laser radar point cloud intensity and camera image grayscale to estimate the distribution of the laser radar point cloud and camera image, and use the entropy of the edge distribution and joint distribution to calculate the mutual information of the point cloud and image M(X,Y)=H(X)+H(Y)-H(X,Y), and then complete the online calibration of the external parameters through the optimization algorithm, where: x represents the distribution of the measured point cloud in the real world, the subscript L represents the laser radar, R represents the millimeter wave radar, and N L and N R Respectively represent the number of points in the laser radar and millimeter wave radar point clouds, represents the set of points in all lidar point clouds, represents the set of points in all millimeter-wave radar point clouds, Indicates that the measured point cloud and After that, the posterior distribution of the true value of the point cloud is Indicates that the mean is z L (i), the covariance matrix is ​​P L Gaussian distribution of , H(x) represents the Rényi entropy of the random variable x, X and Y represent the histogram kernel density estimates of the lidar intensity and camera grayscale respectively, and M(X,Y) represents the mutual information of the random variables X and Y.

7. The autonomous driving multi-sensor data acquisition and processing system according to claim 6, characterized in that: The optimization algorithm includes: optimizing the external parameters between the laser radar and the millimeter wave radar so that the Rényi entropy of all point clouds is minimized, then optimizing the external parameters between the laser radar and the camera so that the above mutual information is minimized, and finally completing the online calibration of the external parameters.

8. The autonomous driving multi-sensor data acquisition and processing system according to claim 6, characterized in that: The internal and external parameters include: the internal parameters of the camera, the transformation matrix between the camera coordinate system and the laser radar coordinate system, the transformation matrix between the laser radar coordinate systems, and the transformation matrix between the millimeter wave radar and the laser radar, to obtain the initial position of the sensor on the test vehicle.

Images