Multi-sensor perception fusion method for autonomous driving technology

Abstract

This invention relates to a multi-sensor perception fusion method for autonomous driving technology, comprising: S1. Sensor calibration, which calibrates the coordinates of the applied sensors and establishes the transformation relationship between different sensor coordinate systems; S2. Target fusion, which performs data fusion based on the point cloud data of LiDAR, fusing obstacle information obtained from camera data and millimeter-wave radar with the point cloud data, and determining the obstacle information in the LiDAR point cloud data; S3. Target set determination, which summarizes to obtain the final obstacle set. This invention achieves mutual fusion of target detection among LiDAR, camera, and millimeter-wave radar, thereby fully utilizing the characteristics of data acquired by different sensors to obtain more accurate obstacle information, thus providing the precision, accuracy, and robustness of obstacle detection, and achieving better environmental perception capabilities.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor vision fusion technology for autonomous driving, and specifically to a multi-sensor perception fusion method for autonomous driving technology. Background Technology

[0002] In specific scenarios such as mining areas, factory areas, and port areas, the types of traffic participants are relatively homogeneous, and the scenarios are relatively predictable, which is conducive to the implementation of autonomous driving technology and represents a promising area for its application and development. While these application scenarios have certain inherent advantages for developing autonomous driving technology, they also present some unavoidable challenges.

[0003] Unlike the smooth, well-planned roads of cities, open-pit mines present harsh working conditions. Obstacles are characterized by their randomness and concealment, such as random rockfalls from heavy mining trucks and unexpected obstacles appearing in blind spots around corners. Due to the varying characteristics of different sensors, a single sensor cannot accurately identify and classify obstacles, thus hindering obstacle detection and posing significant safety hazards. Furthermore, the high dust levels and severe jolting of mining trucks in these environments increase the randomness of sensor malfunctions, leading to more missed and false detections. Overall, the harsh working environment in mines places higher demands on the obstacle recognition capabilities of sensors and algorithms, requiring both high accuracy and reliability in obstacle identification. Summary of the Invention

[0004] To improve the accuracy and reliability of obstacle recognition by autonomous driving sensors and systems in mining applications, this invention provides a multi-sensor perception fusion method for autonomous driving technology.

[0005] The technical solution of the present invention is as follows:

[0006] A multi-sensor perception fusion method for autonomous driving technology includes the following steps:

[0007] S1. Sensor calibration: Perform coordinate calibration on the applied sensors and establish the transformation relationship between different sensor coordinate systems;

[0008] S2. Target fusion: Data fusion is performed based on the point cloud data of LiDAR. The obstacle information obtained from camera data and millimeter-wave radar is fused with the point cloud data, and the obstacle information in the point cloud data of LiDAR is determined.

[0009] S3. Target set determination: The obstacle set determined by the fused camera data, the obstacle set determined by the fused millimeter-wave radar data, and the obstacle set obtained by the lidar data are combined to obtain the final obstacle set.

[0010] Preferably, the S2 target fusion step includes:

[0011] S21. Camera data fusion: Using an image 2D target detection algorithm, output 2D detection boxes for obstacles, determine all point clouds projected within the 2D detection boxes, and use a clustering algorithm to cluster all point clouds of the obstacle to construct a 3D bounding box for the obstacle and update the obstacle information obtained from the camera data.

[0012] Preferably, in the S21 camera data fusion step, after obtaining the 2D detection box of the obstacle, the estimated distance of the obstacle is first determined by interpolation:

[0013] Perform calibration within a range of 0-100 meters, and then convert the world coordinate system of the calibration points to the pixel coordinate system;

[0014] Within the obtained 2D target frame of the camera, take the center point of the bottom edge, find that it lies within the range of 0-100m, and set this value as Zc according to the scale.

[0015] According to Zc, the 2D target box of the camera is restored to the real three-dimensional coordinates by using the pixel coordinate system to world coordinate system transformation.

[0016] Preferably, the S2 target fusion step includes:

[0017] S22. Millimeter-wave radar data fusion: detect millimeter-wave target bounding boxes of obstacles in millimeter-wave radar data, determine all point clouds projected within the millimeter-wave target bounding boxes, and update the obstacle information obtained by millimeter-wave radar based on the point cloud data within the millimeter-wave target bounding boxes.

[0018] Preferably, the S2 target fusion step includes:

[0019] S23. Obstacle detection based on LiDAR data: The point cloud is processed directly, and combined with a pure LiDAR 3D target detection algorithm, the obstacle information in the LiDAR point cloud data is determined.

[0020] Preferably, the S2 target fusion step includes:

[0021] S21. Camera data fusion: Utilizing a 2D object detection algorithm, output 2D bounding boxes for obstacles, determine all point clouds projected within the 2D bounding boxes, and then use a clustering algorithm to cluster all point clouds of the obstacle to construct a 3D bounding box for the obstacle, updating the obstacle information obtained from the camera data.

[0022] S22. Millimeter-wave radar data fusion: This process detects millimeter-wave target bounding boxes of obstacles within the millimeter-wave radar data, identifies all point clouds projected within these bounding boxes, and updates the obstacle information obtained from the millimeter-wave radar based on the point cloud data within the bounding boxes.

[0023] S23. Obstacle detection based on LiDAR data: The point cloud is processed directly, and combined with a pure LiDAR 3D target detection algorithm, the obstacle information in the LiDAR point cloud data is determined.

[0024] Step S23 is executed later than steps S21 and S22. In steps S21 and S22, the point cloud data of the part covered by the camera or millimeter-wave radar obstacle is directly removed from the lidar point cloud data.

[0025] Preferably, in the S3 target set determination step, after the obtained obstacle information is stored in the dataset, time synchronization processing is performed based on the lidar time, and obstacle information exceeding a preset threshold is discarded.

[0026] Preferably, in the S3 target set determination step, the obstacle information in the dataset is calculated based on the intersection-union ratio of the detection boxes and merged into a single target box.

[0027]

[0028]

[0029] in, Let represent the set of bounding boxes for the i-th obstacle, and IOU denote the calculation method. This represents the target bounding box after merging the i-th obstacle. Let represent the intersection of the volumes of all bounding boxes of the i-th obstacle. Let represent the union of the volumes of all target boxes of the i-th obstacle, and p represent the set threshold.

[0030] Preferably, the S1 sensor calibration step includes an internal calibration process and a joint calibration process;

[0031] The internal calibration process uses a unified reference coordinate system for data from sensors of the same type.

[0032] Joint calibration establishes the correspondence between different sensor coordinate systems based on internal calibration. During joint calibration, it is necessary to calibrate the internally calibrated camera coordinate system and the internally calibrated millimeter-wave radar coordinate system to the internally calibrated lidar coordinate system.

[0033] This invention achieves the fusion of target detection between lidar, cameras, and millimeter-wave radar. This allows for full utilization of the characteristics of data acquired by different sensors, resulting in more accurate obstacle information and improved precision and accuracy in obstacle detection. Because it relies almost entirely on the results of a single sensor, the robustness of obstacle detection is also enhanced. Overall, it achieves superior environmental perception capabilities compared to processing multiple sensor channels independently. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the specific operation process of the multi-sensor perception fusion method for autonomous driving technology of the present invention.

[0035] Figure 2 This is a flowchart of the multi-sensor perception fusion method for autonomous driving technology of the present invention;

[0036] Figure 3 This is a schematic diagram of image depth recovery for the multi-sensor perception fusion method of autonomous driving technology of the present invention. Detailed Implementation

[0037] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. In this specification, the dimensions of the drawings do not represent the actual dimensions. They are only used to illustrate the relative positional and connection relationships between the components. Components with the same name or the same reference numeral represent similar or identical structures and are limited to illustrative purposes.

[0038] Unmanned transportation systems within mining areas typically include different types of driverless mining trucks and a cloud control platform. Optional features may also include roadside augmentation systems. The driverless mining trucks determine their travel paths by integrating the cloud control system with their own environmental perception data.

[0039] To ensure the environmental perception capabilities of autonomous mining trucks, a large number of sensors are typically installed. For example, a certain type of unmanned mining vehicle is equipped with millimeter-wave radar, lidar, and cameras. LiDAR is arranged around the vehicle to achieve 360° scene coverage; cameras are arranged around the front and rear of the vehicle to achieve 180° scene coverage. Clearly, the scanning areas of multiple sensors partially overlap, providing a certain degree of redundancy. Therefore, obstacle information can be determined from different sensor data, improving the accuracy of obstacle recognition.

[0040] However, existing technologies typically process information from different sensor channels independently, then aggregate the obstacle information obtained from each sensor channel to form the obstacle set when the system plans the autonomous driving path. Similar technical solutions, relying solely on increasing the type and number of sensors, improve obstacle recognition accuracy, but there is still room for improvement. In reality, different sensors have different application scenarios and their own advantages. Therefore, in specific application scenarios, some sensors may have higher accuracy in obstacle data, while others may have lower accuracy or lower data quality. For example, millimeter-wave radar in mining scenarios suffers from high echo noise and low resolution, and lacks height information, while camera-captured planar images of people lack depth information and cannot determine obstacle distances. Independently processing information from different obstacles does not overcome the inherent problems of these sensors and still leaves room for improvement. The technical solution of this invention aims to achieve fusion processing of data from different types of sensors. By cross-processing different sensor data, it improves the positional accuracy, precision, and reliability of obstacle recognition. Its ultimate goal is to achieve a synergistic effect (1+1>2) in obstacle recognition through sensor combination.

[0041] A schematic diagram illustrating the specific operation process of the multi-sensor perception fusion method for autonomous driving technology of the present invention is shown below. Figure 1 The overall flowchart is as follows Figure 2 The multi-sensor perception fusion method for autonomous driving technology generally includes the following steps:

[0042] S1. Sensor Calibration. This involves calibrating the coordinates of the applied sensors to establish transformation relationships between different sensor coordinate systems. This generally consists of two parts: internal calibration between sensors of the same type and joint calibration between sensors of different types. Internal calibration between sensors of the same type involves unifying the reference coordinate system for data from sensors of the same type. For example, for multiple LiDARs, their device coordinate system is typically defined by the manufacturer on the LiDAR itself. Since the installation locations of the multiple LiDARs are different, it is necessary to unify the coordinate systems of the multiple LiDARs to the same coordinate system. The same applies to multiple camera sensors.

[0043] Joint calibration establishes the correspondence between different sensor coordinate systems based on internal calibration. This involves establishing coordinate system transformation relationships between LiDAR, millimeter-wave radar, and cameras. Formally, it involves determining the transformation matrix between different sensor coordinate systems. In practice, the internally calibrated LiDAR coordinate system is generally used as the intermediate coordinate system because LiDAR data has a general advantage. In other words, during joint calibration, the internally calibrated camera coordinate system and the internally calibrated millimeter-wave radar coordinate system need to be calibrated to the internally calibrated LiDAR coordinate system. Essentially, this involves determining the extrinsic parameter matrix of the camera relative to the LiDAR coordinate system and the transformation matrix of the millimeter-wave radar. Finally, optionally, the LiDAR can be jointly calibrated with the IMU (Inertial Measurement Unit) in the autonomous driving system, calibrating the LiDAR coordinate system to the final mining truck coordinate system.

[0044] The main formula for coordinate transformation is: Where a is a vector in the sensor coordinate system, R is the rotation matrix, t is the translation vector, and a' is the vector after coordinate transformation. The calibration process requires collecting multiple vectors a and a' experimentally, and then solving the system of equations to obtain R and t.

[0045] S2. Target Fusion. Generally, due to the high reliability of LiDAR data, it is typically used as the basis for data fusion. This is precisely why the LiDAR coordinate system is used as the intermediate target in S1. The data fusion process based on LiDAR mainly includes several sub-steps: S21. Camera Data Fusion, S22. Millimeter-Wave Radar Data Fusion, S23. Obstacle Detection Based on LiDAR Data.

[0046] S3. Target set determined.

[0047] Step S2 will yield the obstacle set determined by the fused camera data, the obstacle set determined by the fused millimeter-wave radar data, and the obstacle set determined by the lidar data. These sets will be combined to obtain the final obstacle set. Finally, obstacle matching and target tracking can be performed based on this final obstacle set.

[0048] Sensors generally have a certain degree of redundancy. Specifically, the following are the detailed operations in the S2 target fusion step and the S3 target set determination step.

[0049] S21. Camera Data Fusion. Utilizing 2D object detection algorithms, 2D bounding boxes for obstacles are output. Camera data fusion primarily addresses the problem that 2D obstacle detection lacks depth information, making it impossible to determine the spatial location of obstacles.

[0050] First, the laser point cloud is projected from the LiDAR coordinate system to the pixel coordinate system. Upon receiving the target bounding box from the camera, time synchronization is performed to ensure data synchronization between the sensors. This involves checking if the time difference between the camera's target bounding box and the LiDAR's time difference is greater than a preset time difference. Considering the LiDAR's sampling period is approximately 100ms, this preset time is designed to be less than this sampling period, therefore 90ms can be chosen. Then, all point clouds projected within the 2D detection box are traversed, and a clustering algorithm is used to cluster all point clouds of the obstacle, thereby constructing the obstacle's 3D bounding box and obtaining the obstacle's information.

[0051] The information in the camera's 2D target bounding box includes:

[0052]

[0053] in, This represents the camera bounding box of the i-th obstacle. This represents the center point coordinates of the i-th obstacle bounding box detected by the camera in pixel coordinates. This indicates the width and height of the bounding box for the i-th obstacle detected by the camera. This indicates the category of the i-th obstacle detected by the camera. This represents the confidence level that the camera has detected the i-th obstacle category.

[0054] LiDAR point cloud information includes:

[0055]

[0056] in, This represents the i-th laser point cloud. This represents the coordinates of the i-th laser point cloud. This represents the reflection intensity of the i-th laser point cloud.

[0057] The laser point cloud is transformed from the world coordinate system to the pixel coordinate system as follows:

[0058] (4)

[0059] Among them, z c Represents the camera coordinate system coordinates, u i v i f represents the coordinates of the i-th point cloud in the pixel coordinate system. x ,f y u0, v0 represent the camera's intrinsic parameters, and R and T represent the camera's extrinsic parameters, rotation, and translation matrices. This represents the coordinates of the i-th laser point cloud in the world coordinate system.

[0060] Find all laser point cloud projections within the 2D target bounding box in pixel coordinates, as well as the laser point cloud information mapped to the world coordinate system, as follows:

[0061]

[0062]

[0063] in, Let represent the pixel coordinates of the i-th laser point cloud projection within the 2D target bounding box. This represents the world coordinates mapped from the pixel coordinates of the i-th laser point cloud projection within the 2D target bounding box.

[0064] All identified laser point clouds are clustered using the BFS algorithm, and the 3D bounding boxes of the obstacles are output as follows:

[0065]

[0066] in, This represents the target bounding box fused from the i-th obstacle camera and the laser beam. This indicates the coordinates of the center point of the obstacle's bounding box. This indicates the width, height, and length of the obstacle's bounding box. Laser point cloud representing obstacles.

[0067] Alternatively, after obtaining the 2D detection box of the obstacle, the estimated distance of the obstacle can be determined by interpolation.

[0068] Perform calibration within a range of 0-100 meters, and then convert the world coordinate system of the calibration points to the pixel coordinate system;

[0069] The obtained 2D bounding box from the camera is used to locate the center point of its bottom edge. This point lies within a range of 0-100m, and a value is set to Zc (depth recovery) based on the scale. Figure 3 As shown

[0070]

[0071] in, This represents the pixel coordinates of the midpoint of the bottom edge of the target box of the i-th camera.

[0072] According to Zc, the 2D target bounding box of the camera is restored to its actual 3D coordinates by using a pixel coordinate system to world coordinate system transformation, as follows:

[0073] (15)

[0074] (16)

[0075] in, Let represent the 3D bounding box of the depth recovery of the i-th camera target bounding box. This represents the coordinates of the center point of the front view of the 3D target bounding box recovered from the depth recovery of the i-th camera target bounding box.

[0076] S22. Millimeter-wave radar data fusion.

[0077] Step S4, the millimeter-wave radar information is as follows:

[0078]

[0079] in, This represents the millimeter-wave target bounding box of the i-th obstacle. This represents the coordinates of the center point of the millimeter-wave target bounding box of the i-th obstacle. This represents the width and length of the millimeter-wave target bounding box for the i-th obstacle. This represents the millimeter-wave target category of the i-th obstacle.

[0080] LiDAR point cloud information includes:

[0081] .

[0082] Find all point clouds projected within the millimeter-wave target bounding box, and output the maximum and minimum heights of the point clouds, as follows:

[0083]

[0084]

[0085]

[0086]

[0087] in, This represents the point cloud within the i-th millimeter-wave target bounding box. The maximum and minimum heights of all point clouds within the i-th millimeter-wave target bounding box. The z-value represents the coordinates of the center point of the i-th millimeter-wave target frame. The height of the i-th millimeter-wave target bounding box. Based on the above information from the specific point cloud, the 3D bounding box for the fusion of millimeter-wave and laser data can be determined, specifically:

[0088] .

[0089] S23. Obstacle detection based on LiDAR data. This only requires direct processing of the point cloud and combining it with a pure LiDAR 3D target detection algorithm to output obstacle target information, as detailed below:

[0090]

[0091] in, This represents the bounding box for the i-th obstacle detected using pure laser 3D target detection. This represents the coordinates of the center point of the bounding box for the i-th obstacle detected by pure laser 3D target detection. This represents the width, height, and length of the bounding box for the i-th obstacle detected using pure laser 3D target detection. This represents the rotation angle, category, and category confidence of the bounding box for the i-th obstacle in pure laser 3D target detection. In practice, the order of steps S21, S22, and S23 is not important and can be executed in any available order. However, preferably, the obstacle detection step based on LiDAR data in step S23 can be executed last, and the following task can be further performed in steps S21 and S22: for point cloud data used to handle occlusion during detection—that is, the part of the point cloud data covered by obstacles from the camera or LiDAR—it can be directly removed from the LiDAR point cloud data. This significantly reduces the computational load of target detection in step S23, but has a small impact on the final obstacle set because the removed obstacles have already been confirmed. Experiments show that this alone can improve the efficiency of target detection in LiDAR point clouds by an average of 1 / 3.

[0092] S3. Target Set Determination. The final obstacle set can be obtained by simply combining the obstacle set determined by the fused camera data, the obstacle set determined by the fused millimeter-wave radar data, and the obstacle set obtained by the lidar data into a dataset. However, we need to ensure temporal consistency of the data, that is, to ensure that all obstacles represent obstacle information from the same time period. However, considering the potential delays due to inconsistent frame rates among the sensors, after storing the obtained obstacle information in the dataset, we continue to perform time synchronization processing based on the lidar time, discarding obstacle information exceeding a preset threshold (e.g., the aforementioned 90ms).

[0093] Furthermore, it cannot be ruled out that there may be duplicates among obstacles obtained through different methods. In order to accurately determine the set and number of obstacles, the obstacle information in the dataset can be calculated based on the Intersection over Union (IOU) of the detection boxes and merged into a single target box.

[0094]

[0095]

[0096] in, Let represent the set of bounding boxes for the i-th obstacle, and IOU denote the calculation method. This represents the target bounding box after merging the i-th obstacle. Let represent the intersection of the volumes of all bounding boxes of the i-th obstacle. Let represent the union of the volumes of all bounding boxes of the i-th obstacle, and p represent the set threshold. Based on This enables subsequent data matching and target tracking processes.

[0097] The target tracking process generally employs the extended Kalman filter algorithm, as detailed below:

[0098]

[0099]

[0100]

[0101]

[0102] in, Represents the state variable at time tk. This represents the coordinates of the center point of the j-th target. This represents the length, width, and height of the j-th object detection box. This represents the rate of change of each corresponding variable over time, initialized to 0. Let represent the mapping matrix from state space to measurement space, the covariance matrix of the state variables at time tk, the transpose of the mapping matrix from state space to measurement space, the covariance matrix of the observation vector, and the Kalman gain, respectively. This represents the covariance matrix of the results at time tk. and represent the result of the k-th Kalman prediction and the actual state matrix observation, respectively.

[0103] This multi-sensor perception fusion method for autonomous driving technology can effectively integrate the characteristics of multiple sensors to achieve obstacle detection in complex mining scenarios. It can utilize different sensors for targeted perception fusion in multiple scenarios, improving the robustness of perception in open-pit mines. Furthermore, it can also provide a method for constructing 3D bounding boxes for rockfall detection based on pure camera scenarios in mining environments.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-sensor perception fusion method for autonomous driving technology, characterized in that, Includes the following steps: S1. Perform coordinate calibration on the applied sensors and establish the transformation relationship between different sensor coordinate systems; S2. Based on the point cloud data of the lidar, the camera data and millimeter-wave radar data are fused with the point cloud data of the lidar respectively, and the obstacle information in the point cloud data of the lidar is determined. S2 includes: S21. First, use a 2D target detection algorithm to output the 2D detection box of the obstacle. Then, determine all point clouds projected within the 2D detection box and use a clustering algorithm to cluster all point clouds of the obstacle to construct the 3D bounding box of the obstacle, thereby updating the obstacle information obtained based on camera data. S22. First, the millimeter-wave target bounding boxes of obstacles in the millimeter-wave radar data are obtained by detection, and all point clouds projected within the millimeter-wave target bounding boxes are determined. Then, the obstacle information in the millimeter-wave radar data is updated based on the point cloud data within the millimeter-wave target bounding boxes. S23. Process the lidar point cloud data that is not covered by the camera or millimeter-wave radar, and combine it with the pure laser 3D target detection algorithm to determine the obstacle information in the lidar point cloud data. In steps S21 and S22, the point cloud data of the part covered by the camera or millimeter-wave radar obstacle is directly removed from the lidar point cloud data. S3. The obstacle sets corresponding to the fused camera data, the obstacle sets corresponding to the fused millimeter-wave radar data, and the obstacle sets corresponding to the point cloud data of the lidar are summarized to obtain the final obstacle set. The point cloud data of the lidar is: in, This represents the i-th laser point cloud. This represents the coordinates of the i-th laser point cloud. This represents the reflection intensity of the i-th laser point cloud; The step of determining all point clouds projected within the millimeter-wave target bounding boxes of obstacles obtained from millimeter-wave radar data includes: in, Let represent the coordinates of all point clouds within the i-th millimeter-wave target bounding box, respectively. This represents the coordinates of the center point of the millimeter-wave target bounding box of the i-th obstacle. Let the width and length of the millimeter-wave target bounding box for the i-th obstacle be represented respectively. Let these represent the maximum and minimum heights of all point clouds within the i-th millimeter-wave target bounding box, respectively. The z-value represents the coordinates of the center point of the i-th millimeter-wave target frame. This represents the height of the i-th millimeter-wave target box.

2. The multi-sensor perception fusion method for autonomous driving technology as described in claim 1, characterized in that, S21 further includes: After obtaining the 2D detection box of the obstacle, the estimated distance of the obstacle is first determined by interpolation; Perform calibration within a range of 0-100 meters, and then convert the world coordinate system of the calibration points to the pixel coordinate system; Within the obtained 2D detection frame of the obstacle, take the center point of the bottom edge, find that it lies within the range of 0-100m, and set the value as Zc according to the scale. Based on Zc, the pixel coordinates of the 2D detection box of the obstacle are restored to the real three-dimensional coordinates by using the transformation rules of pixel coordinate system to world coordinate system.

3. The multi-sensor perception fusion method for autonomous driving technology as described in claim 1, characterized in that, In step S3, after the obtained obstacle information is stored in the dataset, time synchronization processing is performed based on the lidar time, and obstacle information exceeding a preset threshold is discarded.

4. The multi-sensor perception fusion method for autonomous driving technology as described in claim 1, characterized in that, In step S3, the obstacle information in the dataset is calculated based on the intersection-union ratio of the detection boxes to merge them into a single target box. in, Let represent the set of bounding boxes for the i-th obstacle, and let IOU represent the ratio of the intersection of the volumes of the bounding boxes to the union of the volumes of the bounding boxes. This represents the target bounding box after merging the i-th obstacle. Let represent the intersection of the volumes of all bounding boxes of the i-th obstacle. Let represent the union of the volumes of all target boxes of the i-th obstacle, and p represent the set threshold.

5. The multi-sensor perception fusion method for autonomous driving technology as described in claim 1, characterized in that, S1 includes an internal calibration process and a joint calibration process; The internal calibration process uses a unified reference coordinate system for data from sensors of the same type. Joint calibration establishes the correspondence between different sensor coordinate systems based on internal calibration. During joint calibration, the internally calibrated camera coordinate system and the internally calibrated millimeter-wave radar coordinate system are calibrated to the internally calibrated lidar coordinate system.

Images