An end-to-end camera and lidar calibration method for autonomous driving

Abstract

The present invention discloses an end-to-end camera and lidar calibration method for autonomous driving, comprising: (1) performing hierarchical feature extraction on point cloud data according to a 3D encoder; (2) performing hierarchical feature extraction on image data according to a 2D encoder; (3) performing attention fusion on point cloud and image features to obtain weighted point features; (4) inputting the hierarchical features and weighted point features obtained by the point cloud encoder into a hierarchical decoder to obtain point-by-point features; (5) predicting whether the point is within the camera's visual cone according to a two-layer shared perceptron based on the point-by-point features, and outputting a binary classification result; (6) predicting the calibration result of the point according to another two-layer shared perceptron based on the point-by-point features; (7) filtering the calibration result according to the binary classification result, and aggregating the filtered calibration results to obtain the final calibration parameters between the camera and the point cloud. The present invention can avoid the error accumulation caused by the pipeline mode and improve the calibration accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and in particular relates to an end-to-end camera and lidar calibration method for autonomous driving. Background Art

[0002] Autonomous driving is a multifunctional, integrated system capable of understanding its environment, independently maneuvering within it, and completing specific tasks. It encompasses multiple core technologies, including environmental perception, motion control, and human-machine interaction. It has evolved from its initial stages of emergency avoidance to today's fully autonomous cruising. In rail transit, advanced driver assistance systems are increasingly being used in many cities around the world. A mature autonomous driving system relies on a variety of onboard sensors installed on autonomous vehicles to capture information about the surrounding environment. As application scenarios become increasingly complex, no single sensor can independently provide stable and comprehensive environmental data for autonomous vehicles. Fusion of data from multiple sensor devices into a more precise signal for comprehensive analysis produces a consistent description of the surrounding environment. This accurately and stably represents the surrounding environment and the vehicle's state, improving the perception module's confidence in understanding complex scenarios. LiDAR and cameras are currently the most widely used sensor combination. External calibration between the two devices is crucial to integrating the disparate data captured by these two devices.

[0003] Early calibration methods between lidar and cameras focused on formulating objective functions based on specific calibration targets (chessboards, spherical calibration plates, etc.). These methods are time-consuming and cumbersome, and are difficult to apply to outdoor scenarios with high real-time requirements, hindering their practical use, especially for vehicles that require frequent recalibration due to sensor drift.

[0004] To achieve online calibration, some work has used handcrafted features such as lines or planes in point clouds and images to develop optimization functions for calibration parameters. For example, the paper "Pixel-level extrinsic self-calibration of high-resolution lidar and camera in targetless environments," published in 2021 at the IEEE Robotics and Automation Letters, a leading robotics conference, has been published. However, these methods have high requirements for scene characteristics and are difficult to generalize to complex scenes.

[0005] In recent years, more and more work has turned its attention to deep learning, using neural networks to extract richer features for subsequent calibration tasks, such as "LCCNet: LiDAR and Camera Self-Calibration Using Cost Volume Network" published in 2021 at the top conference in the field of computer vision IEEE / CVFConference on Computer Vision and Pattern Recognition Workshops. Although these methods have achieved satisfactory accuracy, they all need to rely on initial calibration parameters to project the point cloud data onto the camera plane so that it has an overlapping area with the image in the field of view.

[0006] To address this issue, the papers "DeepI2P: Image-to-Point Cloud Registration via Deep Classification," published at the IEEE Conference on Computer Vision and Pattern Recognition in 2021, and "EFGHNet: A Versatile Image-to-Point Cloud Registration Network for Extreme Outdoor Environment," published at the IEEE Robotics and Automation Letters in 2022, a leading conference in the field of robotics, proposed different frameworks to directly fuse raw point clouds and image features to predict external parameters. However, neither EFGHNet nor DeepI2P, these learning-based methods, exploit the enormous potential of end-to-end models. They typically involve complex post-processing steps, lack accuracy advantages, and incur significant time overhead. Summary of the Invention

[0007] The present invention provides an end-to-end camera and lidar calibration method for autonomous driving, which can complete prediction in a very short time while maintaining performance improvement, avoids error accumulation caused by the pipeline mode, and improves calibration accuracy.

[0008] An end-to-end camera and lidar calibration method for autonomous driving, comprising the following steps:

[0009] (1) Perform hierarchical feature extraction on the input point cloud data according to the 3D point cloud encoder;

[0010] (2) Performing hierarchical feature extraction on the input image data according to the 2D image encoder;

[0011] (3) Attention fusion is performed based on the point cloud features and image features obtained above to obtain weighted point features that contain both point cloud and image information;

[0012] (4) The hierarchical features obtained by the 3D point cloud encoder and the weighted point features after attention fusion are input into the hierarchical decoder to obtain point-by-point features;

[0013] (5) Based on the two-layer shared perceptron, the point-by-point feature is predicted whether the point is within the camera's viewing cone, and a binary classification result CL is output. i ∈{0, 1}; where i represents the i-th point cloud, 1 represents that the point is within the camera's viewing cone, and 0 represents the opposite;

[0014] (6) The calibration result F of each point is predicted based on the other two layers of shared perceptrons. i ;

[0015] (7) The binary classification result CL obtained according to step (5) i , filter the calibration results obtained in step (6) to obtain the calibration results F that belong to the camera cone range f , f∈{s|CL s =1}, the filtered calibration results are aggregated to obtain the final calibration parameters between the camera and the point cloud.

[0016] The present invention can obtain calibration parameters between the point cloud and image data, that is, the rotation change matrix, given a frame of unaligned point cloud data and image data.

[0017] In step (1), the PointNet++ network is used as a 3D point cloud encoder to extract hierarchical features of the point cloud data; the overall point cloud downsampling process is carried out three times, and the point cloud coordinates and corresponding features after each downsampling are recorded as and Where l represents the layer, CP l Represents the dimension of the point cloud features of the lth layer, N l Represents the number of point clouds.

[0018] In step (2), the ResNet34 network is used as a 2D image encoder to perform hierarchical feature extraction on the image data, and the image features extracted from different layers are recorded as Among them, CI l Represents the image feature dimension, H l Represents the height of the feature map, W l Represents the width of the feature map.

[0019] The specific process of step (3) is:

[0020] (3-1) First, extract the global features of the image Compressed into dimension, and then stack the second dimension, the resulting feature vector belongs to Combine it with the extracted point cloud features Splice and get features

[0021] (3-2) Input two layers of shared perceptron to generate weight matrix Provides N l Different image feature weights of each node;

[0022] (3-3) The weight matrix and image features Perform matrix multiplication to finally obtain weighted point features

[0023] The specific process of step (4) is:

[0024] The weighted point features obtained in step (3) Hierarchical features obtained from the point cloud encoding process Splicing is performed on the feature dimension, and then the K-nearest neighbor-based interpolation algorithm in PointNet++ is used to restore the downsampled point features to the input point cloud scale to obtain the final point-by-point feature β∈R CP×N , where N is the number of input point clouds.

[0025] The specific process of step (5) is:

[0026] Use the two-layer shared perceptron to perform binary classification prediction on the point-by-point features obtained by the decoder to obtain the point-by-point prediction result LG i , the value is continuous, ranging from -∞ to +∞, for LG i Use the logistic function for normalization, the formula is as follows:

[0027]

[0028] Normalized CL i The value range is from 0 to 1. Values ​​greater than 0.5 are considered 1, and values ​​less than 0.5 are considered 0.

[0029] The specific process of step (6) is:

[0030] The other two layers of shared perceptrons are used to calibrate the point-by-point features obtained by the decoder for regression prediction, and finally the point-by-point calibration prediction F is obtained.i ∈R 7 , the first three dimensions are the displacement, and the last four dimensions are a quaternion; the quaternion is used to obtain the rotation matrix through transformation.

[0031] In step (7), the filtered calibration result F f The specific process of obtaining the final calibration parameters between the camera and the point cloud is as follows:

[0032] Each input point cloud is rigidly rotated according to the predicted calibration results to obtain the target point. The aggregation process is then converted into a rigid alignment problem between the input point cloud and the target point, and the final calibration parameters are obtained using SVD decomposition.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The calibration method proposed in this invention is a completely end-to-end method. The entire system is designed as a single module, avoiding the tedious data preprocessing and feature engineering steps of previous models, and improving the performance and accuracy of the system by utilizing a powerful neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the overall framework of an end-to-end camera and lidar calibration method for autonomous driving according to the present invention;

[0036] Figure 2 This is a visualization of the misaligned point cloud and image after re-correction by the method of the present invention and the most advanced method. DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0038] like Figure 1 As shown in the figure, the framework of the present invention mainly integrates feature extraction and calibration prediction into one model, forming an end-to-end learnable network to obtain the final camera-to-lidar calibration parameters.

[0039] An end-to-end camera and lidar calibration method for autonomous driving, comprising the following steps:

[0040] S01, performing hierarchical feature extraction on the input point cloud data according to the 3D point cloud encoder;

[0041] S02, performing hierarchical feature extraction on the input image data according to the 2D image encoder;

[0042] S03, performing attention fusion based on the point cloud features and image features obtained above to obtain weighted point features containing both point cloud and image information;

[0043] S04: Input the hierarchical features obtained by the 3D point cloud encoder and the weighted point features after attention fusion into the hierarchical decoder to obtain point-by-point features;

[0044] S05: Based on the two-layer shared perceptron, the point-by-point features are predicted to determine whether the point is within the camera's viewing cone, and a binary classification result CL is output. i ∈{0, 1}; where i represents the i-th point cloud, 1 represents that the point is within the camera's viewing cone, and 0 represents the opposite;

[0045] S06, predict the calibration result F of the point based on the other two layers of shared perceptrons for each point feature i ;

[0046] S07, based on the binary classification result CL obtained in step S05 i Filter the calibration result obtained in step S06 to obtain the calibration result F that belongs to the camera cone range f , f∈{s|CL s =1}, the filtered calibration results are aggregated to obtain the final calibration parameters between the camera and the point cloud.

[0047] The present invention first performs classification prediction based on the point-by-point feature β extracted by the decoder to obtain the binary classification result CL of each point i , a value of 1 indicates that the point cloud is within the camera's frustum, and a value of 0 indicates the opposite. During training, the classification results are optimized according to the Logisitic loss function, as follows:

[0048]

[0049] Among them, L_CL represents the classification loss, y i Represents the true classification value, and N represents the number of point clouds.

[0050] According to point-by-point features Carry out calibration parameter prediction. The present invention selects quaternion as the expression of rotation transformation. Compared with the rotation matrix, its mathematical expression constraint is simpler. It only needs to normalize the 4-dimensional vector to obtain a quaternion. Therefore, for the rotation component in the calibration parameter predicted by the network, the quaternion q = [q0, q1, q2, q3] is used to represent it. The obtained quaternion is converted according to the following formula to obtain the rotation matrix:

[0051]

[0052] Among them, Rq Represents the quaternion q in the form of a rotation matrix. When transforming 3D point cloud coordinates, the rotation matrix is ​​usually used as the expression of the rotation component. The formula for rigid body transformation of 3D point cloud coordinates is as follows:

[0053]

[0054] x, y, z represent the coordinates of point cloud P in 3D space, T (q，t) Represents the rigid body change function with rotation q and displacement t.

[0055] CL i The calibration parameter F predicted to be 0 i Filter, and the remaining calibration parameters are recorded as F G ={F j |CL j =1}. Ultimately, the present invention hopes to obtain a unique calibration parameter. To this end, firstly, the predicted point-by-point calibration parameter F is used. j Perform rigid body transformation on the corresponding input point cloud coordinates to obtain the predicted spatial position of the point cloud:

[0056]

[0057] Among them, x j ,y j , z j Represents the 3D coordinates of the j-th point cloud input, q j and t j They are the calibration parameter prediction F j ∈R 7 The first 4 components and the last 3 components represent rotation and translation respectively. Similarly, the present invention uses the true value of the calibration parameter to perform a rigid body transformation on the input point cloud to obtain where the point should be after the rigid body transformation, which is recorded as

[0058] During the training phase, use and The difference between them is used as the distance loss function, and the formula is as follows:

[0059]

[0060] Where L_F represents the regression loss function, and G represents the number of remaining point clouds after classification and filtering. and The error between them continues to shrink, which also means that the calibration parameters qj and tj will continue to approach the true value.

[0061] The distance loss of each point is supervised separately, which makes it difficult for the network to form global consistency on all points. Therefore, the average loss is proposed to make up for the shortcomings of the distance loss. First, define and The center of mass:

[0062]

[0063]

[0064] Then, calculate and The cross-covariance between :

[0065]

[0066] Finally, the final calibration parameters are solved in the form of closed-form solution:

[0067] R=VU T

[0068]

[0069] Here, R represents the rotation component and t represents the displacement component. These two components constitute the final calibration parameters required. It is important to note that the closed-form solution above is obtained by performing a singular value decomposition (SVD). This operation is implemented in a differentiable manner in PyTorch, enabling end-to-end training of the network.

[0070] To verify the effectiveness of this invention, we compared it with other cutting-edge dialogue generation methods on the public KITTI and Oxford RebotCar datasets. Both the KITTI and Oxford RebotCar datasets are currently the most popular algorithm evaluation datasets in the autonomous driving field. For the KITTI dataset, approximately 20,000 frames of point clouds and images were selected for training, and approximately 2,000 frames of point clouds and images were used for evaluation. For the Oxford dataset, 130,000 frames of point clouds and images were selected for training, and approximately 20,000 frames of point clouds and images were selected for training.

[0071] This paper first compares the rotation error (RRE) and displacement error (RTE). It also compares three currently popular camera-lidar algorithms: 2D3D-MatchNet, LCCNet, MonoDepth2+USIP, MonoDepth2+GT-ICP, and LCCNet, as well as four variants of DeepI2P: Grid Cls+PnP, Fris.ClS+Inv.Proj 3D, Frus.Cls+Inv.Proj 2D, and Direct Regression.

[0072] Table 1

[0073]

[0074] As can be seen from Table 1, the end-to-end camera-lidar calibration method based on deep learning proposed in this paper (E 2 Calib), achieving optimal results across all three metrics, fully demonstrating the superiority of our algorithm. We also compared time efficiency with several variants of the popular DeepI2P method that do not rely on calibration initial values. Table 2 shows that our method significantly outperforms DeepI2P in inference speed, making it suitable for use in scenarios with high real-time requirements.

[0075] Table 2

[0076]

[0077] Finally, the visualization results of the present invention on the Oxford dataset are shown. Figure 2 As shown in the figure: (a) shows the visualization of the point cloud and image after alignment with the true value; (b) shows the visualization result after calibration of the unaligned point cloud data according to the calibration parameters predicted by DeepI2P; (c) shows the visualization result of the method E according to the present invention. 2 Visualization of the calibration results after calibrating misaligned point cloud data using the calibration parameters predicted by Calib. This demonstrates that our method can better align cameras and LiDAR devices than the currently popular DeepI2P method, achieving higher accuracy in predicting calibration parameters.

[0078] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end camera and lidar calibration method for autonomous driving, characterized in that: The following steps are involved: (1) Perform hierarchical feature extraction on the input point cloud data according to the 3D point cloud encoder; The specific process is: The PointNet++ network is used as a 3D point cloud encoder to extract hierarchical features from point cloud data. The point cloud is downsampled three times, and the point cloud coordinates and corresponding features after each downsampling are recorded as and Where l represents the layer, CP l Represents the dimension of the point cloud features of the lth layer, N l Represents the number of point clouds; (2) Performing hierarchical feature extraction on the input image data according to the 2D image encoder; The specific process is: The ResNet34 network is used as a 2D image encoder to perform hierarchical feature extraction on image data, and the image features extracted from different layers are recorded as Among them, CI l Represents the image feature dimension, H l Represents the height of the feature map, W l Represents the width of the feature map; (3) Attention fusion is performed based on the obtained point cloud features and image features to obtain weighted point features that contain both point cloud and image information. The specific process is as follows: (3-1) First, extract the global features of the image Compressed into dimension, and then stack the second dimension, the resulting feature vector belongs to Combine it with the extracted point cloud features Splice and get features (3-2) Input two layers of shared perceptron to generate weight matrix Provides N l Different image feature weights of each node; (3-3) The weight matrix and image features Perform matrix multiplication to finally obtain weighted point features (4) The hierarchical features obtained by the 3D point cloud encoder and the weighted point features after attention fusion are input into the hierarchical decoder to obtain point-by-point features; (5) Based on the two-layer shared perceptron, the point-by-point feature is predicted whether the point is within the camera's viewing cone, and a binary classification result CL is output. i ∈{0,1}; where i represents the i-th point cloud, 1 represents that the point is within the camera's viewing cone, and 0 represents the opposite; (6) The calibration result F of each point is predicted based on the other two layers of shared perceptrons. i ; (7) The binary classification result CL obtained according to step (5) i , filter the calibration results obtained in step (6) to obtain the calibration results F that belong to the camera cone range f , f∈{s|CL s =1}, the filtered calibration result F f Aggregation is performed to obtain the final calibration parameters between the camera and the point cloud.

2. The end-to-end camera and lidar calibration method for autonomous driving according to claim 1, characterized in that: The specific process of step (4) is: The weighted point features obtained in step (3) Hierarchical features obtained from the point cloud encoding process Splicing is performed on the feature dimension, and then the K-nearest neighbor-based interpolation algorithm in PointNet++ is used to restore the downsampled point features to the input point cloud scale to obtain the final point-by-point feature β∈R CP×N , where N is the number of input point clouds.

3. The end-to-end camera and lidar calibration method for autonomous driving according to claim 1, characterized in that: The specific process of step (5) is: Use the two-layer shared perceptron to perform binary classification prediction on the point-by-point features obtained by the decoder to obtain the point-by-point prediction result LG i , the value is continuous, ranging from -∞ to +∞, for LG i Use the logistic function for normalization, the formula is as follows: Normalized CL i The value range is from 0 to 1. Values ​​greater than 0.5 are considered 1, and values ​​less than 0.5 are considered 0.

4. The end-to-end camera and lidar calibration method for autonomous driving according to claim 1, characterized in that: The specific process of step (6) is: The other two layers of shared perceptrons are used to calibrate the point-by-point features obtained by the decoder for regression prediction, and finally the point-by-point calibration prediction F is obtained. i ∈R 7 , the first three dimensions are the displacement, and the last four dimensions are a quaternion; the quaternion is used to obtain the rotation matrix through transformation.

5. The end-to-end camera and lidar calibration method for autonomous driving according to claim 4, characterized in that: In step (7), the filtered calibration result F f The specific process of obtaining the final calibration parameters between the camera and the point cloud is as follows: Each input point cloud is rigidly rotated according to the predicted calibration results to obtain the target point. The aggregation process is then converted into a rigid alignment problem between the input point cloud and the target point, and the final calibration parameters are obtained using SVD decomposition.

Images