A camera-radar extrinsic parameter self-calibration method and system based on deep learning

Abstract

The application discloses a camera-radar external parameter self-calibration method and system based on deep learning, which effectively deals with the real-time online external parameter joint calibration problem in the motion scene of a multi-sensor device. The method comprises the following steps: acquiring camera and laser radar synchronous data; calibrating the camera internal parameter and acquiring the initial relative external parameter; acquiring the projection graph of the initial point cloud data in the image coordinate system; quantifying the loss amount of the point cloud pixelization process by the compensation module; acquiring the predicted depth information of the camera image through the depth estimation model; constructing a camera-laser radar joint external parameter online self-calibration system, and sequentially completing the joint external parameter online calibration through the feature extraction module, the data fusion module, the external parameter regression module and the feature update module. The system based on the method realizes full-automatic online real-time self-calibration, meets the requirements of high precision, real-time performance and high reliability, and has good calibration effect.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a camera-radar extrinsic parameter self-calibration method and system based on deep learning. Background Technology

[0002] Multi-device sensor data fusion has always been an important research direction driving technological progress, playing a crucial role, especially in the fields of complex environment perception and intelligent decision-making. In autonomous driving technology, the application of sensor data fusion is particularly significant, with the complementary fusion of cameras and LiDAR often used to improve the accuracy and robustness of environmental perception. To achieve this data fusion, an efficient and high-precision cross-device joint extrinsic parameter calibration scheme is an essential and critical step.

[0003] Camera sensors primarily acquire two-dimensional RGB image information of the environment. This type of data contains rich texture, color, and detail, but lacks depth information, thus failing to directly reflect the position of objects in three-dimensional space. LiDAR sensors, on the other hand, calculate the time difference by emitting lasers into the environment and receiving the reflected beams, thereby estimating the distance from the target point to the device. Combined with the emission angle, LiDAR can generate the precise position of the target point in three-dimensional space. However, due to limitations in LiDAR scanning methods and emission density, the data it generates typically appears as a non-uniformly distributed sparse point cloud, making it difficult to fully depict the details of complex environments.

[0004] By complementary fusion of RGB images and LiDAR point cloud data, the deficiencies of each data source can be effectively compensated for. RGB images provide detailed color and texture information, while LiDAR point clouds provide accurate 3D geometric information. Combining the two significantly improves the accuracy and richness of environmental perception. This fusion technology not only provides reliable support for navigation, obstacle detection, and path planning in autonomous vehicles but also lays a solid foundation for the development of intelligent systems in related fields. High-precision joint extrinsic parameters are a key guarantee for multi-sensor data fusion.

[0005] Current calibration methods for relative extrinsic parameters between multiple sensors mostly employ offline calibration, requiring specific calibration environments and equipment. However, unforeseen factors during operation, such as equipment vibration or collisions, can alter the relative extrinsic parameters, necessitating recalibration, which is time-consuming and labor-intensive. Currently, there is a lack of effective online automatic calibration methods and systems for sensor calibration. Summary of the Invention

[0006] Based on this, the present invention fully considers the correlation characteristics of the two types of sensor data and proposes a camera-LiDAR joint extrinsic parameter online self-calibration method and system based on a deep learning network model. It successfully realizes the automatic calibration of the relative extrinsic parameters between sensors, can correct extrinsic parameter errors in real time, greatly improves calibration efficiency, and ensures high accuracy.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A deep learning-based method for joint camera-radar extrinsic parameter calibration includes the following steps:

[0009] Step 1) Establish a camera and lidar data acquisition platform to obtain environmental data collected by the camera and lidar at the same time and remove errors caused by motion distortion;

[0010] Step 2) Use a standard device to obtain the intrinsic parameters of the camera model, including the intrinsic projection matrix and distortion coefficients. Obtain the initial relative extrinsic parameters and convert the rotation matrix into a quaternion representation. The rotation matrix in the extrinsic parameter matrix can be expressed as a quaternion:

[0011]

[0012] Unit vector u = (u x u y u z Let P be the axis of rotation, and θ be the rotation angle around the axis. For each point in the radar coordinate system, the homogeneous coordinates P... L The coordinates (0, x, y, z) can all be converted to point coordinates in the camera coordinate system using the following formula:

[0013] P C =qP L q * +t

[0014] Where q * Let q be the conjugate quaternion, and t be the translation vector in the extrinsic matrix;

[0015] Step 3) Based on the initial relative extrinsic parameters, the lidar point cloud data is transformed into the camera coordinate system, and then projected into a point cloud projection map under the camera imaging plane using the camera intrinsic parameter matrix and distortion model. The pixel value represents the depth information.

[0016] Step 4) Introduce a depth estimation model to obtain depth estimates of the RGB image, which serve as a new set of features in addition to texture color. By explicitly specifying depth information as an important direction for feature extraction, the environmental geometric consistency between the two datasets can be fully calculated.

[0017] Step 5) Considering the quantization error during point cloud projection, a spatial quantization compensation module is introduced to interpolate the RGB image data. Specifically, the quantization projection error compensation module retains the decimal part of the UV coordinates of the image during pixelation (u... p u q This process performs bilinear interpolation on the RGB image data to compensate for quantization loss to the greatest extent possible and improve the utilization of point cloud data. The interpolated pixel position values ​​are as follows:

[0018] P′=(1-u p (1-u) q )P 11 +u p (1-u q )P 21 +(1-u p )u q P 12 +u p u q P 22

[0019] Among them, P 11 P 12 P 21 P 22 These represent the pixel values ​​of the four pixels immediately adjacent to the current position at its lower right. Image edges are filled with mirrored values.

[0020] Step 6) The RGB image and point cloud projection map are used as input data for the online calibration network, respectively, and output predicted extrinsic parameters after feature extraction, data fusion, and extrinsic parameter regression. The feature extraction module is divided into an image branch and a point cloud branch. Both use the same ResNet18 network framework as the backbone to perform feature extraction and feature aggregation on the RGB image data and the LiDAR point cloud projection map, respectively, while reducing the resolution to 1 / 8. The feature fusion module concatenates the extracted features to achieve feature channels of the same dimension, and calculates the feature similarity correlation volume through a set of convolutional pooling modules and residual connection convolutional modules. The extrinsic parameter regression module uses the feature similarity as input to the Update Block module to update the latent variable h. t This approach aims to mine the similarity of geometric and semantic information between two types of data. The Update Block module consists of gated loop units, focusing on efficient information updates and state propagation. Based on the current input and historical state information, it flexibly adjusts the latent variable state information by resetting and updating the gate. Latent variable h t After being flattened into a one-dimensional vector by the convolutional layer, the increment Δv of the rotation quaternion vector and the translation vector is regressed through two sets of three fully connected layers. rΔv t By combining the initial extrinsic matrix, a new extrinsic matrix T = ΔTT is obtained. init , where T init The initial given extrinsic parameter matrix;

[0021] Step 7) Treat the predicted extrinsic parameters as the initial extrinsic parameters for the new round, update the radar point cloud projection image, and after n iterations, the final network output extrinsic parameters are:

[0022]

[0023] The results of multiple rounds of extrinsic parameter regression are used as loss weights in the loss error calculation, and the SmoothL1 Loss for the rotation quaternion vector and translation vector are calculated respectively. To more accurately supervise the global matching effect, a set of points uniformly distributed in 3D space is designed, and the spatial distance error between the predicted extrinsic parameters and the actual extrinsic parameters is calculated. Simultaneously, a decay coefficient is set, and the weights of the loss function in each iteration are configured. The loss function is designed as follows:

[0024] L r =SmoothL1(r pred r gt )

[0025] L t =SmoothL1(t pred , t gt )

[0026] L P =||P pred -P gt ||2

[0027]

[0028] L r L t ,L p Let represent the errors of rotation, translation, and point set transformation, respectively, where n is the number of iterations, and α, β, and γ are the weights controlling different error terms.

[0029] Step 8) Complete the online calibration network training on server computing resources;

[0030] Step 9) Based on the trained camera-lidar online self-calibration network, realize the online real-time calibration of the relative extrinsic parameters of the camera and the lidar. Attached Figure Description

[0031] To more clearly illustrate the technical details of the present invention, the main drawings used in the embodiments will be provided below and briefly described.

[0032] Figure 1 This is a flowchart of the program of the present invention;

[0033] Figure 2 This is a schematic diagram of the network structure of the camera-lidar online self-calibration network of the present invention.

[0034] Figure 3 This is a schematic diagram of the network structure of the update module in the camera-LiDAR online self-calibration network of the present invention. Detailed Implementation

[0035] This embodiment is implemented based on the technical solution of the present invention, and the described embodiment is only a part of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. The following references... Figures 1-3 The present invention will be described in detail with reference to specific embodiments and views.

[0036] A camera-lidar online self-calibration method and system based on a deep neural network model includes:

[0037] Establish a camera-LiDAR sensor data acquisition platform. Acquire synchronous data through hardware triggering; employ motion compensation algorithms to eliminate motion distortion; the established platform should ensure a large consensus range area between the camera and LiDAR to guarantee sufficient usable data.

[0038] The camera's internal parameters, including the projection matrix and distortion coefficients, are determined using static calibration; the initial relative extrinsic parameter matrix between devices is obtained.

[0039] Based on the initial extrinsic parameters and camera intrinsic parameters obtained from the above steps, the lidar point cloud data is transformed into a coordinate system and projected to obtain a depth projection map in the camera pixel coordinate system.

[0040] The decimal part of the UV coordinates of the image is retained during the pixelation of the point cloud data. The RGB image data is interpolated by the quantization compensation module. In this embodiment, bilinear interpolation is used, and a mirror copying method is used to supplement the edge positions.

[0041] Depth information from RGB images is obtained through a depth estimation network model and used as the fourth dimension of the RGB image data for feature extraction. This step aims to explicitly acquire the three-dimensional information of the image, ensuring consistency with the point cloud depth projection values, and fully exploring the environmental geometric features between the two sets of data. By calculating the consistency of these geometric features, the regression process for predicting extrinsic parameters is guided. The selected depth estimation network can be either a monocular or binocular depth estimation network, flexibly chosen according to the specific application scenario. The selected network has been proven to have good accuracy and generalization ability.

[0042] The preprocessed data is input into the camera-LiDAR online self-calibration network, and then undergoes steps such as feature extraction, data fusion, and extrinsic parameter regression. Figure 2 as well as Figure 3 As shown, after multiple iterations, the final predicted extrinsic parameters are output, including a rotation quaternion representing the rotational pose and a translation vector representing the spatial pose. Using quaternions to represent the rotational pose relationship between sensors is an efficient and stable method, particularly suitable for pose representation and computation in three-dimensional space. Quaternions not only accurately describe rotation but also avoid the gimbal lock problem common in Euler angle representation. Furthermore, they occupy less storage space than rotation matrices, facilitating computational operations during extrinsic parameter updates.

[0043] This example uses 4 iterations of training. The latent variable h in the first iteration... t Initialized as the feature correlation between the RGB image and the point cloud projection map, the Update Block module continuously improves and updates the latent variable features based on the feedback from each round of prediction extrinsic parameters after multiple iterations, resulting in a significant increase in the accuracy of the prediction extrinsic parameters. Combining accuracy improvement with efficiency and network inference speed, four iterations are a reasonable setting.

[0044] For network training, three sets of supervised loss terms were set up to measure the convergence effect of the network model on rotation error, translation error, and overall transformation error. In this embodiment, the set of points to be transformed is uniformly distributed within a cubic space with a single-axis distance of 20m, totaling 1000 sets of points. The accuracy of the predicted extrinsic parameters is measured by calculating the average Euclidean distance between the corresponding points after the transformation of the true extrinsic parameters and the predicted extrinsic parameters; the hyperparameters mentioned in step 7) use the following values: n = 5, α = 0.6, β = 0.3, γ = 0.8.

[0045] For network training, this example uses the KITTI dataset, commonly used in the field of autonomous driving. Random rotation and translation noise is added to simulate initial extrinsic parameter distortion while simultaneously obtaining supervised ground truth. In practical applications, fine-tuning data can be added based on the specific usage scenario to improve prediction accuracy. In this example, considering that the consensus region between sensors significantly impacts the utilization rate of point cloud data during LiDAR point cloud projection, and that large rotation and translation errors may lead to overly sparse point cloud projections, hindering feature extraction and deviating from practical application requirements, the maximum offset angle for each axis is limited to ±20°, and the maximum offset in each axis direction is ±0.5m. Rotation and translation errors are randomly sampled for network training. The resolution of the input image and point cloud projection map were adjusted to 1216×352, the initial learning rate was set to 2e-4, and the number of iterations (epochs) was 40. The performance of the joint calibration system was evaluated from the perspectives of the absolute value of the rotational angle error around the three axes, the absolute value of the translational error along the three axes, the rotational angle error around the rotation axis, and the spatial displacement error. Under actual testing, the average angle error around the axis was 0.11°, and the average translational error was 0.005m, which meets the practical application requirements of most multi-sensor data fusion scenarios. The embodiment described above achieves the requirements of high precision, real-time performance, and high reliability, and has good calibration results.

[0046] After the designed network is trained on the server side, it is deployed for practical application to build an online joint extrinsic parameter calibration system, which can acquire high-precision camera-lidar relative extrinsic parameters in real time.

Claims

1. A deep learning-based camera-radar extrinsic parameter self-calibration method and system, characterized in that, The application relates to a camera-lidar relative pose self-calibration method based on deep learning. The application comprises the following steps: An initial relative pose is obtained, and a rotation matrix is converted into a quaternion representation: The selected depth estimation model comprises a monocular depth estimation network or a binocular depth estimation network, which can be flexibly arranged according to specific use scenarios; the depth estimation model does not participate in the gradient update of the whole network; by explicitly specifying the depth information as an important direction of feature extraction, the environmental geometric consistency in the two kinds of data can be fully calculated. ​ ​ The concept of quantization error in the point cloud projection process is introduced, a spatial quantization compensation module is added, the decimal part (u p , u q ) of the uv pixel coordinates of the pixelized point cloud data is reserved, the RGB image data is subjected to bilinear interpolation processing, the quantization loss is maximally compensated, and the utilization rate of the point cloud data is improved; the interpolation method is as follows: P' = (1 - u p )P q + u 11 (1 - u p )P q + (1 - u 21 )u p P q + u 12 u p P q + u 22 wherein P' represents the value of the corresponding pixel position after interpolation, P 11 , P 12 , P 21 , P 22 respectively represent the pixel values of the four adjacent pixel points below the current position, and the mirror image value is used to fill the edge position of the image. ​ 2. The camera-radar extrinsic parameter self-calibration method and system based on deep learning according to claim 1, characterized in that, ​ where the unit vector u = (u x , u y , u z ) is the rotation axis and θ is the rotation angle about the axis; the homogeneous coordinates P L = (0, x, y, z) of each point in the radar coordinate system can be converted to the point coordinates in the camera coordinate system by the following formula: P C = qP L q * +t q * The conjugate quaternion is denoted as q, and t is denoted as a translation vector in the extrinsic matrix.

3. The method and system for camera-radar extrinsic self-calibration based on deep learning according to claim 1, characterized in that ​ 4. The camera-radar extrinsic parameter self-calibration method and system based on deep learning according to claim 1, characterized in that, In the extrinsic parameter regression module, the latent variable state is flattened into a one-dimensional vector after passing through the aggregated convolutional layer, and then two sets of three fully connected layers are used to regress the rotation quaternion vector and the translation vector increment Δv r , Δv t represents the increment matrix ΔT, and the new extrinsic parameter matrix T is obtained by combining the initial extrinsic parameter matrix pred = ΔTT init , where T init is the initially given extrinsic parameter matrix; the features The update module is composed of a gated recurrent unit (GRU), which updates the hidden variable state through autonomous learning of the network; the multi-round results of the extrinsic parameter regression are all used as loss weight supervision items to participate in error loss calculation, and the SmoothL1 Loss of the rotation quaternion vector and the translation vector is calculated respectively; in order to more accurately supervise the global matching effect, a group of uniformly distributed points in the 3D space is designed, the spatial distance error between the predicted extrinsic parameter and the real extrinsic parameter after transformation is calculated, and a decay coefficient is set to configure the weight of the loss function of each iteration; The supervision loss is designed as: L r = SmoothL1(r pred , r gt ) L t = SmoothL1(t pred , t gt ) L P =||P pred -P gt ||2 L r ,L t ,L p represent the rotation term, the translation term and the error of point set transformation, respectively, n is the iteration number, and a, b, g are the weights of different error terms; r pred , t pred represent the rotation quaternion and the translation vector of the predicted external parameter, respectively; r gt , t gt represent the rotation quaternion and the translation vector of the real external parameter, respectively; P pred , P gt represent the new point set of the point set P after the coordinate transformation by the predicted external parameter and the real external parameter, respectively.

Images