Calibration method for mutual constraints of laser radar, camera and inertial sensor

Abstract

The application discloses a kind of laser radar, camera and the external parameter calibration method of mutual restraint of inertial sensor, it includes the following steps: making checkerboard radar reflector plate, and form calibration system;Static calibration laser radar and camera, obtain the conversion external parameter of laser radar to camera;Respectively from visual angle and laser radar angle to dynamic calibration for inertial sensor;Residual error optimization model is constructed, and the conversion external parameter between camera and inertial sensor is calculated by least square method to residual error with nonlinear optimization;Based on the mutual conversion relationship between external parameter, the conversion external parameter between laser radar and inertial sensor is obtained, and external parameter calibration is completed.This method combines the method that static calibration laser radar and visual camera, dynamic calibration laser radar and IMU, dynamic calibration visual camera and IMU mutually, so that each sensor forms mutually restrained triangular structure, realizes the time-space consistency and stability between high-precision multi-sensor.

Description

Technical Field

[0001] This invention relates to the field of sensor fusion, and specifically to a method for calibrating extrinsic parameters of a lidar, camera, and inertial sensor that mutually constrain each other. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology refers to a technique where a sensor-equipped subject estimates its own pose and simultaneously builds an environmental model using sensor data and its own motion, even without information about the surrounding environment. Odometry is the front end of SLAM technology; it estimates changes in the subject's position and attitude using data from different sensors.

[0003] Currently, there are many successful odometry frameworks using single measurement sensors, mostly based on LiDAR or vision cameras. However, with the increasing demands and requirements for intelligent robots, SLAM technology also needs to cope with more complex and challenging environments. Therefore, in recent years, sensors such as IMU (Inertial Measurement Unit) and GPS (Global Positioning System) have been introduced for assistance. Multimodal fusion odometry systems combining multiple sensors can integrate the working characteristics and ranges of each sensor, leveraging their strengths and mitigating their weaknesses. They can still work robustly even when some sensors degrade, demonstrating stronger and wider application potential.

[0004] While multi-sensor fusion odometry offers significant advantages, its application in SLAM algorithms is also more complex. To fully trust and utilize the data measured by each sensor, multi-modal fusion odometry requires prior, deterministic coordinate axis transformations (i.e., extrinsic parameters) between sensors. This allows information from different sensors to be converted to a common physical reference frame for calculation. Most existing extrinsic parameter calibration methods are only applicable to static systems and do not include the use of an IMU. The few methods that include IMU calibration are limited to LIO or VIO systems, lacking working examples of combining all three sensors, thus failing to form a fully constrained overall system. Summary of the Invention

[0005] In view of the above-mentioned shortcomings in the prior art, the present invention provides an external parameter calibration method for mutual constraints of lidar, camera and inertial sensor, which solves the problems of poor accuracy and robustness of the existing external parameter calibration methods for lidar, camera and inertial sensor.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A method for calibrating the extrinsic parameters of a lidar, camera, and inertial sensor that are mutually constrained is provided, comprising the following steps:

[0008] S1. Fabricate a checkerboard Gradient reflector; fix the relative positions of the lidar, camera, and inertial sensor to form a calibration system; the checkerboard Gradient reflector has a black and white square checkerboard pattern, and the radar reflectivity of the black and white parts of the pattern is different;

[0009] S2. Based on the checkerboard Greda reflector, statically calibrate the lidar and camera to obtain the conversion extrinsic parameters from the lidar to the camera;

[0010] S3, the mobile calibration system, dynamically calibrates the inertial sensor from both visual and lidar perspectives, obtaining the initial values ​​of external parameters between the camera and the inertial sensor, the initial values ​​of external parameters between the lidar and the inertial sensor, the camera pose over a period of time in the world coordinate system, and the lidar pose over the same period of time in the world coordinate system.

[0011] S4. Based on the data obtained in step S3, construct a residual optimization model, and calculate the conversion extrinsic parameters between the camera and the inertial sensor by performing nonlinear optimization of the residuals using the least squares method.

[0012] S5. Based on the conversion extrinsic parameters between the camera and the inertial sensor and the conversion extrinsic parameters from the lidar to the camera, the conversion extrinsic parameters between the lidar and the inertial sensor are obtained from the mutual conversion relationship between the camera-inertial sensor-lidar extrinsic parameters, and the extrinsic parameter calibration is completed.

[0013] Furthermore, the specific method of step S2 includes the following sub-steps:

[0014] S2-1. Obtain the camera's intrinsic parameters;

[0015] S2-2, a fixed chessboard Greda reflector, which enables the lidar and camera to identify points on the calibration board, thereby obtaining 3D radar point cloud data and 2D image data;

[0016] S2-3. Project the 3D radar point cloud data onto 2D to obtain the projected data;

[0017] S2-4. Align the projected data with the corresponding points in the 2D image data to obtain the conversion extrinsic parameters from the LiDAR to the camera.

[0018] Furthermore, the specific method for dynamically calibrating the inertial sensor from a visual perspective in step S3 includes the following sub-steps:

[0019] A1. Mobile calibration system, which acquires inertial sensor data and camera data;

[0020] A2. Align the inertial sensor data with the camera data;

[0021] A3. Integrate the inertial sensor data between the camera data of frame k and the camera data of frame k+1 to obtain the relative position, velocity and rotation changes of the calibration system between the two frames;

[0022] A4. Using the change in the relative position between the inertial sensor and the camera as the initial value, the reprojection error of the feature points obtained from the image in each frame is taken as the optimization object. Nonlinear optimization is performed on the optimization object to obtain the time points t = [t] represented by different camera frames within the acquisition time. v1 , t v2 , ..., t vK At that time, the camera pose in the world coordinate system Initial values ​​of extrinsic parameters between the camera and the inertial sensor

[0023] Furthermore, the specific method for dynamically calibrating the inertial sensor from the lidar in step S3 includes the following sub-steps:

[0024] B1. Mobile calibration system, which acquires inertial sensor data and lidar data;

[0025] B2. Align the inertial sensor data with the lidar data;

[0026] B3. Calculate the motion estimation values ​​of the LiDAR from frame i to frame i+1 from the LiDAR point cloud using NDT scanning matching. And obtain the lidar pose in the world coordinate system at different time points within a certain period of time. and its corresponding time point {t L1 , t L2 ....t LN};

[0027] B4. According to the formula:

[0028]

[0029] Obtain initial values ​​of extrinsic parameters between the lidar and the inertial sensor. in These are the measurement values ​​corresponding to the inertial sensor and the LiDAR data in the i-th frame; This represents the measurement value corresponding to the inertial sensor and the LiDAR data in the (i+1)th frame.

[0030] Furthermore, the specific method of step S4 includes the following sub-steps:

[0031] S4-1, From Get time point {t L1 , t L2 ....tLN pose data at} That is, from Obtaining and Time-aligned data

[0032] S4-2, According to the formula:

[0033]

[0034] Establish a residual optimization model and use the least squares method. The result obtained when solving for the residual ΔT The value serves as an extrinsic parameter for the conversion between the camera and the inertial sensor; where The conversion extrinsic parameters from lidar to camera;

[0035] Furthermore, the specific method for step S4-1 is as follows:

[0036] For time point t Ln ,like If the corresponding data exists, select that data directly; otherwise, use the data provided. Interpolation method to obtain time point t Ln The corresponding data; where t Ln ∈{t L1 , t L2 ....t LN}

[0037] Furthermore, the specific method for step S5 is as follows:

[0038] The product of the conversion extrinsic parameters between the camera and the inertial sensor and the conversion extrinsic parameters from the lidar to the camera is used as the conversion extrinsic parameter between the lidar and the inertial sensor to complete the extrinsic parameter calibration.

[0039] The beneficial effects of this invention are as follows: This method combines static calibration of the lidar and vision camera, dynamic calibration of the lidar and IMU, and dynamic calibration of the vision camera and IMU to form a triangular structure in which each sensor is mutually constrained. Furthermore, it can be extended to a multi-factor constrained extrinsic parameter space that is connected with more sensors, thereby achieving high-precision spatiotemporal consistency and stability among multiple sensors. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method.

[0041] Figure 2 This is a schematic diagram of the actual chessboard Grada reflector.

[0042] Figure 3 This is a schematic diagram of the actual calibration system. Detailed Implementation

[0043] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0044] like Figure 1 , Figure 2 and Figure 3 As shown, the external parameter calibration method for mutual constraints among the lidar, camera, and inertial sensor includes the following steps:

[0045] S1. Fabricate a checkerboard Gradient reflector; fix the relative positions of the lidar, camera, and inertial sensor to form a calibration system; the checkerboard Gradient reflector has a black and white square checkerboard pattern, and the radar reflectivity of the black and white parts of the pattern is different;

[0046] S2. Based on the checkerboard Greda reflector, statically calibrate the lidar and camera to obtain the conversion extrinsic parameters from the lidar to the camera;

[0047] S3, the mobile calibration system, dynamically calibrates the inertial sensor from both visual and lidar perspectives, obtaining the initial values ​​of external parameters between the camera and the inertial sensor, the initial values ​​of external parameters between the lidar and the inertial sensor, the camera pose over a period of time in the world coordinate system, and the lidar pose over the same period of time in the world coordinate system.

[0048] S4. Based on the data obtained in step S3, construct a residual optimization model, and calculate the conversion extrinsic parameters between the camera and the inertial sensor by performing nonlinear optimization of the residuals using the least squares method.

[0049] S5. Based on the conversion extrinsic parameters between the camera and the inertial sensor and the conversion extrinsic parameters from the lidar to the camera, the conversion extrinsic parameters between the lidar and the inertial sensor are obtained from the mutual conversion relationship between the camera-inertial sensor-lidar extrinsic parameters, and the extrinsic parameter calibration is completed.

[0050] The specific method of step S2 includes the following sub-steps:

[0051] S2-1. Obtain the camera's intrinsic parameters;

[0052] S2-2, a fixed chessboard Greda reflector, which enables the lidar and camera to identify points on the calibration board, thereby obtaining 3D radar point cloud data and 2D image data;

[0053] S2-3. Project the 3D radar point cloud data onto 2D to obtain the projected data;

[0054] S2-4. Align the projected data with the corresponding points in the 2D image data to obtain the conversion extrinsic parameters from the LiDAR to the camera.

[0055] This represents a 3D rotation matrix, indicating the attitude rotation from the lidar coordinate system to the camera coordinate system; `0` is a translation vector, representing the translation relationship from the lidar coordinate system to the origin of the camera coordinate system; `0` is a 1×3 row vector with all elements being 0; `0` represents the coordinates of a point on the chessboard Gregorian reflector in the camera coordinate system. Coordinates in the lidar coordinate system The correspondence is as follows:

[0056] The specific method for dynamically calibrating the inertial sensor from a visual perspective in step S3 includes the following sub-steps:

[0057] A1. Mobile calibration system, which acquires inertial sensor data and camera data;

[0058] A2. Align the inertial sensor data with the camera data;

[0059] A3. Integrate the inertial sensor data between the camera data of frame k and the camera data of frame k+1 to obtain the relative position, velocity and rotation changes of the calibration system between the two frames;

[0060] A4. Using the change in the relative position between the inertial sensor and the camera as the initial value, the reprojection error of the feature points obtained from the image in each frame is taken as the optimization object. Nonlinear optimization is performed on the optimization object to obtain the time points t = [t] represented by different camera frames within the acquisition time. v1 , t v2 , ..., t vK At that time, the camera pose in the world coordinate system Initial values ​​of extrinsic parameters between the camera and the inertial sensor

[0061] The specific method for dynamically calibrating the inertial sensor from the lidar in step S3 includes the following sub-steps:

[0062] B1. Mobile calibration system, which acquires inertial sensor data and lidar data;

[0063] B2. Align the inertial sensor data with the lidar data;

[0064] B3. Calculate the motion estimation values ​​of the LiDAR from frame i to frame i+1 from the LiDAR point cloud using NDT scanning matching. And obtain the lidar pose in the world coordinate system at different time points within a certain period of time. and its corresponding time point {t L1 , t L2 ....t LN};

[0065] B4. According to the formula:

[0066]

[0067] Obtain initial values ​​of extrinsic parameters between the lidar and the inertial sensor. in These are the measurement values ​​corresponding to the inertial sensor and the LiDAR data in the i-th frame; This represents the measurement value corresponding to the inertial sensor and the LiDAR data in the (i+1)th frame.

[0068] The specific method of step S4 includes the following sub-steps:

[0069] S4-1, From Get time point {t L1 , t L2 …·t LN pose data at} That is, from Obtaining and Time-aligned data

[0070] S4-2, According to the formula:

[0071]

[0072] Establish a residual optimization model and use the least squares method. The result obtained when solving for the residual ΔT The value serves as an extrinsic parameter for the conversion between the camera and the inertial sensor; where The conversion extrinsic parameters from lidar to camera;

[0073] The specific method for step S4-1 is as follows: For time point t Ln ,like If the corresponding data exists, select that data directly; otherwise, use the data provided. Interpolation method to obtain time point t Ln The corresponding data; where t Ln ∈{t L1 , t L2 ....t LN}

[0074] The specific method for step S5 is as follows: convert the external parameters between the camera and the inertial sensor. Conversion extrinsic parameters from lidar to camera The product is used as an external parameter for conversion between lidar and inertial sensors. Complete the external parameter calibration, that is

[0075] In one embodiment of the present invention, the pairwise calibration accuracy of the three sensors is approximately at the centimeter level, with the dynamic performance being slightly worse than the static performance, and exhibiting significant fluctuations due to the calibration environment. This method, by using camera and radar calibration to guide the mutual calibration of the other two sets of sensors, doubles the calibration accuracy of the IMU (halving the error). Simultaneously, multi-sensor constraints result in higher stability and effectively improve the variance representation of the results.

Claims

1. A method for calibrating extrinsic parameters of a lidar, camera, and inertial sensor that mutually constrain each other, characterized in that, Includes the following steps: S1. Fabricate a checkerboard Gradient reflector; fix the relative positions of the lidar, camera, and inertial sensor to form a calibration system; the checkerboard Gradient reflector has a black and white square checkerboard pattern, and the radar reflectivity of the black and white parts of the pattern is different; S2. Based on the checkerboard Greda reflector, statically calibrate the lidar and camera to obtain the conversion extrinsic parameters from the lidar to the camera; S3, the mobile calibration system, dynamically calibrates the inertial sensor from both visual and lidar perspectives, obtaining the initial values ​​of external parameters between the camera and the inertial sensor, the initial values ​​of external parameters between the lidar and the inertial sensor, the camera pose over a period of time in the world coordinate system, and the lidar pose over the same period of time in the world coordinate system. S4. Based on the data obtained in step S3, construct a residual optimization model, and calculate the conversion extrinsic parameters between the camera and the inertial sensor by performing nonlinear optimization of the residuals using the least squares method. S5. Based on the conversion extrinsic parameters between the camera and the inertial sensor and the conversion extrinsic parameters from the lidar to the camera, the conversion extrinsic parameters between the lidar and the inertial sensor are obtained from the mutual conversion relationship between the camera-inertial sensor-lidar extrinsic parameters, and the extrinsic parameter calibration is completed.

2. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 1, characterized in that, The specific method of step S2 includes the following sub-steps: S2-1. Obtain the camera's intrinsic parameters; S2-2, a fixed chessboard Greda reflector, which enables the lidar and camera to identify points on the calibration board, thereby obtaining 3D radar point cloud data and 2D image data; S2-3. Project the 3D radar point cloud data onto 2D to obtain the projected data; S2-4. Align the projected data with the corresponding points in the 2D image data to obtain the conversion extrinsic parameters from the LiDAR to the camera.

3. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 1, characterized in that, The specific method for dynamically calibrating the inertial sensor from a visual perspective in step S3 includes the following sub-steps: A1. Mobile calibration system, which acquires inertial sensor data and camera data; A2. Align the inertial sensor data with the camera data; A3. Integrate the inertial sensor data between the camera data of frame k and the camera data of frame k+1 to obtain the relative position, velocity and rotation changes of the calibration system between the two frames; A4. Using the change in the relative position between the inertial sensor and the camera as the initial value, the reprojection error of the feature points obtained from the image in each frame is taken as the optimization object. Nonlinear optimization is performed on the optimization object to obtain the time points t = [t] represented by different camera frames within the acquisition time. v1 ,t v2 ,…,t vK At that time, the camera pose in the world coordinate system Initial values ​​of extrinsic parameters between the camera and the inertial sensor 4. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 3, characterized in that, The specific method for dynamically calibrating the inertial sensor from the lidar in step S3 includes the following sub-steps: B1. Mobile calibration system, which acquires inertial sensor data and lidar data; B2. Align the inertial sensor data with the lidar data; B3. Calculate the motion estimation values ​​of the LiDAR from frame i to frame i+1 from the LiDAR point cloud using NDT scanning matching. And obtain the lidar pose in the world coordinate system at different time points within a certain period of time. and its corresponding time point {t L1 ,t L2 ····t LN }; B4. According to the formula: Obtain initial values ​​of extrinsic parameters between the lidar and the inertial sensor. in These are the measurement values ​​corresponding to the inertial sensor and the LiDAR data in the i-th frame; This represents the measurement value corresponding to the inertial sensor and the LiDAR data in the (i+1)th frame.

5. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 4, characterized in that, The specific method of step S4 includes the following sub-steps: S4-1, From Get time point {t L1 ,t L2 ····t LN pose data of the camera at} That is, from Obtaining and Time-aligned data S4-2, According to the formula: Establish a residual optimization model and use the least squares method. The result obtained when solving for the residual ΔT The value serves as an extrinsic parameter for the conversion between the camera and the inertial sensor; where The conversion extrinsic parameters from lidar to camera; 6. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 5, characterized in that, The specific method for step S4-1 is as follows: For time point t Ln ,like If corresponding camera pose data exists, then that camera pose data is used directly; otherwise, it is used through... Interpolation method to obtain time point t Ln The corresponding camera pose data; where t Ln ∈{t L1 ,t L2 ····t LN } 7. The external parameter calibration method for mutual constraints among lidar, camera, and inertial sensor according to claim 5, characterized in that, The specific method for step S5 is as follows: The product of the conversion extrinsic parameters between the camera and the inertial sensor and the conversion extrinsic parameters from the lidar to the camera is used as the conversion extrinsic parameter between the lidar and the inertial sensor to complete the extrinsic parameter calibration.

Images