Laser radar and camera combined calibration method based on multi-feature three-dimensional target

Abstract

The invention discloses a laser radar and camera combined calibration method based on a multi-feature three-dimensional target, and belongs to the technical field of laser radars. The method comprises the following steps: collecting multi-feature three-dimensional target data of a laser radar and a camera; performing feature extraction on the acquired multi-feature three-dimensional target data of the laser radar and the camera to obtain two-dimensional sub-pixel coordinates of key points under a camera image coordinate system and three-dimensional space coordinates of the key points under a three-dimensional sensor coordinate system; and performing joint external parameter calibration of the laser radar and the camera based on the obtained two-dimensional sub-pixel coordinates and the three-dimensional space coordinates. According to the invention, by designing the target in a specific form, the problem of low joint calibration precision of the multi-modal sensor caused by mismatching of the working range of the laser radar and the working range of the camera is solved, and effective and efficient correspondence of data features acquired by the laser radar and the camera is realized; and thus, high-precision joint calibration between the laser radar and the camera is completed.

Description

A joint calibration method for lidar and camera based on multi-feature stereo targets Technical Field

[0001] This invention belongs to the field of lidar technology, specifically relating to a joint calibration method for lidar and camera based on multi-feature stereo targets. Background Technology

[0002] Joint calibration of LiDAR and cameras aims to spatially align the coordinate systems of LiDAR and cameras using specific algorithms and methods, thereby achieving effective data fusion between the two sensors and providing a data foundation for multimodal measurements. In many fields such as autonomous driving, robot navigation, and 3D reconstruction, LiDAR provides accurate depth information, while cameras provide rich texture and color information; joint calibration of the two is a prerequisite for data fusion.

[0003] Joint calibration of lidar and camera achieves extrinsic parameter calibration of the lidar coordinate system and the camera coordinate system through key features. Target-based joint calibration methods for lidar and camera are relatively mature and widely used.

[0004] Currently, joint calibration of LiDAR and cameras is mainly divided into scene-based targetless joint calibration and target-based joint calibration.

[0005] Targetless calibration methods extract corresponding features from natural images of any scene acquired by two types of sensors using algorithms and calculate structural parameters based on this correspondence. The biggest drawback of this type of method is that its calibration accuracy is directly determined by the quality of environmental features; if the environmental features are not ideal, the calibration results will be very poor, exhibiting poor robustness. Therefore, this type of method is still not used for joint calibration in most practical engineering applications today.

[0006] Target-based calibration methods achieve a correspondence between specific features of the visual sensor and the 3D sensor by designing specific target features, thereby solving for the structural parameters between the sensors. Common targets include checkerboard targets, disc targets, and targets with special geometric shapes. These calibration methods typically have relatively stable calibration accuracy; however, due to differences in the responsiveness of 2D and 3D sensors to features and differences in working distance, this type of calibration is significantly affected by the working space. In the calibration process using 3D targets composed of multiple planes, the edge effect of the LiDAR points is easily affected, and this error severely impacts the joint calibration accuracy. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method for joint calibration of lidar and camera based on a multi-feature stereo target. This method uses the target to perform joint calibration of lidar and camera, thereby improving the accuracy of joint calibration of lidar and camera.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A joint calibration method for lidar and camera based on multi-feature stereo targets includes the following steps:

[0010] Collect multi-feature stereo target data from lidar and cameras;

[0011] Feature extraction is performed on the multi-feature stereo target data collected by lidar and camera to obtain the two-dimensional sub-pixel coordinates of key points in the camera image coordinate system and the three-dimensional spatial coordinates of key points in the three-dimensional sensor coordinate system.

[0012] Based on the obtained two-dimensional sub-pixel coordinates and three-dimensional spatial coordinates, joint extrinsic parameter calibration of LiDAR and camera is performed.

[0013] The beneficial effects of this invention are as follows:

[0014] By designing a specific type of target, the problem of low joint calibration accuracy of multimodal sensors caused by the mismatch between the working range of lidar and camera was solved. This enabled the effective and efficient correspondence of data features collected by lidar and camera, thereby completing high-precision joint calibration between lidar and camera. Attached Figure Description

[0015] Figure 1 is a schematic diagram of the multi-feature three-dimensional target of the present invention;

[0016] Figure 2 is a flowchart of a joint calibration method for lidar and camera based on multi-feature stereo target according to the present invention;

[0017] Figure 3 is a schematic diagram of obtaining camera feature points. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Figure 1 shows a schematic diagram of the multi-feature 3D target used in this invention. The target consists of a checkerboard-patterned plane target with known corner distances and a hemisphere with a known radius. The virtual point at the center of the hemisphere is strictly located on the checkerboard-patterned target plane. Since the measurement scene and measurement space are not entirely the same, the size of the checkerboard and the hemisphere can be adjusted appropriately according to the actual situation. The target provided in the embodiment, as shown in Figure 3, is a checkerboard target with 5 pairs of feature points, totaling 10 checkerboard feature points. The center of the hemisphere is located at the intersection of the lines connecting the 5 pairs of feature points. The overall target features consist of checkerboard corner points, the checkerboard plane, the surface of the hemisphere, and the virtual point at the center of the hemisphere. The checkerboard corner points provide accurate image features for the camera, the hemisphere surface and the checkerboard plane provide multi-constrained 3D geometric features for the lidar, and the virtual point at the center of the hemisphere is a common feature between the two sensors. The overall target material is a diffuse reflective material, and the target structure is machined with high precision to ensure sufficient accuracy.

[0020] As shown in Figure 2, based on the above design, the present invention provides a joint calibration method for lidar and camera based on multi-feature stereo targets, which specifically includes the following steps:

[0021] Step 1: Acquire multi-feature stereo target data from the LiDAR and camera under the same trigger condition. First, install the camera and LiDAR as a rigid body structure according to the measurement requirements, ensuring their relative positions remain fixed and do not shift. Then, place the multi-feature stereo target in the measurement space in front of the camera and LiDAR, ensuring the target is within the common measurement range of both the LiDAR and camera. Acquire LiDAR point cloud and camera images under the same trigger condition. After acquiring one frame, move the target and acquire data again. Repeat the free movement of the target at least six times to ensure the target remains within the working range at all times. To improve calibration robustness and accuracy, the acquisition positions can be increased by more than 10 times until data acquisition from both the LiDAR and camera is complete.

[0022] Step 2: Extract features from the collected lidar and camera data.

[0023] First, feature extraction is performed on the camera data. The target image data acquired by the camera is processed, and a corner extraction method is used to extract the corner points of the checkerboard pattern in the target image. This obtains the pixel coordinates of all checkerboard corner points in each image acquired by the camera. Then, the sub-pixel coordinates of the corner points are calculated using the Taylor formula. After obtaining the coordinates of each corner point, the intersection point can be obtained by connecting the corresponding points as shown in Figure 2. The intersection point is recorded as the sub-pixel coordinates of the virtual point of the hemisphere's center on the checkerboard target plane.

[0024] Specifically, taking the target shown in Figure 1 as an example, its main view is shown in Figure 3. The image coordinates of precise key points in the plane can be obtained in the following way:

[0025] 1) The target design in the example uses 5 pairs of corresponding key points, totaling 10 checkerboard key points, among which the key points... With key points , For the corresponding key points, based on the corner point extraction method, 5 pairs, totaling 10 chessboard key points, can be obtained with high-precision sub-pixel coordinates after distortion correction, denoted as . For chessboard targets with other grid points adjusted according to the measurement scenario, the corresponding points should be selected as the chessboard corner points where the line connecting the corresponding points passes through the center of the hemisphere, and these are recorded as key points. With key points , There are 2n key points on the chessboard, denoted as . .

[0026] 2) The two lines defined in 1) that pass through a pair of checkerboard corner points at the center of the sphere are considered as a pair of corresponding points. Calculate the equation of the straight line connecting the sub-pixel coordinates of the corresponding points. There are 5 examples in total, denoted as... For the chessboard target with other grid points adjusted according to the measurement scenario, there are a total of n lines, denoted as... ;

[0027] 3) Calculate the coordinates of the intersection point between each pair of straight lines. There are 10 examples in total. For chessboard targets with other grid points adjusted according to the measurement scenario, the total number of intersections is... indivual;

[0028] 4) Calculate the average of the intersection point coordinates to obtain the image coordinates of the virtual point of the precise center of the hemisphere in the plane. For the embodiments, coordinates The calculation is as follows:

[0029] .

[0030] For the key point extraction results of the chessboard target with a different number of grid points after adjustment based on the measurement scenario, the coordinates The calculation is as follows:

[0031]

[0032] At this point, all image features of the camera data have been obtained.

[0033] Next, feature extraction is performed on the LiDAR data. The features of the LiDAR data include checkerboard planar point cloud features, hemispherical surface point cloud features, and virtual point features at the center of the hemispherical surface. For the extraction of hemispherical point cloud features, firstly, the hemispherical portion of the LiDAR point cloud in each frame is segmented, and then the hemispherical equation is fitted based on the hemispherical point cloud:

[0034] ,

[0035] In the formula, This is expressed as the loss function for fitting a hemispherical point cloud. This represents the three-dimensional coordinates of the lidar point cloud on the hemispherical surface. For point cloud counting variables, This represents the total number of lidar points used to fit the hemisphere. This represents the actual three-dimensional coordinate position of the center point of the target hemisphere. This represents the actual radius of the target hemisphere. This represents the parameter coefficients after the hemispherical part is transformed into a quadratic form, and the subscript b indicates the parameters of the fitted hemispherical part.

[0036] right conduct The partial derivatives, after simplification, yield:

[0037] ,

[0038] in, The intermediate parameter is represented by the following formula:

[0039] ,

[0040] Therefore, the approximate three-dimensional coordinates of the virtual point at the center of the hemisphere can be obtained, that is... .

[0041] Next, the checkerboard plane portion of the laser point cloud in each frame is segmented, and a plane equation is fitted based on the planar point cloud:

[0042] ,

[0043] In the formula, This is expressed as the loss function for fitting a checkerboard-patterned point cloud. This represents the three-dimensional coordinates of the lidar point cloud on the checkerboard plane. For point cloud counting variables, This represents the total number of lidar points used to fit the checkerboard plane. This represents the equation used to fit the checkerboard plane. The four coefficient parameters.

[0044] Substitute the points of the segmented laser point cloud checkerboard plane into... By performing least squares optimization, the coefficient parameters of the chessboard plane equation can be obtained. Approximate value.

[0045] Finally, using the equations of the virtual center point of the hemisphere calculated by the lidar and the checkerboard plane as global constraints, high-precision calculation of the virtual center point of the hemisphere is achieved. The objective function is established as follows:

[0046] ,

[0047] In the formula, This represents three-dimensional points on the hemispherical portion of the lidar. This represents the three-dimensional points of the lidar on the checkerboard plane. This represents the optimization variable for the three-dimensional coordinates of the virtual point at the center of the hemisphere. Describe the optimization variables of the chessboard plane equation. For Lagrange multipliers, The radius of the hemisphere is known.

[0048] After optimization, the precise global sphere center coordinates are obtained. It was uniquely identified as a precise spatial key point in the three-dimensional sensor coordinate system.

[0049] Based on the extracted sub-pixel coordinates of the camera checkerboard corner points, the camera intrinsic parameters can be calculated using Zhang Zhengyou calibration. This matrix contains information such as camera focal length and principal point.

[0050] Step 3: Joint extrinsic parameter calibration of LiDAR and camera.

[0051] The above method can be used to obtain the two-dimensional pixel coordinates of key points in the camera image coordinate system. and the three-dimensional spatial coordinates of key points in the three-dimensional sensor coordinate system The corresponding homogeneous coordinates are respectively and Based on the spatial coordinate system transformation relationship, the following equations can be established:

[0052] ,

[0053] in, The camera's intrinsic parameter matrix is ​​calculated in step 2. This is the rotation matrix between the camera and the 3D sensor. Let s be the translation vector between the camera and the 3D sensor, and s be the scaling factor.

[0054] because It has three degrees of freedom. Since it also has three degrees of freedom, a minimum of 6 points are needed to solve for the rotation matrix. Translation vector The target was moved multiple times during the actual calibration process to obtain more accurate calibration results.

[0055] By implementing the above steps, the problem of low joint calibration accuracy of multimodal sensors caused by the mismatch in the working range of the lidar and the camera can be solved. This enables effective and efficient correspondence of the data features acquired by the lidar and the camera, thereby achieving high-precision joint calibration between the lidar and the camera. Those skilled in the art can use this invention in tasks such as large-scale, wide-range 3D measurement.

[0056] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for joint calibration of lidar and camera based on multi-feature stereo targets, characterized in that, The process includes the following steps: designing a multi-feature stereo target and collecting multi-feature stereo target data from LiDAR and camera; extracting features from the collected multi-feature stereo target data from LiDAR and camera to obtain the two-dimensional sub-pixel coordinates of key points in the camera image coordinate system and the three-dimensional spatial coordinates of key points in the three-dimensional sensor coordinate system; and performing joint extrinsic parameter calibration of LiDAR and camera based on the obtained two-dimensional sub-pixel coordinates and three-dimensional spatial coordinates.

2. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 1, characterized in that, The process of acquiring multi-feature stereo target data from the lidar and camera includes: installing the camera and lidar as a rigid structure according to measurement requirements; placing the multi-feature stereo target in the measurement space in front of the camera and lidar, and acquiring lidar point clouds and camera images under the same triggering conditions; repeatedly moving the target freely to complete the data acquisition from the lidar and camera.

3. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 2, characterized in that, The multi-feature stereo target is within the common measurement range of the lidar and the camera, and the target material is a diffuse reflective material.

4. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 1, characterized in that, The multi-feature 3D target includes a checkerboard plane target with known corner distances and a hemisphere with a known radius, wherein the virtual point at the center of the hemisphere is strictly located on the checkerboard target plane.

5. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 4, characterized in that, The overall target features consist of checkerboard corner points, checkerboard plane, hemispherical surface, and virtual points at the center of the hemispherical. The checkerboard corner points provide image features for the camera, the checkerboard plane and hemispherical surface provide multi-constrained three-dimensional geometric features for the lidar, and the virtual points at the center of the hemispherical provide common features between the two sensors.

6. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 1, characterized in that, The two-dimensional sub-pixel coordinates of the key points in the camera image coordinate system are obtained through the following steps: marking a total of 2n checkerboard corner points on the target, selecting the checkerboard corner points whose connecting lines pass through the center of the hemisphere as the corresponding key points; connecting a pair of corresponding key points with straight lines, and calculating the straight line equation between the corresponding sub-pixel coordinate points; calculating the coordinates of the intersection point between each pair of straight line equations; and averaging the coordinates of multiple intersection points to obtain the image coordinates of the virtual point of the precise center of the hemisphere in the plane, which is used as the two-dimensional sub-pixel coordinates of the key points in the camera image coordinate system.

7. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 1, characterized in that, The three-dimensional spatial coordinates of the key points in the three-dimensional sensor coordinate system are obtained through the following steps: For each frame, the hemispherical part in the laser point cloud is segmented, and the hemispherical equation is fitted according to the hemispherical point cloud to obtain the approximate three-dimensional coordinates of the virtual point at the center of the hemispherical body; For each frame, the checkerboard plane part in the laser points is segmented, and the checkerboard plane equation is fitted according to the plane point cloud. Using the approximate three-dimensional coordinates of the virtual point at the center of the hemisphere and the equation of the checkerboard plane as global constraints, the precise three-dimensional coordinates of the virtual point at the center of the hemisphere are calculated and used as the three-dimensional spatial coordinates of the key points in the three-dimensional sensor coordinate system.

8. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 7, characterized in that, The process of fitting the hemispherical equation based on the hemispherical point cloud includes: In the formula, This is expressed as the loss function for fitting a hemispherical point cloud. This represents the three-dimensional coordinates of the lidar point cloud on the hemispherical surface. For point cloud counting variables, This represents the total number of lidar points used to fit the hemisphere. This indicates the actual three-dimensional coordinate position of the center point of the target hemisphere. This represents the actual radius of the target hemisphere. This represents the parameter coefficients after the standard form of the hemispherical part is transformed into a quadratic form, and the subscript b indicates that it is the parameter of the fitted hemispherical part.

9. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 7, characterized in that, The process of fitting the checkerboard plane equation based on the planar point cloud includes: In the formula, This is expressed as the loss function for fitting a checkerboard-patterned point cloud. This represents the three-dimensional coordinates of the lidar point cloud on the checkerboard plane. For point cloud counting variables, This represents the total number of lidar points used to fit the checkerboard plane. This represents the equation used to fit the checkerboard plane. The four coefficient parameters.

10. The method for joint calibration of lidar and camera based on multi-feature stereo targets according to claim 1, characterized in that, The joint extrinsic parameter calibration of the LiDAR and camera based on the obtained two-dimensional sub-pixel coordinates and three-dimensional spatial coordinates includes: the two-dimensional sub-pixel coordinates of key points in the camera image coordinate system. and the three-dimensional spatial coordinates of key points in the three-dimensional sensor coordinate system The corresponding homogeneous coordinates are respectively and Based on the spatial coordinate system transformation relationship, the following equations are established: ,in, This is the intrinsic parameter matrix of the camera. This is the rotation matrix between the camera and the 3D sensor. Let s be the translation vector between the camera and the 3D sensor, and s be the scaling factor.

Images