Camera-LiDAR external parameter calibration method and device based on line and plane feature association
By using the method of line and plane feature correlation in camera and LiDAR system, the problem of low efficiency and accuracy of camera-LiDAR external parameter calibration in the prior art is solved, and more efficient and more accurate external parameter calibration is achieved.
Patent Information
- Application Number
- CN202510304005.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-14
- Publication Date
- 2025-06-13
AI Technical Summary
The existing camera-LiDAR external parameter calibration method requires multiple movement of the checkerboard, resulting in low calibration efficiency and accuracy.
Using a method based on line and plane feature correlation, the rotation matrix and translation vector from LiDAR to the camera are extracted by extracting line features and plane features from the camera and LiDAR coordinate system, and the geometric constraint relationship is established, and the feature correspondence relationship in multiple chessboard positions is used to solve the rotation matrix and translation vector from LiDAR to the camera.
The external parameter calibration accuracy of the camera-LiDAR system is significantly improved, the algorithm's robustness to noise and different chessboard pose changes, and the calibration efficiency is improved.
Smart Images
Figure CN120147425A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of driverless, and specifically refers to a camera-LiDAR extrinsic calibration method and device based on the association of line and plane features. Background Art
[0002] With the rapid development of autonomous driving technology, the multi-sensor fusion scheme has extremely high accuracy in the perception of unknown environments and has gradually become the mainstream in autonomous driving. The accurate calibration of the extrinsic parameters of sensors is the prerequisite for the long-term robust operation of the entire autonomous driving system. Cameras and lidars are the most commonly used perception devices in autonomous driving, and they have good complementarity. Cameras can obtain rich visual information in the environment but cannot obtain the three-dimensional spatial information of objects. The point clouds output by lidars have rich geometric features and depth information but cannot obtain the texture and color information of the object surface. Therefore, the accurate calibration of the extrinsic parameters of cameras and lidars can effectively fuse the camera and lidar information, which plays an extremely important role in the environmental perception of autonomous driving. Currently, the extrinsic calibration of lidars and cameras can generally be divided into target calibration based on calibration boards and non-target calibration based on deep learning and natural scene features. At present, the camera-LiDAR extrinsic calibration method based on checkerboard targets is widely used. The simple and convenient checkerboard calibration object makes the calibration process convenient and flexible. They use one or more checkerboards to calibrate the camera-LiDAR extrinsic parameters. However, these methods only use the planar correspondence relationship to constrain the extrinsic parameter vector to be estimated, and need to move the checkerboard multiple times to determine the extrinsic parameters, resulting in relatively low calibration efficiency and accuracy of the system. Summary of the Invention
[0003] In order to overcome the above problems, the present invention proposes a camera-LiDAR extrinsic calibration method based on the correspondence relationship between lines and planes, effectively solving the problem that the above methods need to move the checkerboard multiple times to determine the extrinsic parameters, resulting in relatively low calibration efficiency and accuracy of the system, and improving the efficiency and accuracy of extrinsic calibration.
[0004] To achieve the above object, a camera-LiDAR extrinsic calibration method based on the association of line and plane features designed by the present invention uses the correspondence relationship between lines and planes to constrain the extrinsic parameter vector to be estimated, including first building a calibration system including a lidar, an industrial camera, and a checkerboard, collecting LiDAR scan data and checkerboard images under multiple poses, and then performing the following steps:
[0005] Step 1), in the camera coordinate system, extract line features and plane features from the checkerboard image, calculate the parameters of the plane features, and generate a back-projected plane;
[0006] Step 2), in the LiDAR coordinate system, extract line features and plane features from the LiDAR scan data, and calculate the normal vector, centroid, and direction of the plane in the LiDAR coordinate system, and remove the noisy laser points on the checkerboard plane;
[0007] Step 3), establish the geometric constraint relationship between the camera coordinate system and the LiDAR coordinate system based on the plane features and line features;
[0008] Step 4), use the feature correspondence relationship under multiple checkerboard poses to establish a cost function and an optimization function to solve the rotation matrix and translation vector from LiDAR to the camera, and finally output the extrinsic calibration result between the camera and the LiDAR.
[0009] Further, in step 1), feature extraction is performed on the collected checkerboard images. First, detect the checkerboard and its boundaries in the image, and calculate the plane features of the i-th pose of the checkerboard in the camera coordinate system and the four boundary line features of its image plane where j = 1, 2, 3, 4; among them, the line features are detected in the image using the LSD algorithm. Given a checkerboard target, 4 boundary lines surrounding the checkerboard can be detected and extracted as line features;
[0010] The plane features of the checkerboard image are detected using the ready-made OpenCV software, and then the parameters of the plane features, including the plane normal vector plane equation parameters are calculated through the homography relationship between the checkerboard and its image, and an inverse projection plane is generated.
[0011] Further, the specific implementation method of generating the inverse projection plane is: use homogeneous coordinate transformation to convert the two-dimensional line segment into a three-dimensional line segment According to the four boundaries, an inverse projection plane can be generated Convert and into parameters and respectively, where is the normal vector of the plane , with a length of 1, and represent the direction and projection point of the boundary respectively, where also has a length of 1.
[0012] Further, in step 2), the checkerboard plane and its four boundaries in the LiDAR coordinate system are represented by and respectively, where the laser points on the plane are represented by ; Boundary The laser points on it are represented by where k = 1,..., K ij ; Given and calculate the normal vector of the plane, the centroid and the direction in the LiDAR coordinate system
[0013] Furthermore, the RANSAC (Random Sample Consensus) algorithm is used to detect the laser points on the checkerboard plane and find the boundaries of each scan line, which are the left and right boundaries of the checkerboard. Then, calculate the direction of the line defined by two consecutive points on the boundary, further divide the left and right boundaries into two parts by finding the maximum direction change, and remove the noise through back-projection;
[0014] The specific implementation method of removing noise through back-projection is as follows:
[0015] First, project all laser points onto the plane, then fit a straight line for each scan line, and project the boundary points onto each scan line. Finally, use the RANSAC algorithm to remove potential outliers in the boundary points.
[0016] Furthermore, the geometric constraint relationships in step 3) include: line feature direction alignment constraint, line feature projection point consistency constraint, plane normal vector alignment constraint, and plane point consistency constraint.
[0017] Furthermore, the construction method of the geometric constraint relationships is as follows:
[0018] Step 301), the line feature direction alignment constraint is:
[0019]
[0020] where is the rotation matrix from LiDAR to camera, and are the line direction vectors in the LiDAR and camera coordinate systems respectively;
[0021] Step 302), the line feature projection point consistency constraint is:
[0022]
[0023] where I is the identity matrix, is the k-th laser point on the boundary on the i-th plane in the LiDAR coordinate system, is a projection point of the line feature in the camera coordinate system, is the translation vector;
[0024] In step 303), the plane normal vector alignment constraint is:
[0025]
[0026] where, and are the plane normal vectors in the LiDAR and camera coordinate systems respectively;
[0027] In step 304), a plane point consistency constraint is constructed. The checkerboard plane points in the LiDAR coordinate system need to satisfy the plane equation in the camera coordinate system after being transformed by the rotation matrix and the translation vector . The constraint formula is:
[0028]
[0029] where, is the normal of the plane , is the m-th point on the i-th plane in the LiDAR coordinate system, t is the translation vector from LiDAR to camera, is the constant term of the i-th plane equation in the camera coordinate system, which is determined by the plane equation .
[0030] Furthermore, in step 4), the rotation matrix from LiDAR to camera is solved by minimizing the following cost function The cost function is described as follows:
[0031]
[0032] where N represents the poses of N checkerboards, is the direction vector of the j-th boundary in the i-th checkerboard pose in the LiDAR coordinate system, and represent the normals of the i-th checkerboard pose plane in the LiDAR and camera coordinate systems respectively.
[0033] Furthermore, there is a closed-form solution to the cost function minimization problem. First, the following definitions are given:
[0034]
[0035] M L is the feature matrix in the LiDAR coordinate system, which contains the plane normal vector of each checkerboard pose and the four line directions to MC The corresponding feature matrix in the camera coordinate system; assume M L (M C ) T The singular value decomposition of is USV T , where S is the singular value matrix, rotate the basis vector U of the LiDAR coordinate system to the basis vector V of the camera coordinate system to achieve the optimal alignment of geometric constraints, and obtain Use the centroid of the plane and the centroid of the line to provide constraints for the estimation of the initial translation vector , define Get:
[0036]
[0037] Among them, I is the identity matrix; stack equations (1)(2)(3)(4)(5) to obtain a linear equation system about , and there is also a closed-form solution, obtain the initial estimate and After that, jointly optimize by minimizing the following optimization function, and the optimization function is described as follows:
[0038]
[0039] Among them, N represents the poses of N checkerboards, is the rotation matrix, is the translation vector, is the m-th laser point on the boundary on the i-th plane in the LiDAR coordinate system, K ij is the total number of laser points on the boundary in the LiDAR coordinate system, represents the projection point of the boundary , and t is the translation vector from LiDAR to camera;
[0040] Finally, use the Levenberg-Marquardt method to solve, and obtain the external parameters and
[0041] The present invention also provides a camera-LiDAR external parameter calibration device based on the association of line and plane features, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the camera-LiDAR external parameter calibration method based on the association of line and plane features as described in the above technical solution.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] 1. By combining line and plane features, this method significantly improves the calibration accuracy of the extrinsic parameters of the camera-LiDAR system and enhances the robustness of the algorithm to noise and different checkerboard pose changes.
[0044] 2. Compared with traditional methods, this method can achieve higher calibration accuracy with fewer checkerboard poses, thus improving the calibration efficiency.
[0045] 3. This calibration method is not only applicable to checkerboard targets but also adaptable to various calibration scenarios and targets, showing good generality and adaptability, and is suitable for the high-precision positioning system of driverless vehicles. Brief Description of the Drawings
[0046] Figure 1 It is the system flowchart of the present invention.
[0047] Figure 2 It is the device diagram of the present invention.
[0048] Figure 3 They are acquisition cases of different checkerboard poses.
[0049] Figure 4 They are schematic diagrams of camera feature extraction and LiDAR feature extraction.
[0050] Figure 5 They are the coordinate relationships between the camera and the LiDAR. Detailed Embodiment
[0051] The following further describes the present invention in detail with reference to the drawings and specific embodiments.
[0052] As Figure 1 shown, a method for calibrating the extrinsic parameters of a camera-LiDAR based on the association of line and plane features proposed by the present invention includes the following steps:
[0053] Step 1): The calibration system consists of an RS-LiDAR-16 lidar, an LI-USB30-AR023ZWDRB-6mm industrial camera, and a checkerboard. The device is as Figure 2 shown. Collect LiDAR scan data at multiple checkerboard poses, and the image sampling results at different poses are as Figure 3 shown. When extracting features from the camera, as Figure 4 (a) shown, extract features from the collected checkerboard images. First, detect the checkerboard and its boundaries in the image, and calculate the plane features of the i-th pose of the checkerboard in the camera coordinate system and the four boundary line features of its image plane where \(j = 1, 2, 3, 4\). The line features are detected from the image using the LSD algorithm. Given a checkerboard target, the four boundary lines enclosing the checkerboard can be detected and extracted as line features. To simplify the detection process of planar features, the checkerboard planar features are detected using the off-the-shelf OpenCV software. Then, the parameters of the planar features, including the plane normal vector, are calculated through the homography relationship (i.e., the point-to-point mapping relationship) between the checkerboard and its image. Plane equation parameters And generate a back-projected plane. Specifically: Use homogeneous coordinate transformation to transform the two-dimensional line segment into a three-dimensional line segment A back-projected plane can be generated based on the four boundaries. Let and be parameterized as and respectively, where is the normal of the plane with a length of 1 (i.e., unit length), and represent the direction and projection point of the boundary respectively, and the length of is also 1.
[0054] Step 2): Figure 4 (b) shows the schematic diagram of extracting features in the LiDAR coordinate system. In the LiDAR coordinate system, the checkerboard plane and its four boundaries are represented by and respectively. The laser points on the plane are represented by . The laser points on the boundary are represented by . Given and , the normal , centroid and direction of the plane in the LiDAR coordinate system can be calculated.First, the random sample consensus (RANSAC) algorithm is used to detect the laser points on the checkerboard plane and find the boundaries of each scan line, which are the left and right boundaries of the checkerboard. Then, the direction of the line defined by two consecutive points on the boundary is calculated, and the left and right boundaries are further divided into two parts by finding the maximum direction change amount. This operation is to refine the feature extraction and improve the robustness and accuracy of the calibration process, especially in the presence of noise and variable checkerboard poses. Since there is significant noise in the points on the boundary, the present invention uses the method of back-projection to remove the noise. The steps are as follows: First, project all the laser points onto the plane, then fit a straight line for each scan line, and project the boundary points onto each scan line. Although the scan line should be a conic curve, the present invention uses a straight line to approximate it because the checkerboard area is small and the curvature of the conic curve is very small and can be approximated as a straight line. Finally, the RANSAC algorithm is used to remove the potential outliers among these boundary points.
[0055] Step 3): As Figure 5 shown (schematic diagram of coordinate relationship), four types of geometric constraints are established based on line and plane features:
[0056] Step 301): The line feature direction alignment constraint is:
[0057]
[0058] Wherein, is the rotation matrix from LiDAR to camera, and are the line direction vectors in the LiDAR and camera coordinate systems respectively.
[0059] Step 302): The line feature projection point consistency constraint is:
[0060]
[0061] Wherein, I is the identity matrix, is the k-th laser point on the boundary on the i-th plane in the LiDAR coordinate system, is a projection point of the line feature in the camera coordinate system, is the translation vector.
[0062] Step 303): The plane normal vector alignment constraint is:
[0063]
[0064] Wherein, and are the plane normal vectors in the LiDAR and camera coordinate systems respectively.
[0065] Step 304): Construct the planar point consistency constraint, the checkerboard planar points in the LiDAR coordinate system After being rotated by the rotation matrix and the translation vector The transformation needs to satisfy the plane equation in the camera coordinate system, and the constraint formula is:
[0066]
[0067] where is the normal vector of the plane , is the m-th point on the i-th plane in the LiDAR coordinate system, t is the translation vector from LiDAR to the camera, is the constant term of the i-th plane equation in the camera coordinate system, which is determined by the plane equation .
[0068] Step 4): Solve the rotation matrix of the extrinsic parameters from LiDAR to the camera by minimizing the following cost function The cost function is described as follows:
[0069]
[0070] where N represents N checkerboard poses, is the direction vector of the j-th boundary in the i-th checkerboard pose in the LiDAR coordinate system, and represent the normal vectors of the i-th checkerboard pose plane in the LiDAR and camera coordinate systems respectively. There is a closed-form solution to the cost function minimization problem. First, the following definitions are given:
[0071]
[0072] M L is the feature matrix in the LiDAR coordinate system, which contains the plane normal vectors of each checkerboard pose and the four line directions to M C The corresponding feature matrix in the camera coordinate system. Assume that the singular value decomposition (Singular Value Decomposition, SVD) of M L (M C ) T is USV T , where S is the singular value matrix. Rotating the basis vectors U of the LiDAR coordinate system to the basis vectors V of the camera coordinate system to achieve the optimal alignment of geometric constraints, we can obtain Since there are fewer points on the boundary than on the plane, to avoid deviation, the present invention uses the centroid of the plane and the straight line The centroid of provides constraints for the It can be obtained that:
[0073]
[0074] Stacking equations (1), (2), (3), (4), and (5) can obtain a system of linear equations about There is also a closed-form solution to obtain the initial estimate and Then, jointly optimize by minimizing the following cost function, and the optimization function is described as follows:
[0075]
[0076] where N represents the poses of N checkerboards, t is the translation vector, is the boundary on the i-th plane in the LiDAR coordinate system the m-th laser point on ij K is the total number of laser points on the boundary in the LiDAR coordinate system, represents the boundary projection point.
[0077] Finally, the Levenberg-Marquardt (LM) method is used for solution to obtain the external parameters and
[0078] On the other hand, the embodiment of the present invention also provides a camera-LiDAR external parameter calibration device based on the association of line and plane features, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the camera-LiDAR external parameter calibration method based on the association of line and plane features as described in the above technical solution.
[0079] Except for the above embodiments, the present invention may also have other implementation manners. Any changes, modifications, substitutions, combinations, and simplifications made under any departure from the spirit and principle of the present invention shall be equivalent replacement manners and are all included in the protection scope required by the present invention.
Claims
1. A camera-LiDAR extrinsic calibration method based on line and plane feature association, characterized by: First, build a calibration system including LiDAR, industrial camera and checkerboard, collect LiDAR scanning data and checkerboard images in multiple poses, and then perform the following steps: Step 1), in the camera coordinate system, extract line features and plane features from the checkerboard image, calculate the parameters of the plane features, and generate a back-projection plane; Step 2), in the LiDAR coordinate system, extract line features and plane features of the LiDAR scan data, calculate the normal, centroid and direction of the plane in the LiDAR coordinate system, and remove the noise laser points on the chessboard plane; Step 3), establishing a geometric constraint relationship between the camera coordinate system and the LiDAR coordinate system based on the plane features and the line features; Step 4), using the feature correspondence under multiple chessboard postures, establish the cost function and optimization function to solve the rotation matrix and translation vector from LiDAR to the camera, and finally output the external parameter calibration result between the camera and LiDAR.
2. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: In step 1), the features of the acquired chessboard image are extracted. First, the chessboard and its boundaries in the image are detected, and the plane features of the chessboard at the i-th position in the camera coordinate system are calculated. And the four boundary line features of its image plane Where j = 1, 2, 3, 4; the line features are obtained by detecting the image using the LSD algorithm. Given a checkerboard target, the four boundary lines surrounding the checkerboard can be detected as line features for extraction; The plane features of the chessboard image are detected using the existing OpenCV software, and then the parameters of the plane features, including the plane normal vector, are calculated through the homography relationship between the chessboard and its image. Plane equation parameters And generate the back-projection plane.
3. The camera-LiDAR extrinsic calibration method based on line and plane feature association as claimed in claim 2, characterized in that: The specific implementation method of generating the back-projection plane is: using homogeneous coordinate transformation to transform the two-dimensional line segment Convert to 3D line segment A back-projection plane can be generated based on the four boundaries Will and Parameterized as and in Is a plane The normal of is 1 in length, and Represents the boundaries The direction and projection point of The length of is also 1.
4. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: In step 2), the chessboard plane and its four boundaries in the LiDAR coordinate system are represented by and Indicates that the plane The laser point on express, boundary The laser point on It means, k=1,...,K ij ; Given and Calculate the normal of the plane in the LiDAR coordinate system and centroid And direction 5. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: The random sampling consistency RANSAC algorithm is used to detect the laser points on the chessboard plane and find the boundary of each scan line, that is, the left and right boundaries of the chessboard. Then the direction of the line defined by two consecutive points on the boundary is calculated. The left and right boundaries are further divided into two parts by finding the maximum direction change, and the noise is removed by the back projection method. The specific implementation method of removing noise by back projection is as follows: First, all laser points are projected onto a plane, then a straight line is fitted for each scan line and the boundary points are projected onto each scan line. Finally, the RANSAC algorithm is used to remove potential outliers in the boundary points.
6. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: The geometric constraint relationships in step 3) include: line feature direction alignment constraint, line feature projection point consistency constraint, plane normal vector alignment constraint and plane point consistency constraint.
7. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: The geometric constraint relationship is constructed as follows: Step 301), the line feature direction alignment constraint is: in, is the rotation matrix from LiDAR to camera, and are the line direction vectors in the LiDAR and camera coordinate systems respectively; Step 302), the line feature projection point consistency constraint is: Where I is the identity matrix, is the boundary on the i-th plane in the LiDAR coordinate system The kth laser point on is a projection point of the line feature in the camera coordinate system, is the translation vector; Step 303), the plane normal vector alignment constraint is: in, and are the plane normal vectors in the LiDAR and camera coordinate systems respectively; Step 304), construct plane point consistency constraints, chessboard plane points in the LiDAR coordinate system Rotation Matrix and translation vectors After transformation, the plane equation in the camera coordinate system must be satisfied, and the constraint formula is: in, Is a plane The normal line of is the mth point on the i-th plane in the LiDAR coordinate system, t is the translation vector from LiDAR to the camera, is the constant term of the i-th plane equation in the camera coordinate system, and the plane equation Sure.
8. The camera-LiDAR extrinsic calibration method based on line and plane feature association according to claim 1, characterized in that: In step 4), the rotation matrix from LiDAR to camera is solved by minimizing the following cost function: Cost Function The description is as follows: Where N represents N chessboard positions, is the direction vector of the jth boundary under the i-th chessboard pose in the LiDAR coordinate system, and Represent the normal of the i-th chessboard pose plane in the LiDAR and camera coordinate systems, respectively.
9. The camera-LiDAR extrinsic calibration method based on line and plane feature association as claimed in claim 8, characterized in that: There is a closed-form solution to the cost function minimization problem. First, the following definition is given: M L is the feature matrix in the LiDAR coordinate system, containing the plane normal vector of each chessboard pose And four lines of direction arrive M C The corresponding feature matrix in the camera coordinate system; assuming M L (M C ) T The singular value decomposition of is USV T , where S is the singular value matrix, which rotates the basis vector U of the LiDAR coordinate system to the basis vector V of the camera coordinate system to achieve the optimal alignment of the geometric constraints, and obtains Use plane The centroid and straight line The center of mass is used as the initial translation vector The estimate provides constraints, defining get: Where I is the identity matrix; stacking equations (1)(2)(3)(4)(5) yields a matrix about The linear equations for which there is also a closed form solution are obtained, giving an initial estimate and Then we can jointly optimize by minimizing the following optimization function: The description is as follows: Where N represents N chessboard positions, is the rotation matrix, is the translation vector, is the boundary on the i-th plane in the LiDAR coordinate system The mth laser point on K ij is the total number of laser points on the lower boundary of the LiDAR coordinate system, Representation Boundary Projection point, t is the translation vector from LiDAR to the camera; Finally, the Levenberg-Marquardt method is used to solve the external parameter and 10. A camera-LiDAR extrinsic calibration device based on line and plane feature association, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the camera-LiDAR extrinsic parameter calibration method based on line and plane feature association as described in any one of claims 1 to 9.