A laser radar camera combined online calibration method based on composite boundary features

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a laser radar camera combined online calibration method based on composite boundary features. The laser radar camera combined online calibration method based on composite boundary features provided by the present application uses a regional regrowth method to fit a point cloud plane and obtain a depth discontinuous boundary feature, thereby solving the problem of poor boundary extraction effect caused by low plane fitting quality; the idea of composite boundary is adopted to fully utilize the complementary features of the depth continuous boundary and the depth discontinuous boundary, so as to improve the boundary quality and the calibration accuracy. The present application is tested in a self-collected simulation data set and a real scene data set respectively, in the large-scale simulation data set, the rotation angle error is controlled within 0.3 degrees, the translation matrix error is controlled within 4 centimeters, and compared with other advanced calibration methods, the present application has obvious performance advantages.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a joint online calibration method for lidar cameras based on composite boundary features. Background Technology

[0002] Joint calibration of LiDAR and cameras aims to obtain external calibration matrices between different sensors, thereby fusing depth information from different modalities and achieving complementary advantages between information. The process involves constructing a feature mapping relationship between the LiDAR point cloud and the camera, then establishing a KD tree and using principal component analysis (PCA) and least squares methods to formulate mathematical equations for solving the calibration matrix. The calibration matrix projects the 3D LiDAR points onto the camera coordinate system, and then the camera model is used to project the 3D points onto the pixel plane. Since the calibration results directly affect the information fusion effect, calibration technology is crucial for multi-sensor information interaction. For example, joint calibration technology has been widely used in fields such as autonomous driving, robotics, and mountain mapping.

[0003] Current joint calibration methods have achieved significant results, both in target-based calibration using calibration boards and targetless calibration based on object features, resulting in substantial improvements in calibration accuracy. However, these methods only consider solid-state LiDAR and mechanically scanned LiDAR; they are not applicable to mechanically scanned LiDAR (such as VLP-16). For solid-state and mechanically scanned LiDAR, 3D point cloud data containing continuous depth boundaries can be obtained by accumulating several frames of point cloud data. For scanned LiDAR, the top of the object cannot be detected during operation, resulting in depth discontinuities on the sides and top of the object, leading to depth discontinuities in the 3D point cloud data. Current methods do not utilize this feature, but it is crucial for boundary feature matching. Summary of the Invention

[0004] The purpose of this invention is to provide a joint online calibration method for LiDAR cameras based on composite boundary features. This method uses region complex growth to extract depth discontinuous boundaries and combines two boundary features—depth continuous boundaries—for matching and calibration. This method is applicable not only to mechanically scanned LiDARs but also improves the accuracy of the calibration matrix.

[0005] The technical solution of this invention is:

[0006] A joint online calibration method for lidar cameras based on composite boundary features, such as... Figure 1 As shown, it includes the following steps:

[0007] S1. Acquire point cloud data and image data using LiDAR and camera respectively;

[0008] S2. Based on the acquired point cloud data, perform depth discontinuity boundary extraction and depth continuous boundary extraction respectively. Specifically:

[0009] A region regeneration strategy is used to extract depth discontinuities in point cloud data, such as... Figure 2 As shown, it includes:

[0010] a1. Based on the acquired point cloud data, construct a KD tree and calculate the covariance matrix for each point:

[0011]

[0012] in, It is the covariance matrix. Generated by KNN One point, They are adjacent data points. It is the average vector of all points; the points are obtained by performing singular value decomposition on the covariance matrix. normal vector The curvature at that point is represented by the eigenvalues ​​corresponding to the eigenvectors. ;

[0013] a2. Definition unit point The scale represents the distance between the point and its third nearest neighbor, and its value is used to set the value in step a3 below. Initial value; The nearest neighbor is represented as a set. ;

[0014] a3. Arrange all points according to the rule of increasing curvature, starting from the first point. Begin by creating a file containing... The set of coplanar points, for each unit point in the set Query its Each point in A point is considered a coplanar point and added to the set if it satisfies the following conditions:

[0015]

[0016] in, It is a set constant value. , The region gradually grows as the number of points increases, and eventually all the points are stored in a list.

[0017] a4. Repeat a2 until all points have been processed, and denote the set of all the resulting planar regions as R;

[0018] a5. For any element region in the obtained set of planar regions R , for the region Label the points in the array, and observe each point. If a point in this set has a different label, then the region containing that point with the different label is called the set. The neighborhood of the region can be obtained, thus yielding a neighborhood set. Following steps a1 and a2, the normal vector, curvature, and scale of the region are calculated. Then, the region growing strategy is repeated to perform complex region growing, resulting in a higher quality plane.

[0019] a6. Extract the 2D line segments of the complete plane and backproject them to obtain the 3D line segments, which are the required depth discontinuity boundaries, such as... Figure 3 As shown in (a) in the text;

[0020] Based on voxelization, depth continuity boundaries are extracted. Specifically, the point cloud is divided into voxels of a given size. For each voxel, the RANSAC algorithm is used to fit and extract the planes contained within the voxel. Plane pairs forming corresponding angles are retained, and the intersection lines between these plane pairs are the desired depth continuity boundaries. Depth continuity boundaries and depth discontinuity boundaries are distinguished as follows: Figure 3 As shown in (b);

[0021] S3. Based on the acquired image data, the Canny algorithm is used to extract image boundaries;

[0022] S4. Perform feature matching between the boundaries extracted in S2 and S3, specifically including:

[0023] For each point cloud boundary extracted from S2, sample several points. And transform it to the camera coordinate system:

[0024]

[0025] in, , Represents a rotation matrix Represents the translation matrix. This represents a rigid transformation from the lidar coordinate system to the camera coordinate system, where a point is projected onto the camera's imaging plane to obtain a pixel coordinate. , ,in For distortion correction function, For the camera's projection function;

[0026] For each pixel point extracted from the image boundary in S3 Construct a KD-tree and search of Neighbor ,set up:

[0027]

[0028] For sets A straight line is generated by a point on the line. and normal vector The two parameters describe, yes The eigenvector corresponding to the smallest eigenvalue;

[0029] S5. Establish the characteristic equation and perform optimization of the calibration matrix. The specific solution process is as follows:

[0030] For radar points ,definition For its measurement noise, for a pixel, define Measure the noise;

[0031] After projecting it onto the image plane using a standard initial rotation matrix, it falls on... The boundary formed satisfies the following formula:

[0032]

[0033] Solve iteratively: Let... For the current extrinsic parameter estimation, use Operators in tangent space After parameterization, we obtain the following formula:

[0034]

[0035] in , , , .

[0036] The optimal solution is obtained using maximum likelihood estimation:

[0037]

[0038] The obtained solution is used for iterative updates. :

[0039]

[0040] Iterate continuously until the convergence condition is met: , The initial threshold is set, and the final convergence is... This is the calibration matrix we are looking for.

[0041] The beneficial effects of this invention are as follows: The online calibration method for LiDAR cameras based on composite boundary features provided by this invention uses region complex growth to fit the point cloud plane and obtain depth discontinuity boundary features, solving the problem of poor boundary extraction results caused by low plane fitting quality; by adopting the idea of ​​composite boundaries, it fully utilizes the complementary advantages of depth continuous boundaries and depth discontinuity boundaries, thereby improving boundary quality and calibration accuracy; the entire processing flow is based on the features of the object itself rather than the target calibration board, allowing the algorithm to be deployed on the equilateral end of the robot. This invention was tested on self-collected simulation datasets and real-world scene datasets. In the large-scale simulation dataset, the obtained rotation angle error was controlled within 0.3 degrees, and the translation matrix error was controlled within 4 centimeters, demonstrating significant performance advantages compared to other advanced calibration methods. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0043] Figure 2 This is a schematic diagram of the depth discontinuity boundary extraction process in this invention;

[0044] Figure 3 This is a comparative diagram of depth discontinuity boundaries and composite boundaries in this invention, where (a) is the extraction result of depth discontinuity boundaries and (b) is the extraction result of composite boundaries;

[0045] Figure 4 A visual diagram illustrating the calibration results. Detailed Implementation

[0046] The effectiveness of the present invention will be described below with reference to scenario examples.

[0047] Scenario Example

[0048] Scene calibration involved collecting 12 sets of data from the simulation scene and 6 sets of data from the real scene, and then processing the data. The random transformations between the parameters are then performed, and after data preprocessing, the method of this invention is used for joint calibration of the lidar and camera. An example of the calibration visualization results is shown below. Figure 4 As shown.

[0049] The algorithm designed in this invention, which combines depth discontinuity boundaries with depth continuity boundaries based on region regeneration, was used to perform calibration tests on 18 sets of data from simulation and real-world scenarios. The quantitative evaluation was performed using six-degree-of-freedom error, as shown in Tables 1 and 2.

[0050] Table 1 Comparison of six-DOF errors in GAZEBO simulation data

[0051]

[0052] Table 2 Comparison of Six Degrees of Freedom Errors Based on Real Data

[0053]

[0054] Meanwhile, comparative experiments were conducted on the large-scale KITTI raw data dataset, as shown in Table 3. Data sets 11, 44, and 218 from the urban scene 2011_09_26_drive_0095_sync were selected for experiments, and compared with other advanced deep learning methods. The calibration accuracy of this invention still shows an advantage. A significant advantage of this invention over the compared deep learning methods is that its method can run in real time and has stronger generalization capabilities.

[0055] Table 3 Comparison of six-DOF errors in kitti raw data

[0056]

[0057] Tests show that, thanks to the depth discontinuity of the region's regrown boundaries and the complementary advantages of composite boundaries, the calibration method of this invention significantly improves calibration errors. Furthermore, this invention is based on the characteristics of natural objects, enabling the algorithm to run in real-time within robotic systems.

Claims

1. A joint online calibration method for lidar cameras based on composite boundary features, characterized in that, Includes the following steps: S1. Acquire point cloud data and image data using LiDAR and camera respectively; S2. Based on the acquired point cloud data, perform depth discontinuity boundary extraction and depth continuous boundary extraction respectively. Specifically: A region regeneration strategy is used to extract depth discontinuities in point cloud data, including: a1. Based on the acquired point cloud data, construct a KD tree and calculate the covariance matrix for each point: ， in, It is the covariance matrix. Generated by KNN One point, They are adjacent data points. It is the average vector of all points; the points are obtained by performing singular value decomposition on the covariance matrix. normal vector The curvature at that point is represented by the eigenvalues ​​corresponding to the eigenvectors. ; a2. Definition unit point The size of the value represents the distance between the point and its third nearest neighbor. The nearest neighbor is represented as a set. ; a3. Arrange all points according to the rule of increasing curvature, starting from the first point. Begin by creating a file containing... The set of coplanar points, for each unit point in the set Query its Each point in A point is considered a coplanar point and added to the set if it satisfies the following conditions: ， in, It is a set constant value. , , The initial value is defined in a2. The region gradually grows as the number of points increases, and eventually all the points are stored in a list. a4. Repeat a2 until all points have been processed, and denote the set of all the planar regions obtained as R; a5. For any element region in the obtained set of planar regions R , for the region Label the points in the array, and observe each point. If a point in this set has a different label, then the region containing that point with the different label is called the set. The neighborhood of the region can be obtained, thus a neighborhood set can be obtained; the normal vector, curvature and scale of the region can be obtained by the same method as steps a1 and a2, and then the region growth strategy can be repeatedly used to perform region complex growth to obtain a higher quality plane. a6. Extract the two-dimensional line segments of the complete plane obtained from a5 and back-project them to obtain the three-dimensional line segments, which are the required depth discontinuity boundaries; The method for extracting depth continuity boundaries based on voxelization is as follows: the point cloud is divided into voxels of a given size. For each voxel, the RANSAC algorithm is used to fit and extract the planes contained in the voxel. The plane pairs that form the corresponding angles are retained. The intersection line between the plane pairs is the desired depth continuity boundary. S3. Based on the acquired image data, the Canny algorithm is used to extract image boundaries; S4. Perform feature matching between the boundaries extracted in S2 and S3, specifically including: For each point cloud boundary extracted from S2, sample several points. And transform it to the camera coordinate system: ， in, , Represents a rotation matrix Represents the translation matrix. This represents a rigid transformation from the lidar coordinate system to the camera coordinate system, where a point is projected onto the camera's imaging plane to obtain a pixel coordinate. , ,in For distortion correction function, For the camera's projection function; For each pixel point extracted from the image boundary in S3 Construct a KD-tree and search of Neighbor ,set up: ， For sets A straight line is generated by a point on the line. and normal vector The two parameters describe, yes The eigenvector corresponding to the smallest eigenvalue; S5. Establish the characteristic equation and perform optimization of the calibration matrix. The specific solution process is as follows: For radar points ,definition For its measurement noise, for a pixel, define Measure the noise; After projecting it onto the image plane using a standard initial rotation matrix, it falls on... The boundary formed satisfies the following formula: ， Solve iteratively: Let... For the current extrinsic parameter estimation, use Operators in tangent space After parameterization, we obtain the following formula: ， in , , , ; The optimal solution is obtained using maximum likelihood estimation: ， The obtained solution is used for iterative updates. : ， Iterate continuously until the convergence condition is met: , The initial threshold is set, and the final convergence is... This is the calibration matrix we are looking for.

Images