Lidar-camera decoupled calibration method for structured scenes

Abstract

This invention discloses a decoupled calibration method for LiDAR cameras in structured scenes, belonging to the fields of robot perception, computer vision, and autonomous driving technology. First, straight line features are extracted from both the image and the LiDAR point cloud, estimating the orthogonal vanishing point of the image and the orthogonal direction of the point cloud. Then, high-quality data is selected through multi-dimensional quality scores. Next, the rotation matrices of the camera and LiDAR relative to the world coordinate system are calculated using orthogonal direction constraints, thereby decoupling the LiDAR to camera rotation matrix. Finally, the translation vector is solved by minimizing the projection geometry error by matching the commonly observed straight line features. This invention eliminates the dependence on dedicated calibration boards and adopts an optimization strategy of rotation and translation decoupling, effectively avoiding local optima problems and significantly improving the flexibility, accuracy, and robustness of calibration. It is particularly suitable for indoor structured environments or outdoor scenes with Manhattan world characteristics.

Description

Technical Field

[0001] This invention relates to the fields of robot perception, computer vision and autonomous driving technology, and in particular to a method for decoupling and calibrating LiDAR cameras for structured scenes. Background Technology

[0002] LiDAR can acquire high-precision 3D geometric information of the environment, while cameras provide rich texture and color information. Their fusion applications in fields such as autonomous driving, mobile robotics, and 3D reconstruction are becoming increasingly widespread. Effective multi-sensor fusion relies on accurate extrinsic parameter calibration, i.e., determining the rotation matrix and translation vector between the LiDAR coordinate system and the camera coordinate system. Therefore, researching efficient, stable, and widely applicable joint calibration methods has significant theoretical and practical implications.

[0003] Existing extrinsic parameter calibration methods can be broadly categorized into two types: target-based methods and targetless (natural scene-based) methods. Target-based calibration methods typically use checkerboard patterns, ArUco codes, or calibration boards with specific geometric shapes. They solve for extrinsic parameters by detecting the corresponding features of the target in the image and point cloud (such as corner points and center points) and using point cloud registration or PnP algorithms. Their advantages lie in high accuracy and intuitive principles, but they are highly dependent on dedicated, well-made calibration boards, lack deployment flexibility, and are difficult to adapt to large-scale or dynamically changing outdoor scenes.

[0004] Targetless calibration methods directly utilize structural features present in natural scenes (such as edges, planes, and lines) for calibration by maximizing mutual information, aligning edge features, or using motion estimation. These methods are more flexible as they eliminate the reliance on dedicated targets. However, their calibration accuracy and stability are often limited by the robustness of feature extraction, the richness of scene structure, and the influence of noise. For example, mutual information-based methods are sensitive to initial values, while visual odometry-based methods are prone to failure in texture-poor or dynamic environments.

[0005] Furthermore, in traditional extrinsic parameter optimization frameworks, rotation matrices and translation vectors are typically treated as a coupled whole, jointly solved by minimizing reprojection errors or feature alignment errors. However, this coupled optimization approach suffers from two significant problems: first, the optimization problem is strongly non-convex, easily trapped in local optima, and extremely sensitive to initial values; second, the geometric constraints upon which rotation and translation rely are fundamentally different—rotation is primarily constrained by orientation information, while translation depends on precise positional correspondence. When feature extraction is noisy or occlusion exists in the scene, positional errors directly interfere with rotation estimation, thus limiting the accuracy and robustness of the overall calibration. Summary of the Invention

[0006] The purpose of this invention is to propose a decoupling calibration method for lidar cameras in structured scenarios. This method does not require a special calibration board and directly utilizes the orthogonal straight line features existing in natural scenes or structured environments to achieve high-precision and robust extrinsic parameter calibration through rotation and translation decoupling estimation.

[0007] To achieve the above objectives, this invention provides a method for decoupling and calibrating LiDAR cameras for structured scenes, comprising the following steps: Step S1: Extract straight line features from the image respectively and estimate three sets of orthogonal vanishing points and corresponding orthogonal direction vectors in the camera coordinate system based on the Manhattan world hypothesis. Extract three-dimensional straight line features from the lidar point cloud and determine three sets of orthogonal direction vectors in the lidar coordinate system. Step S2: When there are several sets of synchronously acquired images and lidar point cloud data, the orthogonal geometric constraints of the extracted straight line features are used to calculate the quality score of each set of data, and high-quality data are selected for subsequent calibration. Step S3: Using the camera-world rotation matrix derived from the orthogonal vanishing point of the image and the lidar-world rotation matrix derived from the orthogonal direction of the point cloud, decouple and calculate the rotation matrix from the lidar coordinate system to the camera coordinate system. ; Step S4: Match the line features observed in both the image and the point cloud, construct position constraints based on the solved rotation matrix, and optimize the translation vector from the LiDAR coordinate system to the camera coordinate system by minimizing the geometric error of the line projection. .

[0008] Preferably, step S1 includes the following specific steps: S11: Extract a set of two-dimensional lines from an image using the LSD line detection algorithm in OpenCV. ,in Indicates the first The coordinates of the two endpoints of the straight line in the image; S12: Extracting a set of three-dimensional straight lines from a lidar point cloud using a planar projection method. ,in Indicates the first The coordinates of the two endpoints of a three-dimensional straight line; S13: Based on the extracted image line set Restore the three sets of orthogonal vanishing points that satisfy the Manhattan world hypothesis. and its corresponding camera coordinate system orthogonal direction vector ; S14: Based on the extracted set of 3D lines To restore the three sets of orthogonal direction vectors in the lidar coordinate system that satisfy the Manhattan world hypothesis. .

[0009] Preferably, in step S12, the planar projection method specifically involves: first, performing planar segmentation on the lidar point cloud to obtain each planar point set. Each set of points on the plane is projected onto its corresponding plane to generate a two-dimensional binary image. Two-dimensional lines are then extracted from the binary image and back-projected into three-dimensional space to obtain a set of three-dimensional lines. .

[0010] Preferably, in step S13, the image orthogonal vanishing point estimation specifically includes: S131: Construct a polar coordinate grid centered on the image principal point. The score is calculated by accumulating the number of intersections of all straight lines in the grid cells. S132: Assumption of randomly selecting two lines from the image to generate the first vanishing point Through the camera intrinsic parameter matrix Calculate the corresponding unit direction vector The second orthogonal direction vector is obtained by uniformly sampling on the unit sphere where the direction vector is located. And calculate its corresponding vanishing point. ; S133: Calculate the third orthogonal direction vector using the cross product of vectors. This forms a set of orthogonal vanishing point hypotheses. ; S134: Calculate the total score for each orthogonal vanishing point hypothesis. The hypothesis with the highest score is selected as the optimal orthogonal vanishing point combination. And obtain the corresponding orthogonal direction vector of the camera coordinate system. .

[0011] Preferably, in step S14, determining the orthogonal direction of the point cloud specifically includes: S141: Construct a polar coordinate grid centered at the origin of the lidar coordinate system. The score is calculated by accumulating the number of directions of all three-dimensional lines in each grid cell. S142: Randomly select two three-dimensional straight lines to generate the first orthogonal direction assumption. The second orthogonal direction vector is obtained by uniformly sampling on the unit sphere. ; S143: Calculate the third orthogonal direction vector using the cross product of vectors. This forms a set of orthogonal direction hypotheses. ; S144: Calculate the total score for each orthogonal direction assumption. The hypothesis with the highest score is selected as the optimal orthogonal direction combination. .

[0012] Preferably, in step S2, the data quality screening specifically includes: S21: Calculate the score for the first dimension , used to measure the validity of the orthogonal direction itself, where Total score for orthogonal directions; S22: Calculate the score for the second dimension , is used to measure the significance of the optimal orthogonal direction relative to all possible directions, where The sum of scores for all orthogonal direction assumptions; S23: Calculate the score for the third dimension This is used to measure the balance of scores in the three orthogonal directions, where This indicates taking the second largest of the three values; S24: Calculate the overall data quality score ,in , , The preset weighting coefficients satisfy... ; S25: Scale the overall quality score ,in The scaling factor is used to filter out... Image and point cloud data pairs that exceed a preset threshold.

[0013] Preferably, in step S3, the calculation of the rotation matrix specifically includes: S31: The orthogonal direction vector in the camera coordinate system According to the world coordinate system , , Arrange the axes in order to obtain the rotation matrix from the camera coordinate system to the world coordinate system. ; S32: Orthogonal direction vector in the lidar coordinate system Perform coordinate system alignment transformation and multiply by the transformation matrix. Then according to the world coordinate system , , Arrange the axes in order to obtain the rotation matrix from the lidar coordinate system to the world coordinate system. ; S33: Calculate the rotation matrix from the lidar coordinate system to the camera coordinate system using matrix operations. ,in for The transpose of .

[0014] Preferably, in step S4, the translation vector calculation specifically includes: S41: Set of lines in an image matched manually or automatically With point cloud line set Corresponding straight line pairs jointly observed by China; S42: Using the solved rotation matrix Transform the three-dimensional straight line in the lidar coordinate system to the camera coordinate system; S43: Project a 3D straight line in the camera coordinate system onto the image plane using a camera projection model. Construct a geometric error objective function between the image line and the projected line, and solve for the optimal translation vector. .

[0015] Preferably, the objective function for geometric error in step S43 is: ; in, A parameterized representation of a straight line in an image. For the parameterized representation of a three-dimensional straight line in a lidar system, Let be the rotation matrix from the lidar coordinate system to the camera coordinate system. Let be the translation vector from the lidar coordinate system to the camera coordinate system. This is the camera perspective projection function.

[0016] Therefore, the above-mentioned decoupling calibration method for lidar cameras in structured scenarios, adopted in this invention, has the following advantages: (1) No special calibration plate is required. The calibration can be performed directly by utilizing the orthogonal structural features that are common in indoor and outdoor Manhattan environments, which significantly reduces the calibration cost and expands the calibration scenarios. It is especially suitable for indoor structured environments or outdoor scenarios with Manhattan world characteristics. (2) The decoupling strategy of "rotation first, translation later" is adopted. First, the rotation matrix is ​​robustly solved by the strong directional constraints implied by the vanishing point and the three-dimensional line, which avoids the interference of position noise on rotation estimation in the traditional method. Under the premise that the rotation is known, the solution of the translation vector is transformed into a linear or approximately linear optimization problem with better conditions, which improves the numerical stability and convergence reliability of the overall calibration process. (3) A multi-set data quality screening mechanism was introduced to evaluate the applicability of the data from multiple dimensions, eliminate low-quality data, and further improve the accuracy and robustness of the calibration results.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall process of the lidar camera decoupling calibration method for structured scenes in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the point cloud orthogonal direction extraction method in an embodiment of the present invention; Figure 3 This is a schematic diagram of the image orthogonal vanishing point extraction method in an embodiment of the present invention; Figure 4 This is a schematic diagram of the projection process of a conventional cube on a single surface in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the correspondence between the image and the linear features of the point cloud in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the process of reconstructing the rotation matrix of a cube in laser point cloud and image in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the process of solving the translation vector based on the optimization method in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] Example like Figure 1-7 As shown, this embodiment proposes a decoupling calibration method for LiDAR cameras in structured scenes. Its core lies in utilizing the ubiquitous Manhattan orthogonal structure in the environment to extract dominant orientation information from both the image and point cloud, achieving decoupling estimation of the rotation matrix and translation vector. The specific implementation steps are as follows: Step S1: Extract straight lines from the image and estimate the orthogonal vanishing points based on the Manhattan world hypothesis. Extract 3D straight lines from the LiDAR point cloud and determine the direction vectors of their orthogonal directions. Specifically: S11: Extract the set of lines from the Manhattan world image in S1 using the LSD algorithm in OpenCV. .

[0022] S12: Extract spatial straight lines from the lidar point cloud in S1 using an algorithm. Specifically, it includes: S121: Perform planar segmentation on the lidar point cloud to obtain the point sets of each planar region. ; S122: Combine the set of points on the plane Project it onto its plane to generate a two-dimensional binary image; S123: Extract two-dimensional lines from a binary image and backproject them into three-dimensional space to obtain three-dimensional lines. .

[0023] S13: As Figure 5 , Figure 6 The image shows the set of straight lines extracted from S11 that satisfy the Manhattan world hypothesis. Restoring the orthogonal vanishing point and its corresponding orthogonal direction vector .

[0024] S14: As Figure 3 As shown, the set of spatial lines that satisfy the Manhattan world hypothesis extracted from S12. Restore orthogonal direction Specifically, it includes: S141: Construct a polar coordinate grid centered at the origin. The direction values ​​of the cumulative lines are accumulated; S142: Randomly select a straight line to generate the first orthogonal direction. Assuming, and since the world conforms to the Manhattan world hypothesis, it can be approximated that the world is roughly composed of three orthogonal straight lines, and that a second orthogonal direction is uniformly sampled on its unit sphere. ; S143: Calculate the third orthogonal direction using cross product. This forms a set of orthogonal direction hypotheses. ; S144: As Figure 4 As shown, by searching the polar coordinate grid, the scores of elements in the generated orthogonal direction hypothesis set can be obtained from... Get the maximum score and select it. The corresponding orthogonal directions are the optimal combination of orthogonal directions. .

[0025] Step S2: Utilize the geometric features of the extracted lines, i.e., the relationship of orthogonal directions, such as... Figure 2 As shown, when there are multiple sets of images / LiDAR point clouds, the quality of each set of LiDAR point clouds and images is determined, and the images are filtered.

[0026] S21: Considering the score values ​​in the orthogonal directions, we obtain... ,in Can represent ( This represents the score of an image or point cloud in the orthogonal direction; similar values ​​follow the same pattern and will not be elaborated further. The validity of the direction itself is obtained.

[0027] S22: Considering the ratio between the orthogonal direction and the sum of the scores of all lines, we get... The rationality of a direction relative to other directions is obtained, not from the randomness of the direction itself.

[0028] S23: Considering the proportions of each orthogonal direction, we obtain The rationale for obtaining the values ​​in each direction is not due to any particular direction having a high value, but rather because of the Manhattan world hypothesis.

[0029] S24: Consider obtaining a total score for the image / point cloud. .in This represents the importance of each score.

[0030] S25: To increase the proportion of its value, the value is... Its purpose is to... By scaling the scale, we can obtain more obvious changes.

[0031] Step S3: As Figure 5 , 6 As shown, the rotation matrix from the lidar coordinate system to the camera coordinate system is calculated using orthogonal vanishing points and orthogonal directions. The specific process is as follows: S31: Orthogonalize the image direction vector , and with axis, axis, By arranging the axes in order, we obtain the rotation matrix from the camera coordinate system to the world coordinate system. ; S32: The orthogonal direction vector extracted by the lidar In the camera coordinate system, the Z-axis generally represents the direction directly facing the camera, while in the lidar coordinate system, the Z-axis generally represents the upward direction. Therefore, during the transformation, it is necessary to align their coordinate systems, that is, to right-multiply by a matrix in the orthogonal direction from the lidar to the world. Then take it with axis, axis, The order of the axes yields the rotation matrix from the lidar coordinate system to the world coordinate system. ; S33: Using the two rotation matrices above, obtain the rotation matrix from the lidar coordinate system to the camera coordinate system. .

[0032] Step S4: As Figure 7 As shown, by matching the corresponding straight line features in the image and point cloud, positional constraints are constructed and the translation vector from the LiDAR coordinate system to the camera coordinate system is optimized. Specifically, it includes: S41: Select the image In point clouds Multiple straight-line features observed jointly in the middle; S42: Based on the obtained rotation matrix Transform the straight line in the lidar coordinate system to the camera coordinate system; S43: Construct an objective function and optimize the solution for the translation vector by minimizing the geometric error between the image line and the projected line. .

[0033] The objective function for geometric error is: ; in, A parameterized representation of a straight line in an image. For the parameterized representation of a three-dimensional straight line in a lidar system, Let be the rotation matrix from the lidar coordinate system to the camera coordinate system. Let be the translation vector from the lidar coordinate system to the camera coordinate system. This is the camera perspective projection function.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for decoupled calibration of a laser radar camera for structured scenes, characterized in that, Includes the following steps: Step S1: Extract straight line features from the image respectively and estimate three sets of orthogonal vanishing points and corresponding orthogonal direction vectors in the camera coordinate system based on the Manhattan world hypothesis. Extract three-dimensional straight line features from the lidar point cloud and determine three sets of orthogonal direction vectors in the lidar coordinate system. Step S2: When there are several sets of synchronously acquired images and lidar point cloud data, the orthogonal geometric constraints of the extracted straight line features are used to calculate the quality score of each set of data, and high-quality data are selected for subsequent calibration. Step S3: Decouple the rotation matrix from LiDAR coordinate system to camera coordinate system using the camera-world rotation matrix derived from image orthogonal vanishing points and the LiDAR-world rotation matrix derived from point cloud orthogonal directions ; Step S4: Match the line features observed in both the image and the point cloud, construct position constraints based on the solved rotation matrix, and optimize the translation vector from the LiDAR coordinate system to the camera coordinate system by minimizing the geometric error of the line projection. .

2. The lidar camera decoupling calibration method for structured scenes according to claim 1, characterized in that: Step S1 includes the following specific steps: S11: Extract a set of two-dimensional lines from an image using the LSD line detection algorithm in OpenCV. ,in Indicates the first The coordinates of the two endpoints of the straight line in the image; S12: Extracting a set of three-dimensional straight lines from a lidar point cloud using a planar projection method. ,in Indicates the first The coordinates of the two endpoints of a three-dimensional straight line; S13: Based on the extracted image line set Restore the three sets of orthogonal vanishing points that satisfy the Manhattan world hypothesis. and its corresponding camera coordinate system orthogonal direction vector ; S14: Based on the extracted set of 3D lines To restore the three sets of orthogonal direction vectors in the lidar coordinate system that satisfy the Manhattan world hypothesis. .

3. The lidar camera decoupling calibration method for structured scenes according to claim 2, characterized in that: In step S12, the planar projection method specifically involves: first, performing planar segmentation on the lidar point cloud to obtain each planar point set. Each set of points on the plane is projected onto its corresponding plane to generate a two-dimensional binary image. Two-dimensional lines are then extracted from the binary image and back-projected into three-dimensional space to obtain a set of three-dimensional lines. .

4. The lidar camera decoupling calibration method for structured scenes according to claim 2, characterized in that: In step S13, the estimation of the orthogonal vanishing point of the image specifically includes: S131: Construct a polar coordinate grid centered on the image principal point. The score is calculated by accumulating the number of intersections of all straight lines in the grid cells. S132: Assumption of randomly selecting two lines from the image to generate the first vanishing point Through the camera intrinsic parameter matrix Calculate the corresponding unit direction vector The second orthogonal direction vector is obtained by uniformly sampling on the unit sphere where the direction vector is located. And calculate its corresponding vanishing point. ; S133: Calculate the third orthogonal direction vector using the cross product of vectors. This forms a set of orthogonal vanishing point hypotheses. ; S134: Calculate the total score for each orthogonal vanishing point hypothesis. The hypothesis with the highest score is selected as the optimal orthogonal vanishing point combination. And obtain the corresponding orthogonal direction vector of the camera coordinate system. .

5. The lidar camera decoupling calibration method for structured scenes according to claim 2, characterized in that: In step S14, determining the orthogonal direction of the point cloud specifically includes: S141: Construct a polar coordinate grid centered at the origin of the lidar coordinate system. The score is calculated by accumulating the number of directions of all three-dimensional lines in each grid cell. S142: Randomly select two three-dimensional straight lines to generate the first orthogonal direction assumption. The second orthogonal direction vector is obtained by uniformly sampling on the unit sphere. ; S143: Calculate the third orthogonal direction vector using the cross product of vectors. This forms a set of orthogonal direction hypotheses. ; S144: Calculate the total score for each orthogonal direction assumption. The hypothesis with the highest score is selected as the optimal orthogonal direction combination. .

6. The lidar camera decoupling calibration method for structured scenes according to claim 1, characterized in that: In step S2, data quality screening specifically includes: S21: Calculate the score for the first dimension , used to measure the validity of the orthogonal direction itself, where Total score for orthogonal directions; S22: Calculate the score for the second dimension , is used to measure the significance of the optimal orthogonal direction relative to all possible directions, where The sum of scores for all orthogonal direction assumptions; S23: Calculate the score for the third dimension This is used to measure the balance of scores in the three orthogonal directions, where This indicates taking the second largest of the three values; S24: Calculate the overall data quality score ,in , , The preset weighting coefficients satisfy... ; S25: Scale the overall quality score ,in The scaling factor is used to filter out... Image and point cloud data pairs that exceed a preset threshold.

7. The lidar camera decoupling calibration method for structured scenes according to claim 1, characterized in that: In step S3, the calculation of the rotation matrix specifically includes: S31: The orthogonal direction vector in the camera coordinate system According to the world coordinate system , , Arrange the axes in order to obtain the rotation matrix from the camera coordinate system to the world coordinate system. ; S32: Orthogonal direction vector in the lidar coordinate system Perform coordinate system alignment transformation and multiply by the transformation matrix. Then according to the world coordinate system , , Arrange the axes in order to obtain the rotation matrix from the lidar coordinate system to the world coordinate system. ; S33: Calculate the rotation matrix from the lidar coordinate system to the camera coordinate system using matrix operations. ,in for The transpose of .

8. The lidar camera decoupling calibration method for structured scenes according to claim 1, characterized in that: In step S4, the translation vector calculation specifically includes: S41: Set of lines in an image matched manually or automatically With point cloud line set Corresponding straight line pairs observed jointly by China; S42: Using the solved rotation matrix Transform the three-dimensional straight line in the lidar coordinate system to the camera coordinate system; S43: Project a 3D straight line in the camera coordinate system onto the image plane using a camera projection model. Construct a geometric error objective function between the image line and the projected line, and solve for the optimal translation vector. .

9. The lidar camera decoupling calibration method for structured scenes according to claim 8, characterized in that: The objective function for geometric error in step S43 is: ； in, A parameterized representation of a straight line in an image. For the parameterized representation of a three-dimensional straight line in a lidar system, Let be the rotation matrix from the lidar coordinate system to the camera coordinate system. Let be the translation vector from the lidar coordinate system to the camera coordinate system. This is the camera perspective projection function.

Images