A method for calibrating external parameters of a lidar plus camera system

Abstract

The present invention discloses a method for calibrating the extrinsic parameters of a laser radar and camera system. The method specifically comprises: using a circular plane calibration plate with a two-dimensional coded pattern printed on its center to synchronously collect laser radar point cloud and camera image data from the circular plane calibration plate; using a plane extraction algorithm to segment the laser radar point cloud subset belonging to the calibration plate from the laser radar point cloud, and calculating the centroid coordinates of the point cloud subset as three-dimensional feature points; detecting the corner points of the two-dimensional coded pattern in the camera image and calculating their geometric centers as two-dimensional feature points; establishing multiple sets of three-dimensional and two-dimensional feature point correspondences through multiple scans, and solving the initial extrinsic parameters using a PnP algorithm; and obtaining the final extrinsic parameters by minimizing the reprojection error using a nonlinear optimization algorithm. The proposed method is suitable for application scenarios such as robotics, autonomous driving, or three-dimensional reconstruction, and can achieve accurate, efficient, and automatic extrinsic parameter calibration.

Description

Technical Field

[0001] The present invention relates to the field of sensor calibration technology, and in particular to a method for calibrating extrinsic parameters of a laser radar plus camera system. Background Art

[0002] LiDAR-camera systems are commonly used in autonomous systems and 3D perception to fuse two sensors for data acquisition. LiDAR provides precise depth and 3D structure information, while cameras provide rich color and texture information. To effectively fuse the data from these two sensors and leverage their complementary strengths, accurately calibrating their relative pose (i.e., extrinsic parameters: rotation and translation) is crucial.

[0003] Currently, there are various methods for calibrating the extrinsic parameters of lidar and cameras. Some methods rely on specific calibration objects, such as checkerboards and ArUco marker arrays. These methods typically extract the 2D feature points of the calibration object in the image and the 3D feature points in the point cloud, establish a correspondence, and then solve the PnP problem to estimate the extrinsic parameters. For example, checkerboard-based methods require detecting checkerboard corners in the image, manually selecting or automatically segmenting the checkerboard plane in the point cloud, and then extracting the corner points or plane centers.

[0004] However, existing methods still face several challenges: First, for a square calibration plate, accurately segmenting its boundaries or corners in the point cloud can be difficult due to sparse or noisy point clouds. Second, there are many manual interaction steps, and the degree of automation and calibration efficiency need to be improved. Summary of the Invention

[0005] In response to the above-mentioned problems existing in the prior art, the main purpose of the present invention is to provide a calibration method for the extrinsic parameters of a lidar plus camera system suitable for applications such as robotics, autonomous driving or three-dimensional reconstruction, aiming to improve the robustness and automation of three-dimensional feature extraction of the calibration plate, thereby obtaining accurate extrinsic parameters.

[0006] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0007] A method for calibrating the extrinsic parameters of a laser radar plus camera system, which specifically includes:

[0008] S1. Prepare the calibration device and data acquisition environment; prepare a circular calibration plate, select a material that ensures good flatness of the calibration plate, and place a machine-readable two-dimensional code pattern at the geometric center of the calibration plate; prepare the lidar and camera, set the time synchronization of the lidar and camera, and obtain the camera's intrinsic parameter matrix and distortion coefficients through standard camera calibration methods;

[0009] S2. Power on the LiDAR and camera system, then point the system toward a circular calibration plate. Simultaneously collect the LiDAR 3D point cloud data and camera image data from the calibration plate at multiple different positions and angles. A set of LiDAR 3D point cloud data and camera image data is stored for each position and angle.

[0010] S3. For each set of collected LiDAR 3D point cloud data, segment the point cloud subset belonging to the calibration plate, calculate the average value of all 3D point coordinates in the point cloud subset, that is, the centroid of the point cloud subset, and use it as the 3D feature point coordinates of the calibration plate;

[0011] S4. For each set of collected camera image data, detect the two-dimensional coded pattern and identify all its corner points. Calculate the image pixel coordinates of the geometric mean center of each corner point as the two-dimensional feature point coordinates of the calibration plate.

[0012] S5. Based on the obtained 3D feature point coordinates and 2D feature point coordinates, establish multiple sets of corresponding 3D-2D feature point pairs, and then use the previously obtained camera intrinsic parameters and distortion coefficients to calculate the initial extrinsic parameters from the lidar coordinate system to the camera coordinate system through the PnP algorithm; the initial extrinsic parameters include initial rotation parameters and translation parameters;

[0013] S6. Use a nonlinear optimization algorithm to minimize the reprojection error between the projection point of the three-dimensional feature point onto the image plane and the corresponding two-dimensional feature point, and optimize the external parameters; stop the optimization until the change in the sum of the squares of the reprojection error and the change in the rotation vector and translation vector are less than a certain threshold, and obtain the final optimized external parameters.

[0014] Furthermore, in step S1, the two-dimensional coding pattern is a QR code of a specific ID.

[0015] Furthermore, step S2 is specifically as follows:

[0016] A circular calibration plate is placed within the field of view of both the lidar and camera. By changing the distance and angle between the calibration plate and the sensor, the lidar and camera are synchronously triggered to collect data at multiple positions and postures. Each set of data obtained contains a camera image and a corresponding lidar 3D point cloud frame.

[0017] Furthermore, step S3 specifically includes:

[0018] S31. Process the three-dimensional point cloud data of the LiDAR. First, perform ROI filtering to exclude most irrelevant points. Then, use the RANSAC algorithm to perform plane extraction on the point cloud data to obtain multiple segmented point cloud subsets. For each segmented point cloud subset, calculate the length, width, and height of its maximum bounding box, and record the length and width as and ; For the length and width of the maximum bounding box and , calculate the difference of the diameter R of the calibration plate respectively. If and The absolute value of the size deviation threshold is less than , then this part of the point cloud subset segmented by the RANSAC algorithm is recorded as the point cloud subset belonging to the calibration plate plane;

[0019] S32, through three-dimensional coordinates The centroid of each point cloud subset is calculated by taking the average value as the center of the calibration plate, that is, the coordinates of the three-dimensional feature points in the point cloud. ; The formula is: ,in, Represents the mean function; extract all the valid point clouds extracted from the center of the calibration plate Store them and get a three-dimensional feature point set.

[0020] Furthermore, step S4 specifically includes:

[0021] S41. Detect the two-dimensional coded pattern using an image processing algorithm and accurately identify the pixel coordinates of all corner points in the two-dimensional coded pattern; obtain the pixel coordinates of the corner points of the two-dimensional coded pattern through image preprocessing, feature detection, pattern recognition, and corner point determination;

[0022] S42. Calculate the average value of pixel coordinates of each corner point of the two-dimensional coding pattern to obtain the center point of the coding pattern; the center point is the two-dimensional feature point corresponding to the image. ; Extract all the valid images of the extracted two-dimensional coded pattern corner pixels Store them and get a two-dimensional feature point set.

[0023] Furthermore, the reprojection error is calculated in step S6 as follows:

[0024] The initial rotation vector obtained in step S5 and the initial translation vector As the initial value of nonlinear optimization, and define the reprojection error function: for each 3D point , using the current rotation vector , translation vector , camera intrinsic parameter matrix and distortion coefficients , it can be projected onto the image plane through the camera perspective projection model to obtain the projection point ,Should The corresponding observed two-dimensional point is ; The reprojection error is expressed as .

[0025] Furthermore, the optimization of external parameters in step S6 is specifically as follows:

[0026] Adjust the rotation vector using the Levenberg-Marquardt algorithm , translation vector , to minimize the total reprojection error; after the optimization is completed, the final rotation vector is obtained and translation vectors ;

[0027] The optimized rotation vector Convert to a 3×3 rotation matrix using the Rodrigues formula ;

[0028] ;

[0029] in, represents the axis angle, is the rotation angle, is the unit rotation axis vector, is the identity matrix, is a vector The antisymmetric matrix of is defined as:

[0030] ;

[0031] The three-axis components of the unit rotation axis vector; yes The square of , calculated by matrix multiplication:

[0032] ;

[0033] Final LiDAR To Camera The external transformation matrix Represented as a 4×4 homogeneous matrix:

[0034] ;

[0035] in, is a 3×3 rotation matrix, is a 3×1 translation vector.

[0036] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0037] 1. Robust feature extraction: The circular planar calibration plate creates relatively distinct planar features in the point cloud, making them easily and reliably extractable using plane extraction algorithms (such as the RANSAC algorithm). The centroid of the points within the plane serves as the 3D center, making it insensitive to point cloud noise and sparsity. Extracting the center of the 2D code pattern in the image is sophisticated and accurate.

[0038] 2. Clear correspondence: The center of the 2D coding pattern is designed to be consistent with the physical center of the circular plane calibration plate. Therefore, the coding center detected in the image naturally corresponds to the center of the circular plane segmented in the point cloud.

[0039] 3. High degree of automation: In the entire process, the feature extraction and matching process can be highly automated, reducing manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 1 is a flow chart of a method for calibrating extrinsic parameters of a lidar plus camera system according to the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a circular calibration plate in an embodiment of the present invention;

[0043] Figure 3 2. Schematic diagram of two-dimensional feature point extraction of a camera image according to an embodiment of the present invention;

[0044] Figure 4 It is a re-projection image of the laser radar three-dimensional point cloud projected onto the image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following Figure 1-4 It should be understood that the specific embodiments described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0046] Although the steps in the present invention are arranged with numbers, they are not intended to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0047] like Figure 1 As shown, the present invention proposes a method for calibrating the extrinsic parameters of a laser radar plus camera system, which specifically includes the following steps:

[0048] S1. Prepare the calibration device and data acquisition environment; prepare a calibration plate with a circular plane as the main body, such as Figure 2 As shown in the figure, a material that ensures good flatness of the calibration plate (for example, acrylic, aluminum, or high-density foam) is selected, and a machine-readable 2D coded pattern is placed at the geometric center of the calibration plate. In this embodiment, the 2D coded pattern is a QR code with a specific ID. The physical dimensions of the 2D coded pattern (for example, side length) must be accurately measured in advance, and the circular radius of the calibration plate must also be known in advance. This is mainly used to assist in determining the rationality of the point cloud segmentation area. The key is to ensure that the 3D feature point is the plane centroid.

[0049] Prepare the lidar and camera, set the time synchronization of the lidar and camera, and obtain the camera's intrinsic parameter matrix through standard camera calibration methods (such as Zhang Zhengyou method) and distortion coefficients (like ).in, Radial distortion is caused by the optical properties of the lens, which causes light to deviate from the ideal path when passing through the lens, resulting in the deformation of the image in the radial direction. It is tangential distortion, which is the deformation of the image caused by the tilt or offset between the lens and the imaging plane.

[0050] S2. Power on the LiDAR and camera system, then point the system toward a circular calibration plate. Simultaneously collect the LiDAR 3D point cloud data and camera image data from the calibration plate at multiple different positions and angles. A set of LiDAR 3D point cloud data and camera image data is stored for each position and angle.

[0051] As a preferred embodiment, step S2 is specifically as follows:

[0052] A circular calibration plate is placed within the field of view of both the LiDAR and camera. By varying the distance and angle between the calibration plate and the sensor (in this application, the term "sensor" refers to the LiDAR and camera, which are rigidly connected as a sensor system), the LiDAR and camera are synchronously triggered to collect data at various positions and postures. Each data set contains a camera image and a corresponding LiDAR 3D point cloud frame. In this embodiment, at least 10-15 sets of valid data are collected.

[0053] S3. For each set of collected LiDAR 3D point cloud data, segment the point cloud subset belonging to the calibration plate, calculate the average value of all 3D point coordinates in the point cloud subset, that is, the centroid of the point cloud subset, and use it as the 3D feature point coordinates of the calibration plate;

[0054] As a preferred embodiment, step S3 specifically includes:

[0055] S31. Process the three-dimensional point cloud data of the LiDAR. First, perform ROI filtering to exclude most irrelevant points. Then, use the RANSAC algorithm to perform plane extraction on the point cloud data to obtain multiple segmented point cloud subsets. For each segmented point cloud subset, calculate the length, width, and height of its maximum bounding box, and record the length and width as and ; For the length and width of the maximum bounding box and , calculate the difference of the diameter R of the calibration plate respectively. If and The absolute value of the size deviation threshold is less than , then the point cloud subset segmented by the RANSAC algorithm is recorded as the point cloud subset belonging to the calibration plate plane; this step obtains the parameters of the plane model and the set of internal points belonging to the plane (i.e. Figure 4 point cloud clusters in );

[0056] S32, through three-dimensional coordinates The centroid of each point cloud subset is calculated by taking the average value as the coordinates of the three-dimensional feature points of the calibration plate in the point cloud. ; The formula is: ,in, Represents the mean function; extract all the valid point clouds extracted from the center of the calibration plate Store them and get a three-dimensional feature point set.

[0057] S4. For each set of collected camera image data, detect the two-dimensional coded pattern and identify all its corner points. Calculate the image pixel coordinates of the geometric mean center of each corner point as the two-dimensional feature point coordinates of the calibration plate.

[0058] As a preferred embodiment, step S4 specifically includes:

[0059] S41. Use image processing algorithms to detect the two-dimensional code pattern and accurately identify the pixel coordinates of all corner points in the two-dimensional code pattern; obtain the pixel coordinates of the corner points of the two-dimensional code pattern through image preprocessing, feature detection, pattern recognition and corner point determination (for example, for a square code, there are usually 4 corner points, such as Figure 3 The 4 yellow dots in ));

[0060] S42, calculate the average value of the pixel coordinates of each corner point of the two-dimensional coding pattern, and obtain the center point of the coding pattern (such as Figure 3 As shown); the center point is the two-dimensional feature point corresponding to the image ; Extract all valid images Store them and get a two-dimensional feature point set.

[0061] S5. Based on the obtained three-dimensional feature point coordinates and two-dimensional feature point coordinates, establish multiple sets of corresponding three-dimensional-two-dimensional feature point pairs , and then use the pre-acquired camera intrinsic parameters and distortion coefficients to calculate the initial extrinsic parameters from the lidar coordinate system to the camera coordinate system through the PnP algorithm; the initial extrinsic parameters include initial rotation parameters and translation parameters, which are used to represent the transformation from the lidar coordinate system to the camera coordinate system.

[0062] S6. Using a nonlinear optimization algorithm, minimize the reprojection error between the projection point of the three-dimensional feature point onto the image plane and the corresponding two-dimensional feature point, and optimize the extrinsic parameters. Optimize until the change in the sum of squares of the reprojection error, the change in the rotation vector, and the change in the translation vector are less than a certain threshold (set to 1e-8 in this embodiment), and finally obtain the optimized extrinsic parameters.

[0063] As a preferred embodiment, the reprojection error is calculated in step S6 as follows:

[0064] The initial rotation vector obtained in step S5 and the initial translation vector As the initial value of nonlinear optimization, and define the reprojection error function: for each 3D point , using the current rotation vector , translation vector , camera intrinsic parameter matrix and distortion coefficients , it can be projected onto the image plane through the camera perspective projection model to obtain the projection point ,Should The corresponding observed two-dimensional point is ; The reprojection error is expressed as .

[0065] More specifically, the optimization of external parameters in step S6 is as follows:

[0066] Adjust the rotation vector using the Levenberg-Marquardt algorithm , translation vector , to minimize the total reprojection error; after the optimization is completed, the final rotation vector is obtained and translation vectors ;

[0067] The optimized rotation vector Convert to a 3×3 rotation matrix using the Rodrigues formula ;

[0068] ;

[0069] in, represents the axis angle, is the rotation angle, is the unit rotation axis vector, is the identity matrix, is a vector The antisymmetric matrix of is defined as:

[0070] ;

[0071] The three-axis components of the unit rotation axis vector; yes The square of , calculated by matrix multiplication:

[0072] ;

[0073] Final LiDAR To Camera The external transformation matrix Represented as a 4×4 homogeneous matrix:

[0074] ;

[0075] in, is a 3×3 rotation matrix, is a 3×1 translation vector.

[0076] Finally, the external parameters of the laser radar and camera are obtained through the method of the present invention. After that, the point cloud is reprojected onto the image. Figure 4 As shown in the figure, it can be seen from the results that the point cloud reprojection to the image has a better effect.

[0077] In summary, the present invention provides a method for calibrating the extrinsic parameters of a lidar and camera system suitable for applications such as robotics, autonomous driving, or three-dimensional reconstruction. It can improve the robustness and automation of three-dimensional feature extraction of a calibration plate, thereby obtaining accurate extrinsic parameters and effectively realizing the calibration of the extrinsic parameters of the lidar and camera.

[0078] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0079] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0080] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for calibrating the external parameters of a laser radar plus camera system, characterized in that: The specific steps include: S1. Prepare the calibration device and data acquisition environment; prepare a circular calibration plate, select a material that ensures good flatness of the calibration plate, and place a machine-readable two-dimensional code pattern at the geometric center of the calibration plate; prepare the lidar and camera, set the time synchronization of the lidar and camera, and obtain the camera's intrinsic parameter matrix and distortion coefficients through standard camera calibration methods; S2. Power on the LiDAR and camera system, then point the system toward a circular calibration plate. Simultaneously collect the LiDAR 3D point cloud data and camera image data from the calibration plate at multiple different positions and angles. A set of LiDAR 3D point cloud data and camera image data is stored for each position and angle. S3. For each set of collected LiDAR 3D point cloud data, segment the point cloud subset belonging to the calibration plate, calculate the average value of all 3D point coordinates in the point cloud subset, that is, the centroid of the point cloud subset, and use it as the 3D feature point coordinates of the calibration plate; S4. For each set of collected camera image data, detect the two-dimensional coded pattern and identify all its corner points. Calculate the image pixel coordinates of the geometric mean center of each corner point as the two-dimensional feature point coordinates of the calibration plate. S5. Based on the obtained 3D feature point coordinates and 2D feature point coordinates, establish multiple sets of corresponding 3D-2D feature point pairs, and then use the previously obtained camera intrinsic parameters and distortion coefficients to calculate the initial extrinsic parameters from the lidar coordinate system to the camera coordinate system through the PnP algorithm; the initial extrinsic parameters include initial rotation parameters and translation parameters; S6. Use a nonlinear optimization algorithm to minimize the reprojection error between the projection point of the three-dimensional feature point onto the image plane and the corresponding two-dimensional feature point, and optimize the external parameters; stop the optimization until the change in the sum of the squares of the reprojection error and the change in the rotation vector and translation vector are less than a certain threshold, and obtain the final optimized external parameters.

2. The method for calibrating the external parameters of a laser radar plus camera system according to claim 1, characterized in that: In step S1, the two-dimensional coding pattern is a QR code of a specific ID.

3. The method for calibrating the external parameters of a laser radar plus camera system according to claim 1, characterized in that: Step S2 is specifically as follows: A circular calibration plate is placed within the field of view of both the lidar and camera. By changing the distance and angle between the calibration plate and the sensor, the lidar and camera are synchronously triggered to collect data at multiple positions and postures. Each set of data obtained contains a camera image and a corresponding lidar 3D point cloud frame.

4. The method for calibrating the extrinsic parameters of a laser radar plus camera system according to claim 1, wherein: Step S3 specifically includes: S31. Process the three-dimensional point cloud data of the LiDAR. First, perform ROI filtering to exclude most irrelevant points. Then, use the RANSAC algorithm to perform plane extraction on the point cloud data to obtain multiple segmented point cloud subsets. For each segmented point cloud subset, calculate the length, width, and height of its maximum bounding box, and record the length and width as and ; For the length and width of the maximum bounding box and , calculate the difference of the diameter R of the calibration plate respectively. If and The absolute value of the size deviation threshold is less than , then this part of the point cloud subset segmented by the RANSAC algorithm is recorded as the point cloud subset belonging to the calibration plate plane; S32, through three-dimensional coordinates The centroid of each point cloud subset is calculated by taking the average value as the center of the calibration plate, that is, the coordinates of the three-dimensional feature points in the point cloud. ; The formula is: ,in, Represents the mean function; extract all the valid point clouds extracted from the center of the calibration plate Store them and get a three-dimensional feature point set.

5. The method for calibrating the external parameters of a laser radar plus camera system according to claim 1, wherein: Step S4 specifically includes: S41. Detect the two-dimensional coded pattern using an image processing algorithm and accurately identify the pixel coordinates of all corner points in the two-dimensional coded pattern; obtain the pixel coordinates of the corner points of the two-dimensional coded pattern through image preprocessing, feature detection, pattern recognition, and corner point determination; S42. Calculate the average value of pixel coordinates of each corner point of the two-dimensional coding pattern to obtain the center point of the coding pattern; the center point is the two-dimensional feature point corresponding to the image. ; Extract all the valid images of the extracted two-dimensional coded pattern corner pixels Store them and get a two-dimensional feature point set.

6. The method for calibrating extrinsic parameters of a laser radar plus camera system according to claim 1, characterized in that: The reprojection error is calculated in step S6 as follows: The initial rotation vector obtained in step S5 and the initial translation vector As the initial value of nonlinear optimization, and define the reprojection error function: for each 3D point , using the current rotation vector , translation vector , camera intrinsic parameter matrix and distortion coefficients , it can be projected onto the image plane through the camera perspective projection model to obtain the projection point ,Should The corresponding observed two-dimensional point is ; The reprojection error is expressed as .

7. The method for calibrating extrinsic parameters of a laser radar plus camera system according to claim 6, characterized in that: The optimization of external parameters in step S6 is specifically as follows: Adjust the rotation vector using the Levenberg-Marquardt algorithm , translation vector , to minimize the total reprojection error; after the optimization is completed, the final rotation vector is obtained and translation vectors ; The optimized rotation vector Convert to a 3×3 rotation matrix using the Rodrigues formula ; ； in, represents the axis angle, is the rotation angle, is the unit rotation axis vector, is the identity matrix, is a vector The antisymmetric matrix of is defined as: ； The three-axis components of the unit rotation axis vector; yes The square of , calculated by matrix multiplication: ； Final LiDAR To Camera The external transformation matrix Represented as a 4×4 homogeneous matrix: ； in, is a 3×3 rotation matrix, is a 3×1 translation vector.

Images