Calibration board and laser radar and camera joint calibration method based on the calibration board

Abstract

This invention discloses a calibration board and a joint calibration method for lidar and camera based on the calibration board, relating to the field of multi-sensor calibration technology. The calibration board has a visual calibration pattern and multiple rectangular perforations of the same size, independent and non-connected, located on the top, right, bottom, and left sides of the calibration board and arranged symmetrically with respect to the geometric center of the calibration board. The upper perforations are arranged horizontally, with their left side horizontally from the left edge of the calibration board and their upper side vertically from the top edge of the calibration board being [missing information]. L 2; The perforations on the right side are arranged vertically, and the horizontal distance from the right edge of the calibration plate to the right edge is [missing information]. L 2. The upper side edge is flush with the upper side edge of the upper perforation; the lower perforation is arranged horizontally, and the horizontal distance from its right side edge to the right edge of the calibration plate and the vertical distance from its lower side edge to the lower edge of the calibration plate are both... L 2; The perforations on the left side are arranged vertically, and the horizontal distance from the left edge of the calibration plate to the left edge is [missing information]. L 2. The lower edge is flush with the lower edge of the lower perforation.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor calibration technology, specifically to a calibration board and a method for joint calibration of lidar and camera based on the calibration board. Background Technology

[0002] LiDAR (Light Detection and Ranging) is an active sensor that can directly acquire the geometric structure and depth information of the surrounding environment through generated point cloud data, offering advantages such as high measurement accuracy and long ranging range. However, LiDAR cannot provide rich color and texture information, making it difficult to achieve a comprehensive semantic understanding of the environment relying solely on point cloud data. Cameras, as visual sensors, can acquire images to obtain rich information such as color and texture of the environment, and image-based target recognition, semantic segmentation, and scene understanding technologies are relatively mature. However, since image data does not contain depth information, it is difficult to obtain accurate 3D information. Therefore, by fusing heterogeneous data from LiDAR point clouds and image information, the perceptual features of different sensors can be complemented, effectively compensating for the limitations of a single sensor in environmental perception, thereby improving the overall perception accuracy and robustness of the system.

[0003] The key to heterogeneous data fusion is the accurate calibration between multiple sensors, which requires establishing the correspondence between LiDAR point clouds and image pixels. Assuming... It is the camera coordinate system. This refers to the lidar coordinate system. Assume there is a point P in the world coordinate system, which is a three-dimensional point P in the lidar coordinate system. L =(x L ,y L ,z L ) T Its coordinates in the camera coordinate system are P. C =(x c ,y c ,z c ) T The so-called calibration is to establish P C With P L The correspondence between them, that is, to give the representation P of a certain spatial point in the world coordinate system in the lidar coordinate system. L The calibration results allow us to find the corresponding P in the pixel coordinate system. C Current mainstream calibration methods typically rely on calibration boards with distinct structural features to extract co-located feature points or feature surfaces from the LiDAR point cloud and the image, establishing coordinate transformation relationships. However, existing methods face the following problems in practical applications: Existing calibration boards have limited geometric feature types, making it difficult to meet the requirements for feature observability in complex environments and under multi-view conditions. Especially in environments with large viewpoint deviations or non-open spaces, it is difficult to ensure that features can be effectively observed and extracted in both point clouds and images, resulting in insufficient alignment between the two source data and affecting registration accuracy. At the same time, some calibration boards with complex structures have the problems of high manufacturing precision and high cost, which limits their large-scale deployment in engineering.

[0004] Existing methods are typically optimized based on the premise that the geometric features of the calibration board can be accurately extracted. However, in real-world scenarios with sparse point clouds, complex backgrounds, or significant occlusion interference, effective features are easily lost, leading to calibration failure or increased errors. When the calibration board's viewpoint is limited or partially occluded, existing methods often cannot construct stable geometric constraints, thus limiting their applicability.

[0005] Most existing methods rely on local features such as sparse edges and corners in point clouds. In the presence of point cloud noise, missing target structures, or low laser reflectivity, feature extraction errors or registration failures are likely to occur. They lack sufficient feature redundancy and structural diversity to enhance the robustness of registration constraints, and are particularly unstable in long-distance or low-density point cloud scenarios.

[0006] Most existing methods require multi-view or multi-frame observation data to establish a complete registration relationship. In situations where space is limited or operational conditions are complex, this not only increases operational complexity but also increases the cost of manual intervention and reduces system deployment efficiency.

[0007] Most existing methods require first calibrating camera intrinsic parameters and completing image distortion correction, and then establishing the correspondence between the image and the point cloud through coordinate transformation. This process relies heavily on prior conditions and has a long error propagation chain. Summary of the Invention

[0008] In view of the shortcomings of the prior art, the present invention provides a calibration board and a joint calibration method for lidar and camera based on the calibration board.

[0009] The technical solution of this invention is: A calibration plate having a visual calibration pattern for visual recognition, and also having a plurality of rectangular perforations for LiDAR recognition.

[0010] Furthermore, according to the calibration plate, the rectangular perforations are of the same size and are independent and non-connected to each other, and are located on the top, right, bottom and left sides of the calibration plate, forming a centrally symmetrical arrangement with respect to the geometric center of the calibration plate.

[0011] Furthermore, according to the calibration plate, the upper perforations are arranged horizontally, close to the upper edge of the calibration plate, and the horizontal distance between the left side of the perforations and the left edge of the calibration plate is [missing information]. L2. The vertical distance between the upper edge and the upper edge of the calibration plate is... L 2; The perforations on the right side are arranged vertically, and the horizontal distance from the right edge of the perforation to the right edge of the calibration plate is: L 2. The upper side edge is flush with the upper side edge of the upper perforation; the lower perforation is arranged horizontally, close to the lower edge of the calibration plate, and the horizontal distance between its right side edge and the right edge of the calibration plate is [missing information]. L 2. The vertical distance between the lower edge and the lower edge of the calibration plate is... L 2; The perforations on the left side are arranged vertically, and the horizontal distance from the left edge of the left side to the left edge of the calibration plate is... L 2. The lower edge is flush with the lower edge of the lower perforation.

[0012] Furthermore, according to the calibration plate, the visual calibration pattern is set in the upper left region of the calibration plate, the upper side of the visual calibration pattern region is flush with the lower side of the upper perforation, and the left side is aligned with the right side of the left perforation; from the camera's perspective, the complete calibration plate image has neither axial symmetry nor central symmetry.

[0013] Furthermore, according to the calibration plate, the side length of the calibration plate is 11. L 2; The width of the rectangular perforation is L 2. Length is 7 L 2; The visual calibration pattern occupies an area of ​​5. L 2×6 L 2.

[0014] A joint calibration method for lidar and camera based on the above calibration board includes the following steps: Step 1: Rigidly connect the lidar and the camera to lock their relative position and orientation; adjust the orientation of the lidar and the camera so that their perceived fields of view overlap. Step 2: Place the calibration board in the common field of view of both the LiDAR and the camera. Use the LiDAR to collect point cloud data containing the entire calibration board, and use the camera to collect images of the calibration board containing the visual calibration pattern. During the acquisition process, change the position or orientation of the calibration board multiple times, and at each position or orientation of the calibration board, simultaneously collect multiple sets of LiDAR point cloud data and camera image data pairs. Step 3: Generate a relatively dense and structurally complete target point cloud based on the point cloud collected in Step 2; Step 4: Construct a local neighborhood of the point cloud with a point P in the target point cloud as the center, and use principal component analysis to extract the geometric features of each neighborhood. Then, calculate the entropy function value of each neighborhood based on its geometric features. Finally, select the neighborhood corresponding to the minimum entropy value as the optimal neighborhood to obtain the initial calibration board planar point cloud. Step 5: Use the angle relationship between the normal vectors of adjacent points in the initial calibration board planar point cloud to determine the geometric continuity of the planar point set formed by the initial calibration board planar point cloud and identify misclassified points; then use the points that have been confirmed to be continuous as seed points, and perform a region growing algorithm in the initial calibration board planar point cloud to remove misclassified points and obtain the geometric features of the calibration board point cloud; Step 6: Using the calibration board image containing the visual calibration pattern acquired in Step 2 as the input image, identify the pose of the calibration board in the image, extract its significant edge contours and the two-dimensional projection coordinates of key points in the image, and obtain the geometric features of the calibration board image. Step 7: Based on the known physical geometric relationships of the calibration board, optimize and register the geometric features of the calibration board image extracted in Step 6 with the geometric features of the calibration board point cloud obtained in Step 5, and construct a nonlinear cost equation; use the Levenberg-Marquardt algorithm to achieve least-squares optimization of the nonlinear cost equation, and solve for the optimal extrinsic parameter matrices of the LiDAR and camera. Including rotation matrix With translation vector The joint calibration of lidar and camera was completed.

[0015] Furthermore, according to the aforementioned joint calibration method for lidar and camera, the method for generating a relatively dense and structurally complete target point cloud based on the point cloud acquired in step 2 in step 3 is as follows: Select multiple consecutive frames of point cloud data acquired in step 2, and filter each frame of point cloud according to a preset Euclidean distance threshold, removing points whose Euclidean distance exceeds the threshold; take the pose of the first frame of point cloud after filtering as the reference pose; solve the spatial transformation matrix from the other frames of point cloud to the reference pose; based on the spatial transformation matrix, register the other frames of point cloud to the reference pose; after registration, merge the multiple frames of point cloud to generate a relatively dense and structurally complete target point cloud.

[0016] Furthermore, according to the aforementioned joint calibration method for lidar and camera, a spherical radius search is used in step 4 to construct the local neighborhood; during the search process, principal component analysis is used to analyze the geometric features of the local neighborhood, and the entropy function value of the neighborhood under the corresponding search radius is calculated based on the geometric features.

[0017] Furthermore, according to the aforementioned joint calibration method for lidar and camera, step 7 specifically includes the following steps: Step 7.1: Perform back projection on the edge pixels of the calibration board contour extracted in Step 6, and perform optimal fitting on the point cloud features of the calibration board using principal component analysis. Step 7.2: Project the optimal cloud edge line features fitted in Step 7.1 onto the visual image, construct a cost function with Intersection over Union (IoU) as the metric, optimize the edge pixels used for registration with geometric feature points in the point cloud, establish image geometric features that match the point features and line features in the point cloud obtained in Step 7.1, and form the optimal geometric feature pair with the point features and line features obtained in Step 7.1; Step 7.3: Based on the optimal geometric feature pairs of the calibration board in the point cloud and image obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, select a set of optimal edges, and select a set from the optimal geometric feature pairs based on the optimal edges to construct the constraint equation and coarsely calculate the extrinsic parameter matrix; Step 7.4: Based on the coarsely estimated rotation matrix and coarsely estimated displacement vector obtained in Step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and image. This leads to the further construction of a cost function minimizing the cost based on Euclidean distance, and the Levenberg-Marquardt algorithm is used to achieve least-squares optimization, accurately solving for the extrinsic parameter matrix between the LiDAR and the camera—the rotation matrix. Translation vector .

[0018] Furthermore, according to the aforementioned joint calibration method of lidar and camera, the method for selecting a set of optimal edges is as follows: based on the three sets of edges—the outer edge of the calibration plate, the outer edge of the perforated area, and the inner edge of the perforated area—the optimal edge set is dynamically selected according to the observed edge line length and its continuity.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The calibration board of this invention features a rich geometric structure, significantly enhancing feature extraction and registration capabilities during the calibration process. Its specially designed asymmetric geometry effectively avoids axisymmetric interference, ensuring the unique determinism of the coordinate system. Furthermore, the calibration board integrates various geometric elements such as edges, corners, holes, lines, and surfaces, enabling the observation and extraction of key features from both images and point clouds. This not only improves the diversity of matching features but also increases feature density, providing strong support for high-precision calibration.

[0020] This invention simplifies the feature extraction process and improves registration stability by providing redundant features. Geometric constraints are constructed using planar and line features, enhancing the stability of feature registration and effectively solving the problem of rigid transformations between sensor coordinate systems.

[0021] (3) Achieve adaptive calibration constraints and improve applicability. By capturing multiple geometric features in point clouds and images, the calibration task can still be completed even when only part of the calibration board is visible, which enhances the applicability of the method, expands the application range of the calibration board, reduces manual adjustments, and improves overall efficiency.

[0022] (4) Based on the high-precision mapping model of point cloud and image pixels, this invention introduces the intersection-over-union (IoU) alignment strategy and combines entropy calculation and geometric fitting methods to enable the calibration process to be unrestricted by scene complexity. Even in non-open environments, when the calibration board is only partially visible, or when there are complex backgrounds or interfering objects, it can still stably establish high-quality constraint relationships, ensuring the robustness and accuracy of the calibration process and significantly expanding the applicability of the calibration board.

[0023] (5) Supports calibration of single observation data, simplifies the process, and reduces manual operation. This invention can extract sufficient line and surface features and construct complete constraint equations through geometric feature extraction in a single observation to complete the calibration task, avoids dependence on multi-frame or multi-view data, significantly simplifies the calibration process, reduces the frequency of manual adjustment, and improves overall efficiency and ease of operation.

[0024] (6) The present invention does not require prior acquisition of camera intrinsic parameters during joint calibration, reduces pre-requisites, lowers the threshold for system integration and use, and improves the versatility and engineering operability of the method.

[0025] (7) This invention significantly improves the accuracy and robustness of initial matching through a highly stable and discriminative geometric feature extraction method. During the feature registration stage, it effectively suppresses outlier interference and reduces redundant iterations. While ensuring calibration accuracy, it shortens calibration time, making it particularly suitable for scenarios with high real-time requirements. In actual deployment, it demonstrates good efficiency and usability. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the calibration plate in this embodiment; Figure 2 This is a flowchart illustrating the joint calibration method of lidar and camera in this embodiment; Figure 3 This is a schematic diagram of the observation during the calibration process in this embodiment; The following are explanations of the labels in the attached diagram: 1—Substrate, 21—Upper perforation area, 22—Right perforation area, 23—Lower perforation area, 24—Left perforation area, 3—Checkerboard pattern area, 1a-1d—Outer edge of calibration board, 2a-2d—Outer edge of perforation area, 3a-3d—Inner edge of perforation area. Detailed Implementation

[0027] To facilitate understanding of this application, a more comprehensive description of this application will be provided below with reference to the accompanying drawings.

[0028] like Figure 1As shown, this embodiment provides a calibration board for joint calibration of multi-line LiDAR and camera. The calibration board includes a rectangular substrate 1 with a certain thickness. The rectangular substrate has an upper surface and a lower surface. The thickness of the calibration board is generally controlled to be around 3mm-10mm, preferably 5mm, to balance structural rigidity and ease of installation on various platforms. The side length of the calibration board is L = 1100 mm. Two types of calibration feature areas are integrated on the substrate: rectangular perforated areas 21-24 for LiDAR identification and checkerboard pattern areas 3 for visual identification.

[0029] In this embodiment, the rectangular perforated areas of the calibration plate are located around the perimeter of the substrate, specifically on the top, right, bottom, and left sides of the calibration plate. These four rectangular perforated areas (21, 22, 23, 24) are independent and not connected. All four perforated areas are identical in size, being rectangles with a length of 700 mm and a width of 100 mm. Figure 1 As shown, the black portion represents the substrate, and the grid lines indicate rectangular perforation areas, specifically including: The upper perforation area 21 is arranged horizontally and is close to the upper edge of the substrate. The horizontal distance from its left side to the left edge of the substrate is 100 mm, and the vertical distance from its upper side to the upper edge of the substrate is 100 mm. The right perforation area 22 is arranged vertically, with its right side horizontally 100 mm from the right edge of the substrate, and its upper side is flush with the upper side of the upper perforation area 21 (i.e. at the same horizontal height).

[0030] The lower perforation area 23 is arranged horizontally and is close to the lower edge of the substrate. The horizontal distance from the right edge of the perforation area to the right edge of the substrate is 100 mm, and the vertical distance from the lower edge of the perforation area to the lower edge of the substrate is 100 mm. The perforation area 24 on the left side is arranged vertically. The horizontal distance between its left side edge and the left edge of the substrate is 100 mm, and its lower side edge is flush with the lower side edge of the lower perforation area 23.

[0031] All the perforated areas described above are rectangular through-hole structures, preferably manufactured using laser cutting or CNC machining methods. The hole edges are straight, the dimensions are precise, and the hole thickness is consistent with the thickness of substrate 1. The four perforated areas are arranged in a centrally symmetrical manner with respect to the geometric center of the substrate, which facilitates the formation of clear geometric discontinuities such as boundaries, line segments, and corners in the point cloud, which are used for the extraction and fitting of edges or corners in subsequent LiDAR data.

[0032] The upper left region of the substrate 1 has a checkerboard pattern area 3. The upper edge of the checkerboard pattern area 3 is flush with the lower edge of the perforated area 21, and the left edge is aligned with the right edge of the perforated area 24. The checkerboard pattern uses a 5×6 square array, with a total of 30 black and white squares. Each square has a side length of 100mm, and the overall area is 500mm×600mm. The checkerboard pattern is made of matte black and white material spraying, which improves image contrast and adapts to different lighting conditions, ensuring that image features are clear and identifiable.

[0033] It should be noted that the checkerboard pattern can also be replaced with other types of visual labeling patterns, such as ArUco codes, AprilTag tags, or regular dot arrays, depending on the specific visual labeling requirements, thus having good versatility and scalability.

[0034] To improve calibration accuracy and the uniqueness of calibration board identification, the calibration board is intentionally designed to eliminate axial symmetry and central symmetry from the camera's viewpoint, constructing a unique feature structure relationship. This structure facilitates the identification and recovery of the calibration board's pose information in images or point clouds, and is particularly suitable for pose estimation and reconstruction under partial visibility conditions. For example, by jointly identifying the corner points of the visual calibration pattern region and the edge corner points of the perforated region, the center position and normal direction of the calibration board in three-dimensional space can be accurately determined.

[0035] The calibration plate can be mounted on a vertical surface or a support platform with a certain angle via fixing holes, hook structure, or tripod clips on the back, making it suitable for multi-sensor joint calibration tasks in various indoor and outdoor environments. The camera and lidar can simultaneously observe the calibration plate, thereby extracting corresponding features from the image and point cloud for joint calibration calculations.

[0036] Based on the aforementioned customized calibration board, this invention proposes a high-efficiency and high-precision joint calibration method for multi-line lidar and camera.

[0037] Assume the rigid transformation from the lidar coordinate system to the camera coordinate system is denoted as... Its rotation matrix is The translation vector is ,in It is a rotation group in three-dimensional space. This is a three-dimensional vector space. For a point P in the camera coordinate system... C Its coordinates in the lidar point cloud are P L The conversion relationship between the two can be expressed as equation (1).

[0038]

[0039] The ultimate goal of the high-efficiency and high-precision joint calibration method for multi-line lidar and camera is to obtain the extrinsic parameters. as well as This enables joint calibration of lidar and camera, such as... Figure 2 The specific steps are as follows: Step 1: Based on the extrinsic parameter calibration requirements between the lidar and the camera, the lidar and the camera are rigidly connected together to form a rigid body structure, providing a stable spatial reference frame for subsequent joint calibration.

[0040] In this step, the lidar and camera are rigidly fixed to the same mounting base, locking their relative position and orientation; the orientation of the lidar and camera is adjusted so that their perceived fields of view overlap, and this overlap corresponds to more than 50% of the effective imaging area within the camera's field of view.

[0041] Step 2: Based on the calibration data acquisition requirements, the calibration board proposed in this embodiment of the invention is placed in the common field of view of both the LiDAR and the camera. The LiDAR is used to acquire point cloud data containing the calibration board, and the camera is used to acquire the corresponding image, providing the raw input for subsequent point cloud and image feature matching.

[0042] In this step, the calibration plate is fixed at a position that can be observed by both the lidar and the camera, and at an appropriate angle, so that each edge of the rectangular outer edge of the calibration plate is crossed by at least three lidar scanning lines, thereby ensuring that the edge structure of the calibration plate can be clearly reflected in the lidar point cloud data; at the same time, the camera should be able to clearly image the visual calibration pattern on the calibration plate to avoid image blurring affecting subsequent feature extraction.

[0043] At each calibration board pose, no fewer than 50 pairs of LiDAR point cloud data and camera image data are simultaneously acquired. To enhance the diversity and spatial coverage of the calibration data, after completing data acquisition at one location, the pose of the calibration board is changed, and the above acquisition process is repeated.

[0044] Step 3: Preprocess the point cloud collected in Step 2, and obtain the target point cloud by fusing multiple frames of point cloud.

[0045] This implementation preprocesses the point cloud obtained in step 2 by merging multiple point clouds to obtain a dense and complete target point cloud, thus solving the problem of incomplete observation information caused by sparse point clouds. In this implementation, three consecutive frames of point cloud data are selected, and each frame is filtered according to a preset Euclidean distance threshold to remove points whose Euclidean distance exceeds the threshold (i.e., outliers, usually points too far from the sensor). The pose of the first frame of the filtered point cloud is used as the reference pose. The spatial transformation matrices from the second and third frame point clouds to the reference pose are solved using the Iterative Closest Point (ICP) algorithm or the Normal Distributions Transform (NDT) algorithm, respectively. Based on the spatial transformation matrices, the second and third frame point clouds are registered to the reference pose. After registration, the three frames are merged to generate a relatively dense and structurally complete target point cloud.

[0046] Step 4: By constructing local neighborhoods and extracting geometric features using principal component analysis, and with the help of adaptive neighborhood partitioning guided by the entropy function, accurate segmentation of the local structure of the point cloud is achieved.

[0047] Based on the target point cloud obtained in step 3, a local neighborhood is first constructed with the query point as the center. Principal component analysis (PCA) is used to extract the eigenvalues ​​of the local covariance matrix, and the dimensional and geometric features of the point are calculated accordingly. Subsequently, within the overall target point cloud, the entropy function values ​​of neighborhoods at different scales are calculated in combination with the local dimensional features. The neighborhood corresponding to the minimum entropy value is selected as the optimal neighborhood, realizing the adaptive partitioning of the optimal neighborhood and obtaining the initial calibration board planar point cloud.

[0048] In this implementation step, a local neighborhood of the point cloud is constructed centered on a point P in the target point cloud. This local neighborhood refers to a set of neighboring points surrounding point P. The neighborhood construction is performed using a spherical radius search, assuming an initial search radius lower limit of... The upper limit is The search change step size is Then the first The search radius is During the search process, Principal Component Analysis (PCA) is used to analyze the dimensional features and entropy function of the local neighborhood. The entropy function measures the uncertainty or information content of the local neighborhood information in the point cloud. The specific implementation of the entire process of determining the optimal neighborhood based on the analysis of local dimensional features using PCA is as follows: For the local neighborhood of a spatial point P in the target point cloud, its covariance matrix can be represented as a symmetric positive definite matrix C. Then, by eigenvalue decomposition of matrix C using equation (2), three eigenvalues ​​can be obtained. Let the eigenvalues ​​be... ,in and These represent the principal eigenvalues ​​of the local neighborhood of the corresponding point cloud in the plane, and the corresponding eigenvectors are expressed as follows: , This represents the smallest eigenvalue, and the corresponding eigenvector is represented as: It is approximately the direction of the normal vector of point P.

[0049]

[0050] After obtaining the feature values, the point cloud dimension features are calculated according to Equation (3) to determine the geometric features (linear, planar, curved) of the point cloud.

[0051]

[0052] In the formula, Represents a straight line; Represents a plane; Represents a curved surface.

[0053] To determine the optimal neighborhood for point cloud segmentation, the entropy function value under the corresponding search radius is calculated according to equation (4). The optimal neighborhood is the one with the minimum entropy value. This allows us to obtain the initial calibration board point cloud.

[0054] Step 5: Analyze the geometric continuity of the planar point set formed by the initial calibration board planar point cloud by using the angle relationship between the normal vectors of adjacent points in the initial calibration board planar point cloud. Identify misclassified points caused by surface disturbance and edge curvature of the calibration board. Then, use the region growing algorithm to segment the initial calibration board planar point cloud to remove misclassified points. Finally, accurately describe the spatial pose of the calibration board in the lidar coordinate system through parametric modeling.

[0055] In actual data acquisition, if a calibration board support is not used, the calibration board may exhibit slight warping, occlusion, or reflection in its natural placement, resulting in small-area three-dimensional curved surfaces at its edges. These areas are easily misclassified as planes in traditional distance or angle threshold-based plane discrimination methods, affecting calibration accuracy. To improve the recognition capability of geometric primitives, this embodiment proposes a plane extraction method combining Spatial Feature Analysis (SFA) and region growing. This method effectively improves the robustness of calibration board planar geometry extraction in sparse, noisy point cloud environments, and is particularly suitable for complex indoor or outdoor scenes.

[0056] In this step, the planar point set formed by the initial calibration board planar point cloud is refined in the target point cloud. For points on the same horizontal line, adjacent points are calculated according to equation (5). and Angle between the normal vectors If the angle changes significantly and exceeds a preset angle threshold, the point is considered invalid. With point Discontinuous, not on the same plane, determination point This is a misclassification point.

[0057]

[0058] In the formula, Points and The normal vector.

[0059] The points that are confirmed to be continuous by the analysis of the angle between the normal vectors and belong to the calibration plate plane are used as seed points. The region growth algorithm is executed in the initial calibration plate plane point cloud to segment the calibration plate plane point cloud and remove misclassified points. The calibration plate plane model can be parameterized as Equation (6).

[0060] In the formula, C is the center of the calibration plate in the lidar coordinate system; It is the unit vector along the y-axis in the lidar coordinate system; n b It is the normal vector of the calibration plate plane; α It is a vector With the calibration plate axis X b The angle between them; the three-dimensional coordinates of point C in the lidar coordinate system are ;n b The expression is as shown in equation (7).

[0061] In the formula, and They represent exist and The directional component.

[0062] Step 6: Using the calibration board image containing the visual calibration pattern acquired in Step 2 as the input image, an image processing algorithm is used to identify the pose of the calibration board in the image, and extract its salient edge contours and the projection coordinates of key points in the image, i.e., the geometric features of the calibration board image. This is used for solving the extrinsic parameter calibration between the LiDAR and the camera in the subsequent process.

[0063] In this implementation, the calibration board image containing the visual calibration pattern obtained in step 2 is used as the input image. Target matching is performed by detecting the calibration board and the visual calibration pattern (checkerboard or other markings) in the image to determine the two-dimensional position of the calibration board in the visual image. Within the extracted calibration board area, the Canny edge detection algorithm is used to extract geometric features, i.e., extracting its contour edge pixels. Based on the predefined corner positions or other key points (such as center points, intersection points, etc.) in the calibration board's geometric structure, its two-dimensional projection coordinates in the visual image are calculated. A KD tree data structure is used to store the spatial information of the edge pixels in the image, which serves as the geometric features of the calibration board image. This facilitates subsequent matching and optimization processing with point cloud edge features.

[0064] Step 7: Optimize and register the geometric features of the calibration board image extracted in Step 6 with the geometric features of the calibration board point cloud obtained in Step 5, constructing a nonlinear cost equation. The Levenberg-Marquardt algorithm is used to perform least-squares optimization of the cost equation, solving for the optimal extrinsic parameter matrices of the LiDAR and camera. Including rotation matrix With translation vector The joint calibration of lidar and camera was completed.

[0065] Step 7.1: Based on the calibration board contour edge pixels extracted in Step 6, perform back projection, and use principal component analysis to optimally fit the point cloud line features of the calibration board.

[0066] In this implementation step, based on the set of calibration board contour edge pixels extracted in step 6, for each edge pixel, according to the manually measured initial extrinsic parameter matrix... Back-projected onto the lidar coordinate system, corresponding points are selected as candidate points from the calibration board point cloud obtained in step 5. A search is then performed on these candidate points. The nearest neighbors form a local point set. .

[0067] For each local point set T i Principal component analysis (PCA) was used to fit the optimal line characteristics, resulting in the fitted straight line shown in equation (8). .Wire Points on As shown in equation (9).

[0068]

[0069]

[0070] Step 7.2: Project the optimal cloud edge features fitted in Step 7.1 onto the visual image according to the projection model, construct a cost function with intersection-over-union (IoU) as the metric, optimize the edge pixels used for registration with geometric feature points in the point cloud, and improve the registration accuracy between the image and the geometric features in the point cloud.

[0071] In this implementation step, the optimal cloud edge features fitted in step 7.1 are projected onto the visual image plane. Combining the imaging principles of LiDAR and camera, the initial estimated extrinsic parameter matrix is ​​obtained through manual measurement. Using the projection model shown in Equation (10), the point cloud plane is mapped to the image plane to obtain the correspondence between spatial points and image edges.

[0072] In the formula, This represents a point in a 3D point cloud in the lidar coordinate system. This indicates the actual coordinates of the point in the image after projection. It is a projection model that takes into account camera distortion.

[0073] The projection points of the optimal cloud edge features onto the visual image plane are optimized using the intersection-over-union (IoU) cost function to improve the geometric accuracy for registration. The IoU cost function is defined as Equation (11). In the formula, The calibration plate area is calculated using image edge features; It is the projected area of ​​the calibration board point cloud onto the image; It is the area of ​​the polygon where the edges of the image and the point cloud overlap. (Polygon area) Calculate according to formula (12).

[0074] In the formula, The number of vertices of the polygon. Represents the vertices of a polygon. The edge pixels used for registration with geometric feature points in the point cloud are optimized using Equation (13) by maximizing the IoU cost. . In the formula, It optimizes the leading edge pixels. To optimize edge pixels.

[0075] This establishes image geometric features that match the point and line features obtained in step 7.1, and these features, together with the point and line features obtained in step 7.1, form the optimal geometric feature pair.

[0076] Step 7.3: Based on the optimal geometric feature pairs of the calibration board in the point cloud and image obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, select a set of optimal edges, and select a set from the optimal geometric feature pairs based on the optimal edges to construct the constraint equation and coarsely calculate the extrinsic parameter matrix; In this implementation step, based on the optimal geometric feature pairs obtained in steps 7.1 and 7.2, and according to the constraint relationship between their coordinates in space and their projection positions in the image, an estimation equation for the extrinsic parameter matrix is ​​constructed. This includes a coarse estimation of the rotation matrix. The calculation is obtained by optimizing equation (14), using the initial rotation matrix obtained through measurement. To optimize the starting point.

[0077]

[0078] In the formula, N Represents the total number of poses of the calibration plate; This represents the rotation matrix. Indicates the first The normal direction of the calibration plate in the camera coordinate system under the pose of the calibration plate. Indicates the first The normal direction of the calibration plate in the lidar coordinate system at each pose. and These are the camera coordinate system and the lidar coordinate system, respectively. The first position under the first position Each line has a characteristic direction vector.

[0079] Because the calibration board edge points in the lidar point cloud are few and the noise is high, in the implementation of this step, such as Figure 3 The invention described herein utilizes three sets of edges: the outer edge 1a-1d of the calibration plate, the outer edge 2a-2d of the perforated area, and the inner edge 3a-3d of the perforated area. Based on the observed edge line lengths and their continuity indicators, the optimal edge set is dynamically selected. For example, when the lidar is installed at a height of 1.2 meters, the calibration plate is placed vertically, and its diagonals are parallel and perpendicular to the ground, respectively, when the distance from the camera to the calibration plate is greater than 1.2 meters but less than 2.8 meters, the inner edge 3a-3d of the perforated area and the internal area on the calibration plate can be fully observed, while the outer edge 1a-1d and the outer edge 2a-2d of the perforated area are only partially visible. In this case, only the edge line features of 3a-3d are extracted. When the distance is greater than 2.8 meters, the calibration plate can be completely observed, and all line features on the calibration plate are extracted.

[0080] Adaptive optimization of the point cloud plane based on the extracted line features The centroid. To simplify the notation, an auxiliary matrix is ​​defined. Equation (15).

[0081]

[0082] In the formula, It is the identity matrix. and These are the camera coordinate system and the lidar coordinate system, respectively. Under the pose of the calibration board, the first Each line has a characteristic direction vector.

[0083] Using the pose of the calibration plate in the lidar coordinate system and the pose of the calibration plate in the camera coordinate system, the translation vector is roughly estimated by solving the linear equation system composed of equations (16) and (17).

[0084] In the formula, To roughly estimate the translation vector; This is for a rough estimate of the rotation matrix. It is the centroid of the calibration plate in the lidar coordinate system; It is the distance from the camera to the centroid of the calibration plate; These are the coordinates of the feature points selected on the calibration plate in the radar coordinate system; while These are the coordinates of the feature point in the camera coordinate system.

[0085] Step 7.4: Based on the coarsely estimated rotation matrix and coarsely estimated displacement vector obtained in Step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and image. This leads to the further construction of a cost function minimizing the cost based on Euclidean distance, and the Levenberg-Marquardt algorithm is used to achieve least-squares optimization, accurately solving for the extrinsic parameter matrix between the LiDAR and the camera—the rotation matrix. Translation vector .

[0086] In this implementation step, a cost function is defined based on Euclidean distance, and the nonlinear cost equation shown in equation (18) is constructed to jointly optimize the extrinsic parameters. Specifically, N calibration plate pose data are collected, assuming that on the... Individual poses are collected laser points , No. On the edge Each point is used. Constraints are constructed using geometric features extracted from the point cloud and image, and least-squares optimization is performed using the Levenberg-Marquardt (LM) algorithm to solve the nonlinear optimization problem. The rotation matrix obtained in step 7.3 is used for computation. Roughly estimate the translation vector To optimize the starting point.

[0087] In the formula, N Represents the number of poses; No. Total number of laser points in each pose; For the defined auxiliary matrix, No. In the first pose, the second The first edge feature on the ... k The coordinates of a laser point in the lidar coordinate system This represents the coordinates of the 3D reprojection point of the corresponding edge pixel in the camera image after internal parameters and distortion correction; Indicates the first The normal direction of the calibration plate in the camera coordinate system at each pose; It is the first The plane offset of the calibration plate in the camera coordinate system at each pose (i.e., the constant term in the normal equation).

[0088] The rotation matrix is ​​precisely calculated using the above formula. Translation vector This enables the joint calibration of multi-line lidar and camera.

[0089] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A method for joint calibration of lidar and camera, characterized in that, The method includes the following steps: Step 1: Rigidly connect the lidar and the camera to lock their relative position and orientation; adjust the orientation of the lidar and the camera so that their perceived fields of view overlap. Step 2: Place the calibration board in the common field of view of both the LiDAR and the camera. Use the LiDAR to collect point cloud data containing the entire calibration board, and use the camera to collect images of the calibration board containing the visual calibration pattern. During the acquisition process, change the position or orientation of the calibration board multiple times, and at each position or orientation of the calibration board, simultaneously collect multiple sets of LiDAR point cloud data and camera image data pairs. The calibration plate has a visual calibration pattern for visual recognition and multiple rectangular perforations for lidar recognition; the multiple rectangular perforations are of the same size and are independent and non-connected to each other, and are located on the top, right, bottom and left sides of the calibration plate, forming a centrally symmetrical arrangement with respect to the geometric center of the calibration plate. Step 3: Generate a relatively dense and structurally complete target point cloud based on the point cloud collected in Step 2; Step 4: Construct a local neighborhood of the point cloud with a point P in the target point cloud as the center, and use principal component analysis to extract the geometric features of each neighborhood. Then, calculate the entropy function value of each neighborhood based on its geometric features. Finally, select the neighborhood corresponding to the minimum entropy value as the optimal neighborhood to obtain the initial calibration board planar point cloud. Step 5: Use the angle relationship between the normal vectors of adjacent points in the initial calibration board plane point cloud to determine the geometric continuity of the plane point set formed by the initial calibration board plane point cloud, and identify misclassified points; Then, the points that confirm continuity are used as seed points, and a region growing algorithm is executed in the initial calibration board point cloud to remove misclassified points and obtain the geometric features of the calibration board point cloud; Step 6: Using the calibration board image containing the visual calibration pattern acquired in Step 2 as the input image, identify the pose of the calibration board in the image, extract its significant edge contours and the two-dimensional projection coordinates of key points in the image, and obtain the geometric features of the calibration board image. Step 7: According to the known physical geometric relationship of the calibration board, the geometric features of the calibration board image extracted based on step 6 are optimized and registered with the geometric features of the calibration board point cloud obtained in step 5, and a nonlinear cost equation is constructed; the least square optimization of the nonlinear cost equation is realized through the Levenberg-Marquardt algorithm, and the optimal external parameter matrix of the laser radar and the camera is solved , including a rotation matrix and a translation vector , completing the joint calibration of the laser radar and the camera; Step 7 specifically includes the following steps: Step 7.1: Perform back projection on the edge pixels of the calibration board contour extracted in Step 6, and perform optimal fitting on the point cloud features of the calibration board using principal component analysis. Step 7.2: Project the optimal cloud edge line features fitted in Step 7.1 onto the visual image, construct a cost function with Intersection over Union (IoU) as the metric, optimize the edge pixels used for registration with geometric feature points in the point cloud, establish image geometric features that match the point features and line features in the point cloud obtained in Step 7.1, and form the optimal geometric feature pair with the point features and line features obtained in Step 7.1; Step 7.3: Based on the optimal geometric feature pairs of the calibration board in the point cloud and image obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, select a set of optimal edges, and select a set from the optimal geometric feature pairs based on the optimal edges to construct the constraint equation and coarsely calculate the extrinsic parameter matrix; Step 7.4: Based on the coarsely estimated rotation matrix and coarsely estimated displacement vector obtained in Step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and image. This leads to the further construction of a cost function minimizing the cost based on Euclidean distance, and the Levenberg-Marquardt algorithm is used to achieve least-squares optimization, accurately solving for the extrinsic parameter matrix between the LiDAR and the camera—the rotation matrix. Translation vector .

2. The method according to claim 1, characterized in that, The method for generating a relatively dense and structurally complete target point cloud based on the point cloud collected in step 2 in step 3 is as follows: Select multiple consecutive frames of point cloud data collected in step 2, and filter each frame of point cloud according to a preset Euclidean distance threshold to remove points whose Euclidean distance exceeds the threshold. The pose of the filtered first frame point cloud is used as the reference pose; the spatial transformation matrix from the point cloud of each other frame to the reference pose is solved respectively. Based on the spatial transformation matrix, the point clouds of other frames are registered to the reference pose. After registration, the point clouds of multiple frames are merged to generate a relatively dense and structurally complete target point cloud.

3. The method according to claim 1, characterized in that, In step 4, a spherical radius search method is used to construct the local neighborhood. During the search process, principal component analysis is used to analyze the geometric features of the local neighborhood, and the entropy function value of the neighborhood under the corresponding search radius is calculated based on the geometric features.

4. The method according to claim 1, characterized in that, The method for selecting an optimal set of edges is as follows: based on the three sets of edges—the outer edge of the calibration plate (1a-1d), the outer edge of the perforation (2a-2d), and the inner edge of the perforation (3a-3d)—the optimal set of edges is dynamically selected according to the observed edge line length and its continuity.

5. A calibration board for implementing the joint calibration method of lidar and camera as described in any one of claims 1-4, characterized in that, The upper perforation is arranged horizontally, close to the upper edge of the calibration plate. The horizontal distance from its left side to the left edge of the calibration plate is L2, and the vertical distance from its upper side to the upper edge of the calibration plate is L2. The right perforation is arranged vertically, with its right side horizontally distanced from the right edge of the calibration plate by L2, and its upper side is flush with the upper side of the upper perforation. The lower perforation is arranged horizontally, close to the lower edge of the calibration plate. The horizontal distance from its right side to the right edge of the calibration plate is L2, and the vertical distance from its lower side to the lower edge of the calibration plate is L2. The left perforation is arranged vertically, with its left side horizontally distanced from the left edge of the calibration plate by L2, and its lower side is flush with the lower side of the lower perforation.

6. The calibration plate according to claim 5, characterized in that, The visual calibration pattern is set in the upper left area of ​​the calibration plate. The upper side of the visual calibration pattern area is flush with the lower side of the upper perforation, and the left side is aligned with the right side of the left perforation. From the camera's perspective, the complete calibration plate image has neither axial symmetry nor central symmetry.

7. The calibration plate according to claim 6, characterized in that, The calibration plate has a side length of 11L2; the rectangular perforation has a width of L2 and a length of 7L2; the visual calibration pattern occupies an area of ​​5L2×6L2.

Images