Calibration board and laser radar and camera combined calibration method based on calibration board

Abstract

The invention discloses a calibration board and a laser radar and camera combined calibration method based on the calibration board, and relates to the technical field of multi-sensor calibration. The calibration plate is provided with a visual calibration pattern and a plurality of rectangular through holes which are consistent in size, are mutually independent, are not communicated and are respectively positioned on the upper, right, lower and left sides of the calibration plate and form central symmetry relative to the geometric center of the calibration plate; the horizontal distance between the left side edge of the upper side through hole and the left edge of the calibration plate and the vertical distance between the upper side edge of the upper side through hole and the upper edge of the calibration plate are both L2; the horizontal distance between the right side edge of the right through hole and the right edge of the calibration plate is L2, and the upper side edge of the right through hole is flush with the upper side edge of the upper through hole; the horizontal distance between the right side edge of the lower side through hole and the right edge of the calibration plate and the vertical distance between the lower side edge of the lower side through hole and the lower edge of the calibration plate are both L2; the horizontal distance between the left side edge of the left through hole and the left edge of the calibration plate is L2, and the lower side edge of the left through hole is flush with the lower side edge of the lower through hole.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-sensor calibration, and in particular to a calibration board and a laser radar and camera joint calibration method based on the calibration board. BACKGROUND

[0002] Laser radar (LiDAR) is an active sensor that can directly obtain the geometric structure and depth information of the surrounding environment through generated point cloud data, and has the advantages of high measurement accuracy and long ranging distance. However, laser radar cannot provide rich color and texture information, so it is difficult to achieve comprehensive semantic understanding of the environment relying only on point cloud data. As a visual sensor, a camera can capture images to obtain rich information such as color and texture of the environment, and target recognition, semantic segmentation and scene understanding based on images are relatively mature. However, since image data does not contain depth information, it is difficult to obtain accurate three-dimensional information. Therefore, through the heterogeneous data fusion of laser radar point cloud and image information, the complementary of the sensing characteristics of different sensors can be realized, and the limitations of a single sensor in environmental perception can be effectively compensated, thereby improving the overall sensing accuracy and robustness of the system.

[0003] The key to heterogeneous data fusion is the accurate calibration between multiple sensors, which requires establishing the correspondence between laser radar point cloud and image pixels. Assuming is the camera coordinate system, is the laser radar coordinate system, and assuming that there is a space point P in the world coordinate system, which is a three-dimensional space point P L =(x L ,y L ,z L ) T in the laser radar coordinate system; and its coordinates in the camera coordinate system are P C =(x c ,y c ,z c ) T , and the calibration is to establish the correspondence between P C and P L , that is, to give the representation of a space point in the world coordinate system in the laser radar coordinate system P L , and through the calibration result, the corresponding P C of the point in the pixel coordinate system can be found. The current mainstream calibration method usually relies on a calibration board with obvious structural features, extracts co-visible feature points or feature surfaces in the laser radar point cloud and the image, and establishes a coordinate transformation relationship. However, the existing method faces the following problems in actual application: The existing calibration board has single geometric feature type, and it is difficult to meet the feature observability under complex environment and multi-view conditions. Especially in large view angle deviation or non-empty environment, it is difficult to ensure that the features can be effectively observed and extracted in the point cloud and the image, which leads to the difficulty in full alignment of the two source data and affects the registration accuracy. At the same time, some calibration boards with complex structure have high machining precision and high cost, which limits their large-scale deployment in engineering.

[0004] The existing method usually optimizes under the premise that the geometric features of the calibration board can be accurately extracted, but in the actual scene with sparse point cloud, complex background or obvious shielding interference, the effective features are easy to be missing, which leads to calibration failure or error increase. When the view angle of the calibration board is limited or partially shielded, the existing method usually cannot construct stable geometric constraints, and the application range is limited.

[0005] Most of the existing methods mainly rely on sparse edges, corner points and other local features in the point cloud. In the case of point cloud noise, target structure missing or low laser reflectivity, feature extraction error or registration failure is easy to occur, and there is a lack of sufficient feature redundancy and structural diversity to enhance the robustness of registration constraints, especially in long-distance or low-density point cloud scenes.

[0006] Most of the existing methods need multi-view or multi-frame observation data to establish complete registration relationship, which not only increases the operation complexity, but also increases the cost of manual intervention and reduces the efficiency of system deployment in the case of limited space or complex operation conditions.

[0007] Most of the existing methods need to calibrate the camera intrinsic parameters and complete image de-warping, and then establish the corresponding relationship between the image and the point cloud through coordinate conversion. This process relies on strong prior conditions and has a long error propagation chain. SUMMARY

[0008] In view of the above deficiencies of the prior art, the present application provides a calibration board and a laser radar and camera joint calibration method based on the calibration board.

[0009] The technical scheme of the present application is: A calibration board has a visual calibration pattern for visual recognition, and also has a plurality of rectangular perforations for laser radar recognition.

[0010] Further, according to the calibration board, the rectangular perforations have consistent sizes and are independent and not connected with each other, and are respectively located on the upper, right, lower and left sides of the calibration board, and constitute a central symmetric arrangement relative to the geometric center of the calibration board.

[0011] Further, according to the calibration board, the upper perforation is arranged in a horizontal direction and close to the upper edge of the calibration board, and the horizontal distance between the left side of the upper perforation and the left edge of the calibration board is L2, the vertical distance between the upper edge of the upper side margin calibration plate is L 2; the right side hole is arranged in a vertical direction, and the horizontal distance between the right edge of the right side margin calibration plate is L 2, the upper side of the upper side hole is flush with the upper side of the upper side hole; the lower side hole is arranged in a horizontal direction, close to the lower edge of the calibration plate, and the horizontal distance between the right edge of the right side margin calibration plate is L 2, the vertical distance between the lower edge of the lower side margin calibration plate is L 2; the left side hole is arranged in a vertical direction, and the horizontal distance between the left edge of the left side margin calibration plate is L 2, the lower side of the lower side hole is flush with the lower side of the lower side hole.

[0012] Further, according to the calibration plate, the visual calibration pattern is arranged in the upper left area of the calibration plate, the upper side of the visual calibration pattern area is flush with the lower side of the upper side hole, and the left side is aligned with the right side of the left side hole; under the camera visual angle, the complete calibration plate image has neither axial symmetry nor central symmetry.

[0013] Further, according to the calibration plate, the length of the calibration plate is 11 L 2; the width of the rectangular hole is L 2, the length is 7 L 2; the visual calibration pattern occupies an area of 5 L 2x6 L 2.

[0014] A laser radar and camera joint calibration method based on the above calibration plate, the method comprising the following steps: Step 1: rigidly connect the laser radar and the camera, lock the relative position and attitude of the two, adjust the orientation of the laser radar and the camera, so that the perception field of view of the two produces an overlapping area; Step 2: place the calibration plate in the common field of view of the laser radar and the camera at the same time, use the laser radar to collect point cloud data containing the whole calibration plate, and use the camera to collect calibration plate image containing visual calibration pattern at the same time; during the collection, the position or attitude of the calibration plate is changed many times, and a plurality of sets of laser radar point cloud data and camera image data pairs are synchronously collected under each calibration plate position or attitude; Step 3: generate a relatively dense and complete target point cloud based on the point cloud collected in step 2; Step 4: construct a local neighborhood of the point cloud with a certain point P in the target point cloud as the center, extract the geometric features of each neighborhood by principal component analysis, calculate the entropy function value of each neighborhood based on the geometric features of each neighborhood, and finally select the neighborhood corresponding to the minimum entropy value as the optimal neighborhood, thereby obtaining the initial calibration plate plane point cloud; Step 5: Determine the geometric continuity of the plane point set formed by the initial calibration board plane point cloud by using the angle relationship between the normal vectors of adjacent points in the initial calibration board plane point cloud, and determine the misclassified points; then take the points that confirm the continuity as seed points, and perform a region growing algorithm in the initial calibration board plane point cloud to remove the misclassified points, to obtain the geometric features of the calibration board point cloud; Step 6: Take the calibration board image containing the visual calibration pattern collected in step 2 as an input image, recognize the pose of the calibration board in the image, extract the significant edge contour and the two-dimensional projection coordinates of the key points in the image, and obtain the geometric features of the calibration board image; Step 7: According to the known physical geometric relationship of the calibration board, optimize and register the geometric features of the calibration board image extracted based on step 6 and the geometric features of the calibration board point cloud obtained in step 5, construct a nonlinear cost equation; realize the least squares optimization of the nonlinear cost equation through the Levenberg-Marquardt algorithm, and solve the optimal extrinsic parameter matrix of the laser radar and the camera , including the rotation matrix and the translation vector , to complete the joint calibration of the laser radar and the camera.

[0015] Further, according to the laser radar and camera joint calibration method, the method for generating a relatively dense and structurally complete target point cloud based on the point cloud collected in step 2 is: selecting continuous multiple frames of point cloud data collected in step 2, filtering each frame of point cloud according to a preset Euclidean distance threshold, and removing points with a Euclidean distance exceeding the threshold; taking the pose of the first frame of filtered point cloud as a reference pose; solving the spatial transformation matrix of each frame of point cloud to the reference pose; based on the spatial transformation matrix, registering each frame of point cloud to the reference pose, and after registration, merging the multiple frames of point cloud to generate a relatively dense and structurally complete target point cloud.

[0016] Further, according to the laser radar and camera joint calibration method, in step 4, the local neighborhood is constructed by using a spherical radius search method; in the search process, the principal component analysis method is used to analyze the geometric features of the local neighborhood, and the entropy function value of the neighborhood under the corresponding search radius is calculated based on the geometric features.

[0017] Further, according to the laser radar and camera joint calibration method, step 7 specifically includes the following steps: Step 7.1: Back-project the calibration board contour edge pixels extracted in step 6, and optimally fit the line features of the calibration board point cloud through the principal component analysis method; Step 7.2: Project the optimal edge line feature of the point cloud fitted in step 7.1 onto the visual image, construct a cost function with the intersection over union IoU as the metric, optimize the edge pixel points for registration with the geometric feature points in the point cloud, establish the image geometric features matched with the point features and line features obtained in step 7.1, and form the optimal geometric feature pairs with the point features and line features obtained in step 7.1; Step 7.3: Based on the optimal geometric feature pairs of the calibration board obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, a set of optimal edges is selected, and a set of optimal edges is selected from the optimal geometric feature pairs to construct a constraint equation to coarsely calculate the extrinsic parameter matrix; Step 7.4: Based on the coarse rotation matrix and the coarse displacement vector obtained in step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and the image, so as to further construct a minimum cost function based on the Euclidean distance definition, and the least squares optimization is realized by using the Levenberg-Marquardt algorithm to accurately solve the extrinsic parameter matrix between the laser radar and the camera, i.e. the rotation matrix and the translation vector .

[0018] Further, according to the laser radar and camera joint calibration method, the method for selecting a set of optimal edges is: based on the three groups of edges of the outer edge of the calibration board, the outer edge of the perforated area and the inner edge of the perforated area, the optimal edge group is dynamically selected according to the observed edge line length and its continuity.

[0019] Compared with the prior art, the present application has the following beneficial effects: The calibration board of the present application has a rich geometric structure, which significantly enhances the feature extraction and registration capability in the calibration process. The special designed asymmetric geometric structure effectively avoids axisymmetric interference and ensures the uniqueness of the coordinate system. At the same time, the calibration board integrates multiple geometric elements such as edges, corners, holes, lines and surfaces, so that key features can be observed and extracted in both images and point clouds, which not only improves the diversity of matched features, but also increases the feature density, providing strong support for high-precision calibration.

[0020] The present application simplifies the feature extraction process by providing redundant features, and improves the registration stability. The use of plane and line features to construct geometric constraints enhances the stability of feature registration and effectively solves the rigid transformation problem between sensor coordinate systems.

[0021] (3) Realize adaptive calibration constraint and improve applicability. By capturing various geometric features in the point cloud and the image, the calibration task can still be completed even if only part of the calibration board is visible, which enhances the applicability of the method, expands the application range of the calibration board, reduces manual adjustment, and improves the overall efficiency.

[0022] (4) Based on the high-precision mapping model of point cloud and image pixel, the present application introduces the IoU alignment strategy, combines the entropy value calculation and geometric fitting method, so that the calibration process is not limited by the scene complexity. Even under the conditions of non-empty environment, calibration board only partially visible, complex background or existence of interference objects, stable high-quality constraint relationship can still be established, ensuring the robustness and accuracy of the calibration process, significantly expanding the application range of the calibration board.

[0023] (5) Support single observation data calibration, simplify the process, reduce manual operation. The present application can obtain sufficient line and surface features and construct complete constraint equations through geometric feature extraction in single observation, complete the calibration task, avoid relying on multi-frame or multi-view data, significantly simplify the calibration process, reduce the frequency of manual adjustment, improve the overall efficiency and operation convenience.

[0024] (6) The present application can obtain the camera intrinsic parameter without pre-acquisition in the joint calibration process, reduce the pre-reliance, reduce the system integration and use threshold, improve the universality and engineering operability of the method.

[0025] (7) The present application significantly improves the accuracy and robustness of the initial matching through the high-stability and high-discrimination geometric feature extraction method. In the feature registration stage, the abnormal point interference is effectively suppressed, and the redundant iteration process is reduced. While ensuring the calibration accuracy, the calibration time is shortened, which is especially suitable for scenes with high real-time requirements, and shows good efficiency and usability in actual deployment. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is the structure schematic diagram of the calibration board of the present embodiment; Figure 2 is the flow schematic diagram of the laser radar and camera joint calibration method of the present embodiment; Figure 3 is the observation schematic diagram in the calibration process of the present embodiment; Among them, the figure mark explanation is: 1-base plate, 21-upper side perforated area, 22-right side perforated area, 23-lower side perforated area, 24-left side perforated area, 3-chessboard pattern area, 1a-1d-outer edge of the calibration board, 2a-2d-outer edge of the perforated area, 3a-3d-inner edge of the perforated area. DETAILED DESCRIPTION

[0027] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings.

[0028] As Figure 1As shown, the embodiment provides a calibration board for combined calibration of multi-line laser radar and camera, which comprises a rectangular substrate 1 with a certain thickness, the rectangular substrate having an upper surface and a lower surface, the thickness of the calibration board is generally controlled at about 3-10 mm, preferably 5 mm, so as to balance the structural rigidity and facilitate installation and use on various platforms. The side length of the calibration board is L = 1100 mm. Two types of calibration feature areas are integrated on the substrate: rectangular perforated areas 21-24 for laser radar recognition, and checkerboard pattern area 3 for visual recognition.

[0029] The rectangular perforated areas of the calibration board according to the embodiment are arranged around the substrate, i.e. on the upper, right, lower and left sides of the calibration board. The four rectangular perforated areas (21, 22, 23, 24) are independent and not connected to each other, and the size of the four perforated areas is consistent, all being rectangular with a length of 700 mm and a width of 100 mm, as shown in Figure 1 The black part is the substrate entity, and the grid represents the rectangular perforated area, which specifically includes: The upper perforated area 21 is arranged in a horizontal direction and close to the upper edge of the substrate. The horizontal distance between the left side of the upper perforated area 21 and the left edge of the substrate is 100 mm, and the vertical distance between the upper side of the upper perforated area 21 and the upper edge of the substrate is 100 mm. The right perforated area 22 is arranged in a vertical direction, and the horizontal distance between the right side of the right perforated area 22 and the right edge of the substrate is 100 mm. The upper side of the right perforated area 22 is flush with the upper side of the upper perforated area 21 (i.e. at the same horizontal height).

[0030] The lower perforated area 23 is arranged in a horizontal direction and close to the lower edge of the substrate. The horizontal distance between the right side of the lower perforated area 23 and the right edge of the substrate is 100 mm, and the vertical distance between the lower side of the lower perforated area 23 and the lower edge of the substrate is 100 mm. The left perforated area 24 is arranged in a vertical direction. The horizontal distance between the left side of the left perforated area 24 and the left edge of the substrate is 100 mm, and the lower side of the left perforated area 24 is flush with the lower side of the lower perforated area 23.

[0031] The above-mentioned perforated areas are all rectangular through-hole structures, which are preferably made by laser cutting or numerical control (CNC) processing. The hole edges are straight, the size is accurate, and the hole thickness is consistent with the thickness of the substrate 1. The four perforated areas are centrally symmetrically arranged relative to the geometric center of the substrate, which facilitates the formation of clear boundaries, line segments, corner points and other geometric discontinuous features in the point cloud, and is used for the extraction and fitting of edges or corner points in the subsequent laser radar data.

[0032] The upper left area of the substrate 1 is provided with a checkerboard pattern area 3, the upper side of the checkerboard pattern area 3 is flush with the lower side of the perforated area 21, and the left side is aligned with the right side of the perforated area 24. The checkerboard adopts a 5*6 square grid array, a total of 30 black and white squares, each square has a length of 100mm, and the overall area is 500mm*600mm. The checkerboard is made of matte black and white materials sprayed, which improves the image contrast and adapts to different lighting conditions, and ensures that the image features are clear and identifiable.

[0033] It should be noted that the checkerboard pattern can also be replaced by other types of visual calibration patterns according to specific visual calibration needs, such as ArUco code, AprilTag label or regular dot array, etc., which has good universality and scalability.

[0034] In order to improve the calibration accuracy and the uniqueness of the calibration board identification, the calibration board intentionally removes the axial symmetry and central symmetry in the camera view angle in the design, and constructs a unique feature structure relationship. This structure is convenient for identifying and restoring the pose information of the calibration board in the image or point cloud, and is especially suitable for pose estimation and reconstruction in the case of partial visibility. For example, through the joint identification of the corner points of the visual calibration pattern area and the corner points of the perforated area, the center position and normal direction of the calibration board in the three-dimensional space can be accurately determined.

[0035] The calibration board can be installed on a vertical surface or a support platform with a certain inclination through the fixing hole, hook structure or tripod buckle provided on the back, and is suitable for multi-sensor joint calibration tasks in various indoor and outdoor environments. The camera and the laser radar can observe the calibration board at the same time, so as to extract corresponding features in the image and the point cloud for joint calibration calculation.

[0036] On the basis of the above-mentioned customized calibration board, the embodiment of the application proposes a multi-line laser radar and camera high-efficiency and high-precision joint calibration method.

[0037] Suppose the rigid transformation from the laser radar coordinate system to the camera coordinate system is The rotation matrix is The translation vector is , wherein is the rotation group in three-dimensional space, is a three-dimensional vector space. For a point P C in the camera coordinate system, its coordinates in the laser radar point cloud are P L At this time, the conversion relationship between the two can be expressed as formula (1).

[0038]

[0039] The ultimate goal of the multi-line laser radar and camera high-efficiency and high-precision joint calibration method is to obtain the external parameter and , realize the joint calibration of laser radar and camera, as Figure 2 The specific steps are as follows: Step 1: According to the calibration requirement of the external parameters between the laser radar and the camera, the laser radar and the camera are rigidly connected and fixed together to form a rigid body structure, which provides a stable spatial reference frame for subsequent joint calibration.

[0040] In the embodiment of this step, the laser radar and the camera are rigidly fixed on the same mounting base, and the relative position and attitude of the two are locked; the orientations of the laser radar and the camera are adjusted so that the sensing fields of view of the two overlap, and it is ensured that the overlapping region corresponds to more than 50% of the effective imaging area in the camera field of view.

[0041] Step 2: Based on the calibration data acquisition requirement, the calibration board proposed in the embodiment of the present application is placed in the common field of view of the laser radar and the camera at the same time, the point cloud data containing the calibration board is collected by using the laser radar, and the corresponding image is collected by using the camera, which provides the original input for subsequent point cloud and image feature matching.

[0042] In the embodiment of this step, the calibration board is fixed at a position that can be observed by the laser radar and the camera, and is kept at an appropriate angle, so that each side of the rectangular outer edge of the calibration board is crossed by at least three laser radar scanning lines, thereby ensuring that the edge structure of the calibration board can be clearly reflected in the laser point cloud data; at the same time, the camera should be able to clearly image the visual calibration pattern on the calibration board, avoiding image blur affecting subsequent feature extraction.

[0043] Under each calibration board pose, at least 50 sets of laser radar point cloud data and camera image data pairs are synchronously collected. In order to enhance the diversity and spatial coverage of the calibration data, after completing the data collection at one position, the pose of the calibration board is changed, and the above collection process is repeated.

[0044] Step 3: The point cloud collected in step 2 is preprocessed, and the target point cloud is obtained by fusing multiple frames of point cloud.

[0045] The embodiment solves the problem of incomplete observation information caused by sparse point cloud by merging multiple frames of point cloud to pre-process the point cloud obtained in step 2 to obtain a dense and complete target point cloud. In the embodiment of this step, three consecutive frames of point cloud data are selected, and each frame of point cloud is filtered according to a preset Euclidean distance threshold to remove points (outliers, usually points far away from the sensor) whose Euclidean distance exceeds the threshold. The pose of the filtered first frame of point cloud is taken as the reference pose. The spatial transformation matrix of the second frame of point cloud and the third frame of point cloud to the reference pose is solved by using the Iterative Closest Point (ICP) algorithm or the Normal Distributions Transform (NDT) algorithm, respectively. Based on the spatial transformation matrix, the second frame of point cloud and the third frame of point cloud are registered to the reference pose. After registration, the three frames of point cloud are merged to generate a relatively dense and complete target point cloud.

[0046] Step 4: By constructing a local neighborhood and combining principal component analysis to extract geometric features, and using an entropy function to guide adaptive neighborhood division, the local structure of the point cloud is accurately segmented.

[0047] Based on the target point cloud obtained in step 3, a local neighborhood is first constructed around the query point, the eigenvalues of the local covariance matrix are extracted using principal component analysis (PCA), and the dimension and geometric features of the point are calculated based on this. Then, in the overall range of the target point cloud, the entropy function values of different scale neighborhoods are calculated based on the local dimension features, and the neighborhood with the minimum entropy value is selected as the optimal neighborhood to achieve adaptive division of the optimal neighborhood and obtain the initial calibration plate plane point cloud.

[0048] In the embodiment of this step, a local neighborhood of the point cloud is constructed around a point P in the target point cloud, which refers to a group of neighboring points around the point P in the point cloud. The neighborhood is constructed using a spherical radius search method, i.e. assuming that the lower limit of the initial search radius is , the upper limit is , and the search step size is , then the th search radius is , and the principal component analysis (PCA) method is used to analyze the local neighborhood dimension features and the entropy function during the search process. The entropy function is a measure function that measures the uncertainty or information content of the local neighborhood of the point cloud. The specific implementation of the entire process of determining the optimal neighborhood based on the principal component analysis of the local dimension features is as follows: For a local neighborhood of a spatial point P in the target point cloud, the covariance matrix of the local neighborhood can be expressed as a symmetric positive definite matrix C, and three eigenvalues can be obtained by eigenvalue decomposition of the matrix C according to formula (2). Let the eigenvalues be , and respectively, where , represent the principal direction eigenvalues of the local neighborhood of the corresponding point cloud in the plane, and the corresponding eigenvectors are represented as .

[0049]

[0050] After obtaining the eigenvalues, the dimension feature of the point cloud is calculated according to formula (3), so as to determine the geometric feature (linear, planar, curved surface) of the point cloud.

[0051]

[0052] In formula (3), represents a straight line; represents a plane; represents a curved surface.

[0053] In order to determine the optimal neighborhood for point cloud segmentation, the entropy function value under the corresponding search radius is calculated according to formula (4) , and the optimal neighborhood corresponds to the minimum entropy value. Thus, the initial calibration board planar point cloud is obtained.

[0054] Step 5: Analyze the geometric continuity of the planar point set constituted by the initial calibration board planar point cloud by using the angle relationship between the normal vectors of adjacent points in the initial calibration board planar point cloud, and identify the misclassified points caused by the surface disturbance of the calibration board and the curvature of the edge of the calibration board. Then, the initial calibration board planar point cloud is segmented by using the region growing algorithm to remove the misclassified points, and the space pose of the calibration board in the laser radar coordinate system is accurately described by parameterized modeling.

[0055] In the actual acquisition process, if the calibration board support is not used, the calibration board may be slightly warped, occluded or reflective in the natural placement state, thereby forming a small area of three-dimensional curved surface in the edge region. Such regions are easily misclassified as planes in traditional plane discrimination methods based on distance or angle threshold, affecting the calibration accuracy. In order to improve the recognition ability of geometric primitives, the embodiment proposes a plane extraction method combining spatial feature analysis (SFA, Spatial Feature Analysis) and region growing, which effectively improves the robustness of calibration board plane geometric structure extraction in sparse and noisy point cloud environment, and is especially suitable for complex indoor or outdoor scenes.

[0056] In an embodiment of this step, the set of planar points consisting of the initial calibration board planar point cloud is refined in the target point cloud. For points on the same horizontal line, the normal vector angle between adjacent points is calculated according to formula (5) and If the angle changes greatly, exceeding the preset angle threshold, it is considered that the points and are discontinuous and not on the same plane, and the point is determined as a misclassified point.

[0057]

[0058] In the formula, and are the normal vectors at points and

[0059] The points on the calibration board plane that are confirmed to be continuous by the normal vector angle analysis described above are taken as seed points, and a region growing algorithm is performed on the initial calibration board planar point cloud to segment the calibration board planar point cloud and remove misclassified points. The calibration board plane model can be parameterized as formula (6).

[0060] In the formula, for the calibration board plane in the laser radar coordinate system, C is the center of the calibration board; is the unit vector in the y-axis direction of the laser radar coordinate system; n b is the normal vector of the calibration board plane; α is the angle between the vector and the X b axis of the calibration board; the three-dimensional coordinates of the C point in the laser radar coordinate system are ; and the expression of n b is as shown in formula (7).

[0061] In the formula, and represent the components of in the and directions, respectively.

[0062] Step 6: Take the calibration board image containing the visual calibration pattern collected in step 2 as the input image, use image processing algorithms to recognize the pose of the calibration board in the image, extract the significant edge contour and key point projection coordinates in the image, i.e., the geometric features of the calibration board image. Used for solving the extrinsic parameter calibration between the laser radar and the camera.

[0063] ​In the implementation of this step, the image of the calibration board containing the visual calibration pattern obtained in step 2 is taken as the input image, target matching is performed by detecting the calibration board and the visual calibration pattern (checkerboard or other markers) in the image, the two-dimensional position of the calibration board in the visual image is determined, and in the extracted calibration board region, the geometric features are extracted by using the Canny edge detection algorithm, that is, the contour edge pixels are extracted; the two-dimensional projection coordinates of the pre-defined corner points or other key points (such as the center of the circle, the intersection point, etc.) in the geometric structure of the calibration board in the visual image are calculated; the K-D tree data structure is used to store the spatial information of the edge pixel points in the image, which is used as the geometric features of the calibration board image. So as to facilitate subsequent matching and optimization processing with the point cloud edge features.

[0064] Step 7: The calibration board image geometric features extracted based on step 6 and the calibration board point cloud geometric features obtained in step 5 are optimized and registered, and a non-linear cost equation is constructed. The least squares optimization of the cost equation is realized by the Levenberg-Marquardt algorithm, and the optimal extrinsic matrix of the laser radar and the camera is solved , including the rotation matrix and the translation vector , to complete the joint calibration of the laser radar and the camera.

[0065] Step 7.1: Based on the calibration board contour edge pixels extracted in step 6, back projection is performed, and the point cloud line features of the calibration board are optimally fitted by principal component analysis.

[0066] In the implementation of this step, based on the set of calibration board contour edge pixels extracted in step 6, for each edge pixel, according to the initial extrinsic matrix measured manually, back projection is performed in the laser radar coordinate system, and the corresponding points in the calibration board plane point cloud obtained in step 5 are selected as candidate points. For the candidate points, the nearest neighbor points are searched to form a local point set .

[0067] For each local point set T i , the optimal line feature is fitted by principal component analysis (PCA) to obtain the fitted straight line as shown in equation (8). The points on the line are as shown in equation (9).

[0068]

[0069]

[0070] Step 7.2: Project the optimal edge line feature of the point cloud fitted in step 7.1 to the visual image according to the projection model, construct a cost function with the intersection over union (IoU) as the metric, and optimize the edge pixel points used for registration with the geometric feature points in the point cloud to improve the registration accuracy of the image and the geometric features in the point cloud.

[0071] In the implementation of the present step, the optimal edge line feature of the point cloud fitted in step 7.1 is projected to the visual image plane, and the initial estimated extrinsic parameter matrix obtained by manual measurement is combined with the imaging principles of the laser radar and the camera. The projection model shown in equation (10) is used to map the point cloud plane to the image plane to obtain the correspondence between the spatial points and the image edges.

[0072] In the formula, represents a point in the three-dimensional point cloud in the laser radar coordinate system, represents the actual coordinates of the point in the image after projection, is the projection model considering camera distortion.

[0073] The projection points of the optimal point cloud edge line feature projected to the visual image plane are optimized using an intersection over union (IoU) cost function to improve the geometric accuracy for registration. The IoU cost function is defined as equation (11) In the formula, is the area of the calibration board calculated by the image edge features; is the projection area of the point cloud of the calibration board on the image; is the polygon area of the overlapping region of the image and the point cloud edge. The polygon area is calculated according to equation (12).

[0074] In the formula, is the number of polygon vertices, represents the polygon vertices. The edge pixel points used for registration with the geometric feature points in the point cloud are optimized using the maximum IoU cost according to equation (13) . In the formula, is the edge pixel point before optimization, is the edge pixel point after optimization.

[0075] Thus, the image geometric features matching the point features and line features in the point cloud obtained in step 7.1 are established. And the optimal geometric feature pairs are formed with the point features and line features obtained in step 7.1.

[0076] Step 7.3: Based on the optimal geometric feature pairs of the calibration board in the point cloud and image obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, select a set of optimal edges, and select a set from the optimal geometric feature pairs based on the optimal edges to construct the constraint equation and coarsely calculate the extrinsic parameter matrix; In this implementation step, based on the optimal geometric feature pairs obtained in steps 7.1 and 7.2, and according to the constraint relationship between their coordinates in space and their projection positions in the image, an estimation equation for the extrinsic parameter matrix is ​​constructed. This includes a coarse estimation of the rotation matrix. The calculation is obtained by optimizing equation (14), using the initial rotation matrix obtained through measurement. To optimize the starting point.

[0077]

[0078] In the formula, N This represents the total number of poses of the calibration plate; This represents the rotation matrix. Indicates the first The normal direction of the calibration plate in the camera coordinate system under the pose of the calibration plate. Indicates the first The normal direction of the calibration plate in the lidar coordinate system at each pose. and These are the camera coordinate system and the lidar coordinate system, respectively. The first position under the first position Each line has a characteristic direction vector.

[0079] Because the calibration board edge points in the lidar point cloud are few and the noise is high, in the implementation of this step, such as Figure 3 The invention described herein utilizes three sets of edges: the outer edge 1a-1d of the calibration plate, the outer edge 2a-2d of the perforated area, and the inner edge 3a-3d of the perforated area. Based on the observed edge line lengths and their continuity indicators, the optimal edge set is dynamically selected. For example, when the lidar is installed at a height of 1.2 meters, the calibration plate is placed vertically, and its diagonals are parallel and perpendicular to the ground, respectively, when the distance from the camera to the calibration plate is greater than 1.2 meters but less than 2.8 meters, the inner edge 3a-3d of the perforated area and the internal area on the calibration plate can be fully observed, while the outer edge 1a-1d and the outer edge 2a-2d of the perforated area are only partially visible. In this case, only the edge line features of 3a-3d are extracted. When the distance is greater than 2.8 meters, the calibration plate can be completely observed, and all line features on the calibration plate are extracted.

[0080] Adaptive optimization of the point cloud plane based on the extracted line features The centroid. To simplify the notation, an auxiliary matrix is ​​defined. Equation (15).

[0081]

[0082] In the formula, It is the identity matrix. and These are the camera coordinate system and the lidar coordinate system, respectively. Under the pose of the calibration board, the first Each line has a characteristic direction vector.

[0083] Using the pose of the calibration plate in the lidar coordinate system and the pose of the calibration plate in the camera coordinate system, the translation vector is roughly estimated by solving the linear equation system composed of equations (16) and (17).

[0084] In the formula, To roughly estimate the translation vector; This is for a rough estimate of the rotation matrix. It is the centroid of the calibration plate in the lidar coordinate system; It is the distance from the camera to the centroid of the calibration plate; These are the coordinates of the feature points selected on the calibration plate in the radar coordinate system; while These are the coordinates of the corresponding feature points in the camera coordinate system.

[0085] Step 7.4: Based on the coarsely estimated rotation matrix and coarsely estimated displacement vector obtained in Step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and image. This leads to the further construction of a cost function minimizing the cost based on Euclidean distance, and the Levenberg-Marquardt algorithm is used to achieve least-squares optimization, accurately solving for the extrinsic parameter matrix between the LiDAR and the camera—the rotation matrix. Translation vector .

[0086] In this implementation step, a cost function is defined based on Euclidean distance, and the nonlinear cost equation shown in equation (18) is constructed to jointly optimize the extrinsic parameters. Specifically, N calibration plate pose data are collected, assuming that on the... Individual poses are collected laser points , No. On the edge Each point is used. Constraints are constructed using geometric features extracted from the point cloud and image, and least-squares optimization is performed using the Levenberg-Marquardt (LM) algorithm to solve the nonlinear optimization problem. The rotation matrix obtained in step 7.3 is used for computation. Roughly estimate the translation vector To optimize the starting point.

[0087] In the formula, N Represents the number of poses; No. Total number of laser points in each pose; For the defined auxiliary matrix, No. In the first pose, the second The first edge feature on the ... k The coordinates of a laser point in the lidar coordinate system This represents the coordinates of the 3D reprojection point of the corresponding edge pixel in the camera image after internal parameters and distortion correction; Indicates the first The normal direction of the calibration plate in the camera coordinate system at each pose; It is the first The plane offset of the calibration plate in the camera coordinate system at each pose (i.e., the constant term in the normal equation).

[0088] The rotation matrix is ​​calculated precisely using the above formula. Translation vector This enables the joint calibration of multi-line lidar and camera.

[0089] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A calibration plate having a visual calibration pattern for visual recognition, characterized in that, It also has a plurality of rectangular perforations for laser radar recognition.

2. The calibration board of claim 1, wherein, The rectangular perforations are uniform in size and independent of each other and are not connected, and are respectively located on the upper left, upper right, lower right and lower left sides of the calibration plate, and are arranged in central symmetry relative to the geometric center of the calibration plate.

3. The calibration board of claim 2, wherein, The upper side hole is arranged horizontally, close to the upper edge of the calibration plate, and the horizontal distance between the left side of the upper side hole and the left edge of the calibration plate is L 2; the vertical distance between the upper side and the upper edge of the calibration plate is L 2; the right side hole is arranged vertically, and the horizontal distance between the right side of the right side hole and the right edge of the calibration plate is L 2; the upper side is flush with the upper side of the upper side hole; the lower side hole is arranged horizontally, close to the lower edge of the calibration plate, and the horizontal distance between the right side of the lower side hole and the right edge of the calibration plate is L 2; the vertical distance between the lower side and the lower edge of the calibration plate is L 2; the left side hole is arranged vertically, and the horizontal distance between the left side of the left side hole and the left edge of the calibration plate is L 2; the lower side is flush with the lower side of the lower side hole.

4. The calibration board of claim 3, wherein, The visual calibration pattern is arranged in the upper left area of the calibration plate, the upper side of the visual calibration pattern area is flush with the lower side of the upper side perforation, and the left side is aligned with the right side of the left side perforation; under the camera view, the complete calibration plate image has neither axial symmetry nor central symmetry.

5. The calibration board of claim 4, wherein, The calibration board side length is 11 L 2; the rectangular hole width is L 2, length is 7 L 2; the visual calibration pattern occupies an area of 5 L 2 x 6 L 2.

6. The combined laser radar and camera calibration method based on the calibration board according to any one of claims 1-5, characterized in that, The method comprises the following steps: Step 1: rigidly connecting the laser radar and the camera, locking the relative position and attitude of the two; adjusting the orientation of the laser radar and the camera so that the perception fields of the two overlap; Step 2: placing the calibration plate in the common field of view of the laser radar and the camera at the same time, using the laser radar to collect point cloud data containing the whole calibration plate, and using the camera to collect calibration plate images containing the visual calibration pattern at the same time; during the collection, the position or attitude of the calibration plate is changed multiple times, and multiple sets of laser radar point cloud data and camera image data pairs are collected synchronously under each calibration plate position or attitude; Step 3: generating a relatively dense and complete target point cloud based on the point cloud collected in step 2; Step 4: constructing a local neighborhood around a point P in the target point cloud, extracting the geometric features of each neighborhood using principal component analysis, calculating the entropy function value of each neighborhood based on the geometric features of each neighborhood, and finally selecting the neighborhood corresponding to the minimum entropy value as the optimal neighborhood to obtain the initial calibration plate plane point cloud; Step 5: using the angle relationship between the normal vectors of adjacent points in the initial calibration plate plane point cloud to judge the geometric continuity of the plane point set constituted by the initial calibration plate plane point cloud, and identifying the misclassified points; Then the points confirming the continuity are taken as seed points, and a region growing algorithm is performed in the initial calibration plate plane point cloud to remove the misclassified points, and the calibration plate point cloud geometric features are obtained; Step 6: taking the calibration plate image containing the visual calibration pattern collected in step 2 as an input image, recognizing the pose of the calibration plate in the image, extracting the two-dimensional projection coordinates of the significant edge contour and key points in the image, and obtaining the calibration plate image geometric features; Step 7: According to the known physical geometric relationship of the calibration board, the geometric features of the calibration board image extracted based on step 6 are optimized and registered with the geometric features of the calibration board point cloud obtained in step 5, and a nonlinear cost equation is constructed; the least square optimization of the nonlinear cost equation is realized through the Levenberg-Marquardt algorithm, and the optimal external parameter matrix of the laser radar and the camera is solved ; including a rotation matrix and a translation vector , completing the joint calibration of the laser radar and the camera.

7. The method of claim 6, wherein, The method for generating a relatively dense and complete target point cloud based on the point cloud collected in step 2 in step 3 is: selecting continuous multiple frames of point cloud data collected in step 2, filtering each frame of point cloud according to a preset Euclidean distance threshold, and removing points with a Euclidean distance exceeding the threshold; Taking the pose of the first frame of filtered point cloud as the reference pose; solving the spatial transformation matrix of each frame of point cloud to the reference pose; Based on the spatial transformation matrix, registering each frame of point cloud to the reference pose, and after registration, merging the multiple frames of point cloud to generate a relatively dense and complete target point cloud.

8. The method of claim 6, wherein, In step 4, the local neighborhood is constructed by spherical radius search; in the search process, the principal component analysis method is used to analyze the geometric features of the local neighborhood, and the entropy function value of the neighborhood under the corresponding search radius is calculated based on the geometric features. 9.The method of combined laser radar and camera calibration according to claim 6, wherein, The step 7 specifically comprises the following steps: Step 7.1: back-project the contour edge pixels of the calibration board extracted in step 6, and optimally fit the line features of the point cloud of the calibration board by principal component analysis; Step 7.2: project the optimal edge line features of the point cloud fitted in step 7.1 onto the visual image, construct a cost function with the intersection over union IoU as the metric, optimize the edge pixel points for registration with the geometric feature points in the point cloud, establish the image geometric features matched with the point features and line features in the point cloud obtained in step 7.1, and form an optimal geometric feature pair with the point features and line features obtained in step 7.1; Step 7.3: based on the optimal geometric feature pairs of the calibration board in the point cloud and the image obtained in steps 7.1 and 7.2, and according to the known physical geometric relationship on the calibration board, select an optimal edge group, select a group from the optimal geometric feature pairs according to the optimal edge group, and construct a constraint equation to coarsely calculate the external parameter matrix; Step 7.4: On the basis of the coarse rotation matrix and coarse displacement vector obtained in step 7.3, constraints are constructed according to the geometric features of the calibration board in the point cloud and the image, so as to further construct a minimum cost function defined based on the Euclidean distance, and the least square optimization is realized by using the Levenberg-Marquardt algorithm, so as to accurately solve the extrinsic parameter matrix between the laser radar and the camera, that is, the rotation matrix and the translation vector .

10. The method of claim 9, wherein, The method for selecting an optimal edge group is: based on the three groups of edges of the outer edges (1a-1d) of the calibration board, the outer edges (2a-2d) of the perforated region, and the inner edges (3a-3d) of the perforated region, dynamically select an optimal edge group according to the observed edge line length and its continuity.

Images