A LIDAR and camera joint calibration method and device in a weak constraint environment

Abstract

This invention relates to the field of autonomous driving perception, and specifically to a joint calibration method for LiDAR and CAMERA in weakly constrained environments. The method includes the following steps: determining a calibration site; setting a first calibration board and a second calibration board in front of the vehicle; the first calibration board being a checkerboard QR code calibration board with circular holes; the second calibration board being multiple QR code calibration boards; estimating the pose of the second calibration board; obtaining the extrinsic parameters of a telephoto camera and a fisheye camera based on the pose of the second calibration board; solving for the extrinsic parameters of the telephoto camera and LiDAR based on the first calibration board; optimizing the extrinsic parameters of the telephoto camera and LiDAR based on the second calibration board; acquiring traffic sign features from the data of the telephoto camera, fisheye camera, and LiDAR; and determining whether the extrinsic parameters of the telephoto camera, fisheye camera, and LiDAR are abnormal based on the traffic sign features. This invention can improve the calibration performance of LiDAR and cameras in weakly constrained scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception, and specifically to a method and apparatus for joint calibration of LIDAR and CAMERA under weak constraints. Background Technology

[0002] LiDAR and camera extrinsic calibration is a key technology commonly used in multi-sensor fusion applications. Its purpose is to determine the geometric relationship between the LiDAR and camera to achieve accurate sensor data fusion and scene reconstruction. LiDAR provides high-precision 3D point cloud data, while the camera provides rich texture information; combining the two enables more comprehensive and accurate environmental perception.

[0003] Common techniques for extrinsic parameter calibration of LiDAR and cameras include calibration methods based on specific calibration boards and calibration methods based on scene features.

[0004] Calibration methods based on specific calibration boards require placing a specific calibration board in the scene, with specific markers on the board, such as a checkerboard pattern. By photographing the calibration board at different positions and orientations, and using data from the LiDAR and camera, the extrinsic parameters between them can be calculated. The advantages of this method are its simplicity and high accuracy. Chinese invention patent CN113253246B describes a method for transforming the edge points of the calibration board to a pseudo-camera coordinate system and projecting them onto the pseudo-camera's imaging plane to obtain a projected image. From this projected image, the method finds the coordinates of the corner points of the calibration board in the pseudo-camera coordinate system. Chinese invention patent CN202111350523.5 describes a method for calibrating LiDAR and camera using a special calibration board and image processing methods. This method automatically adjusts equipment parameters, selects data frames, and calculates calibration results throughout the process, requiring no manual intervention.

[0005] Scene-feature-based calibration methods utilize feature points or feature planes in the scene to calculate the external parameters between the LiDAR and the camera. This method does not require a specific calibration board and is suitable for calibration in real-world scenarios. Summary of the Invention

[0006] This invention provides a method, apparatus, equipment, and medium for joint calibration of LiDAR and CAMERA under weak constraints, which solves the problems of slow calibration speed, poor accuracy, and difficulty in automatic verification of LiDAR and camera in the prior art.

[0007] In a first aspect, embodiments of this application provide a joint calibration method for LIDAR and CAMERA under weak constraints, including:

[0008] Step S1: Determine the calibration site, including: setting up a first calibration board and a second calibration board in front of the vehicle. The first calibration board is a checkerboard QR code calibration board with round holes, and the second calibration board is multiple QR code calibration boards. The second calibration board is placed on both sides of the first calibration board.

[0009] Step S2: Calibrate the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board.

[0010] Step S3: Calibrate the telephoto camera and lidar of the vehicle, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board.

[0011] Preferably, the checkerboard QR code calibration plate with round holes is an adjustable-angle ChArUco calibration plate with four round holes, the four round holes being symmetrically arranged at the four corners of the ChArUco calibration plate, and the plurality of QR code calibration plates being four or eight adjustable-angle ArUco calibration plates.

[0012] Preferably, in step S2, estimating the pose of the second calibration board and obtaining the calibration extrinsic parameters of the telephoto camera and the fisheye camera based on the pose of the second calibration board specifically includes:

[0013] Step S2-1: Acquire images of the second calibration board using a telephoto camera and a fisheye camera respectively, detect the corner points and corner point IDs of the ArUco calibration board in the images, and distinguish the ArUco calibration board based on the corner point IDs; estimate the relative pose of the ArUco calibration board based on the pixel coordinates of the corner points on each ArUco calibration board and the three-dimensional coordinates of the corner points in the world coordinate system.

[0014] Step S2-2: Based on the corner point IDs obtained in step S2-1, perform ArUco calibration board matching on the two second calibration board images. If more than two corner points are detected in the ArUco calibration board, it is determined as a valid matching calibration board. Then, calculate the extrinsic parameters of the telephoto camera and the fisheye camera based on the relative pose of each valid matching calibration board. Finally, take the average value as the final calibration result.

[0015] Preferably, solving for the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board includes:

[0016] Step S3-1: Project the 3D center point in the lidar coordinate system onto the image captured by the telephoto camera using the initial extrinsic parameters;

[0017] Step S3-2: Detect the 2D center point in the image, and form an optimized point pair with the projection point of the 3D center point and the detected 2D center point;

[0018] Step S3-3: Using the Euclidean distance between the optimized point pairs as the optimization objective, optimize the initial extrinsic parameters using an optimization algorithm.

[0019] Preferably, the 2D center point is the center of the circle corresponding to the hole in the image of the ChArUco calibration board; the 3D center point is the center of the circle corresponding to the hole in the point cloud of the ChArUco calibration board.

[0020] The method for obtaining the 2D center point includes: detecting the corner points of the ChArUco calibration board; solving the transformation matrix from the world coordinate system to the camera coordinate system based on the 3D coordinates of the corner points in the world coordinate system and the 2D coordinates in the pixel coordinate system; calculating the 3D coordinates of the center point in the world coordinate system; and projecting the 3D coordinates of the center point into the pixel coordinate system using the camera pose, camera intrinsic parameters, and distortion coefficients in the world coordinate system.

[0021] The method for obtaining the 3D center point includes: extracting multi-frame point clouds of interest from the area near the ChArUco calibration board; segmenting the planar point cloud of the ChArUco calibration board; transforming the planar point cloud into 2D planar points and sorting the point cloud; querying the row and column values ​​of the lower right corner point and deleting the non-face points below it; calculating the initial value of the center point corresponding to the circular hole in the four point clouds based on the lower right corner point and the size of the ChArUco calibration board; obtaining random points around the initial value of the center point; traversing the points within a circle with a radius of 20cm centered on the random points to obtain the number of points within the circle, and determining the center point of the circle with the fewest points within the circle; for the center point of the circle with the fewest points within each frame of the multi-frame point cloud of interest, calculating the average value of the center points of the multi-frame point cloud of interest, determining it as the final 3D center point, and transforming it to the lidar coordinate system.

[0022] Preferably, in step S3, optimizing the calibration extrinsic parameters of the telephoto camera and the LiDAR based on the second calibration board includes:

[0023] Step S3-4: Detect the corner points of the ArUco calibration board in the image; use the corner point information and PNP algorithm to solve the pose of each ArUco calibration board in the camera coordinate system; determine the position of the four corners of each ArUco calibration board in the pixel coordinate system according to the size and pose of the calibration board, and segment the pixel region corresponding to each ArUco calibration board.

[0024] Steps S3-5: Segment the ground point cloud and non-ground point cloud; perform clustering processing on the non-ground point cloud to identify the ArUco calibration board point cloud; for each clustered point cloud cluster, segment the corresponding ArUco calibration board planar point cloud; based on the calibration extrinsic parameters of the telephoto camera and LiDAR obtained by the solution, project the planar point cloud onto the pixel region corresponding to the segmented ArUco calibration board; optimize the calibration extrinsic parameters between the telephoto camera and LiDAR based on the reprojection error between the projection point and the image.

[0025] Preferably, the method further includes: step S4, verifying the calibration effect, including: acquiring traffic sign features from the data of the telephoto camera, fisheye camera, and lidar, and determining whether the external parameters of the telephoto camera, fisheye camera, and lidar are abnormal based on the traffic sign features.

[0026] Preferably, the traffic sign features include traffic sign image features and traffic sign point cloud features;

[0027] The process of obtaining traffic sign image features from the data includes: segmenting the image into blue, yellow, and red regions based on RGB or HSV colors to extract regions of the corresponding colors; applying medium-mean-weighted filtering to the segmented regions to reduce noise in the image; performing morphological operations, including erosion and dilation; calculating the contours and their minimum bounding boxes in the image, and removing smaller rectangles that contain each other; and selecting rectangles that meet the criteria based on the aspect ratio and contour area of ​​the traffic sign as traffic sign image features.

[0028] Alternatively, semantic segmentation can be performed using a deeplabV3+ network trained on the cityscapes dataset to obtain traffic sign image features;

[0029] The process of obtaining traffic sign point cloud features from the data includes: performing ground segmentation on the original point cloud and removing ground point clouds; using horizontal hierarchical clustering and vertical clustering methods to perform obstacle clustering on the non-ground point cloud; and filtering the point cloud based on the reflection intensity of the point cloud to obtain traffic sign point cloud features.

[0030] Preferably, determining whether the external parameters of the telephoto camera, fisheye camera, and lidar are abnormal based on the traffic sign features includes:

[0031] Verification is performed using multiple frames of data. The optimal extrinsic parameter and the original extrinsic parameter of each frame are compared. If the difference between the optimal extrinsic parameter and the original extrinsic parameter exceeds a preset threshold, the extrinsic parameter of that frame is marked as abnormal. The number of abnormal extrinsic parameter frames is counted. If it exceeds the preset threshold, it is determined that the extrinsic parameter of the corresponding sensor needs to be recalibrated; otherwise, the extrinsic parameter is determined to be normal.

[0032] The optimal extrinsic parameters for each frame are obtained as follows: based on the calibrated extrinsic parameters, the point cloud is projected onto the image; the intersection-union ratio (IUR) of the point cloud and the image features is calculated; the extrinsic parameters are optimized according to the IUR, and the extrinsic parameter with the largest IUR is set as the optimal extrinsic parameter.

[0033] Secondly, embodiments of this application provide a joint calibration device for LIDAR and CAMERA under weak constraints, comprising:

[0034] The determination unit is used to determine the calibration site, including: setting a first calibration board and a second calibration board in front of the vehicle, wherein the first calibration board is a checkerboard QR code calibration board with round holes, and the second calibration board is multiple QR code calibration boards;

[0035] The first calibration unit is used to calibrate the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board.

[0036] The second calibration unit is used to calibrate the vehicle's telephoto camera and lidar, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board.

[0037] Compared with existing technologies, this invention, specifically designed for the weakly constrained environment of mining areas, utilizes a calibration scenario combining a customized ChArUco calibration board and four ordinary ArUco calibration boards, providing high-precision prior data. Vehicles in mining environments are equipped with various onboard sensors, such as telephoto cameras, fisheye cameras, and LiDAR, requiring individual calibration and LiDAR calibration. To address this, this invention breaks down the calibration process into two steps: calibrating the telephoto and fisheye cameras using the ArUco calibration board, and calibrating the telephoto camera and LiDAR using the customized ChArUco calibration board, followed by optimization of the calibration results using the ArUco calibration board. This process improves the speed and accuracy of LiDAR and camera calibration scenarios. Furthermore, the verification step of this invention includes traffic sign recognition and detection, enabling verification of calibration results in daily production environments without introducing new verification markers, achieving parameter self-verification and improving the convenience of calibration result verification. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0039] Figure 1 This is a flowchart of the joint calibration method of LIDAR and CAMERA under weak constraints disclosed in this invention.

[0040] Figure 2 This is a schematic diagram of the customized ChArUco calibration plate disclosed in this invention;

[0041] Figure 3a This is a top view of the calibration site disclosed in this invention;

[0042] Figure 3b This is the calibration site design drawing disclosed in this invention;

[0043] Figure 4 This is a flowchart of the telephoto camera and lidar calibration algorithm disclosed in this invention;

[0044] Figure 5 This is a flowchart of the calibration algorithm for telephoto cameras and fisheye cameras disclosed in this invention.

[0045] Figure 6 This is a flowchart of the calibration extrinsic parameter verification algorithm disclosed in this invention;

[0046] Figure 7 This is a schematic diagram of the algorithm for the joint calibration method of LIDAR and CAMERA under weak constraints disclosed in this invention.

[0047] Figure 8 This is a block diagram of the LIDAR and CAMERA joint calibration device under weak constraint environment disclosed in this invention.

[0048] Figure 9 This is a block diagram of a computing device according to several embodiments of the present invention; Detailed Implementation

[0049] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0050] The mining environment suffers from severe degradation, and mining trucks have blind spots, making it difficult to adapt to the requirements of existing calibration algorithms. Existing calibration algorithms and calibration kits (artificially designed to accommodate all necessary markers) are incompatible. Therefore, this invention provides a joint LiDAR and CAMERA calibration method for mining trucks under weakly constrained environments.

[0051] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution of this invention is provided through a specific embodiment.

[0052] Please see Figure 1This invention provides a joint calibration method for LiDAR and CAMERA under weak constraints, comprising the following steps:

[0053] Step S1, determine the calibration site, the calibration site includes: a first calibration board and a second calibration board set in front of the vehicle, the first calibration board is a checkerboard QR code calibration board with round holes, and the second calibration board is multiple QR code calibration boards;

[0054] The calibration site is the final calibration scene obtained after determining all calibration algorithms and processes. Since one of the goals of this calibration task is to complete all static calibration tasks in a unified scene, the design of the site is a crucial factor in determining whether this goal can be achieved. The algorithm needs to be calibrated based on a self-designed site and kit. On the one hand, the placement distance must meet the field of view of each sensor. When calibrating multiple sensors, the common field of view area needs to have certain characteristic obstacles to meet the calibration requirements. On the other hand, the obstacles required for calibration cannot conflict with each other, obstruct each other, or interfere with each other. Otherwise, the calibration function of some sensors will not be realized, or the scene will need to be manually changed, which violates the requirements of the calibration system to be one-click, convenient, and unmanned.

[0055] like Figure 3a As shown in the figure, the top-down view after overall testing and individual verification demonstrates that the scene meets all site requirements for static calibration. This includes a relatively flat and level ground, three height-adjustable aluminum radar corner reflectors, a high-density calibration plate with four circular holes around its perimeter and adjustable angles, eight angle-adjustable acrylic calibration plates, and tools such as a level, measuring tape, screwdriver, and wrench.

[0056] The ChArUco (Chessboard ArUco) calibration board is a special calibration board used for camera calibration. It combines a traditional checkerboard calibration board with ArUco codes. ArUco codes are two-dimensional barcodes commonly used in applications such as augmented reality and camera pose estimation. They consist of black and white squares, each with a unique identifier that can be recognized by the camera. ArUco codes offer good robustness and detection accuracy. The ChArUco calibration board consists of a series of fixed-size checkerboard squares and embedded ArUco codes. These ArUco codes provide additional information that helps camera calibration algorithms more accurately estimate the camera's intrinsic and extrinsic parameters.

[0057] In some embodiments, the checkerboard QR code calibration plate with circular holes in the first calibration plate is a ChArUco calibration plate with four circular holes around its perimeter and an adjustable angle, such as... Figure 2As shown, the standard ChArUco calibration plate has four round holes around it; the second calibration plate is a multi-QR code calibration plate, which is an acrylic ArUco calibration plate with 4 or 8 adjustable angles.

[0058] like Figure 3b As shown in the figure, the location design of the calibration site is as follows: the left and right boundaries of the site are 10 meters apart in the vehicle width direction, and the distance between the front boundary of the site and the front of the vehicle is 15 meters. The reason for this choice is that, after testing, the common field of view of the left and middle lidars is in a part of the left front area. However, beyond 10 meters, the lidar beams for filling blind spots are relatively sparse, making them less accurate for calibration and unsuitable as target areas. Similarly, within 15 meters in front, the lidar common field of view is more effective. In addition, it is difficult to obtain high resolution images from cameras at greater distances, especially for fisheye cameras. Beyond 10 meters, it is difficult to identify patterns after distortion correction. Considering that the initial y-direction error before calibration cannot exceed 5 meters, choosing a distance of 15 meters is more appropriate. Meanwhile, the calibration of a single lidar is from its own coordinate system to the world coordinate system. Compared with a manually set horizontal surface, a level and flat ground is a more easily selected scenario. Even if the algorithm detects and eliminates bumps or missing parts of the ground, the overall levelness of the ground is still an important condition for determining whether the lidar can be accurately calibrated to the world coordinate system.

[0059] In response to the characteristics of the weakly constrained environment in mining areas, this invention designs a calibration scenario combining a customized ChArUco and four ordinary ArUco blocks. This provides high-precision prior data and is easy to operate. Combined with the subsequent automatic calibration algorithm, it can quickly solve the calibration relationship.

[0060] Step S2, calibrating the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board.

[0061] The extrinsic parameters represent the camera's position and orientation relative to a certain world coordinate system, including the translation vector T and rotation parameters R. The translation vector T describes the three-dimensional spatial displacement of the origin of the camera coordinate system relative to the origin of the lidar coordinate system, and is usually represented by a three-dimensional vector (t). x , t y , t z The rotation parameter R is used to describe the angle by which the camera coordinate system needs to be rotated relative to the lidar coordinate system to be aligned. It is represented by a rotation matrix or quaternion, which can describe any rotation in three-dimensional space.

[0062] The extrinsic parameter calibration algorithm for binocular cameras is designed and developed based on four ArUco calibration boards, two on each side, placed in the middle of the static calibration site. It mainly utilizes the relative attitude information of the ArUco calibration boards under the binocular camera to obtain the final calibration result.

[0063] In some embodiments, the flowcharts of the telephoto camera and fisheye camera calibration algorithms are as follows: Figure 5 As shown, the algorithm includes:

[0064] 1) ArUco calibration board pose estimation:

[0065] First, an ArUco calibration board needs to be prepared. Each calibration board has multiple markers, each with a unique ID, such as 1, 2, 3, 4. These IDs are used to distinguish different calibration boards. Images containing the ArUco calibration boards are taken from multiple angles using different cameras. To improve the accuracy of pose estimation, it is best to ensure that the calibration boards are clearly visible in each image. At the beginning of the algorithm, the input images are processed to detect the marker corners on the ArUco calibration boards in the image. This can be achieved using edge detection, corner detection, and other image processing algorithms. By identifying the ID of each corner, different ArUco calibration boards in the image can be distinguished, allowing for independent pose estimation for each calibration board in subsequent processing. For each calibration board, the algorithm estimates the pose of the calibration board relative to the camera, i.e., the position and orientation of the camera, based on the pixel coordinates of the detected corners and the predefined 3D coordinates of these corners in the world coordinate system.

[0066] 2) Calculation of extrinsic parameters for binocular cameras:

[0067] In two images captured by different cameras, calibration boards are matched based on the marker corner IDs detected during the pose estimation stage. A calibration board is considered a valid match only if at least two marker corners are detected in both images. For each validly matched calibration board, the algorithm independently calculates the camera's extrinsic parameters. Extrinsic parameters refer to the rotation and translation matrices of the camera coordinate system relative to the world coordinate system, used to determine the camera's exact position and orientation in space. If multiple calibration boards are validly matched, the algorithm calculates the average of the camera extrinsic parameters corresponding to all valid calibration boards. This average is used to eliminate measurement errors that may be introduced by a single calibration board, providing a more stable and reliable camera extrinsic parameter estimation result. Finally, the algorithm outputs the average of all camera extrinsic parameters as the extrinsic parameter calibration result of the stereo camera system. This result can be used in subsequent applications such as 3D reconstruction and machine vision measurement to ensure an accurate correspondence between the images captured by the camera and the actual world coordinate system.

[0068] As can be seen, in the above algorithm, different calibration boards can be distinguished by detecting corner point IDs. Only when two or more corner points are detected in the calibration board is it determined to be a valid matching calibration board, which can improve the accuracy of calibration board monitoring information, remove erroneous corner point information, and improve the accuracy of extrinsic parameter estimation by using the mean of the extrinsic parameters as the final calibration result.

[0069] Step S3 involves calibrating the vehicle's telephoto camera and lidar, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board.

[0070] In some embodiments, the flowchart of the telephoto camera and lidar calibration algorithm is as follows: Figure 4 As shown, the algorithm includes:

[0071] 1) Solving calibration extrinsic parameters based on a customized ChArUco calibration board:

[0072] a. Image circle center detection: Based on the static calibration site, firstly detect the corner points in the ChArUco calibration board, and solve the transformation matrix from the world coordinate system to the camera coordinate system based on the 3D coordinates of the corner points in the world coordinate system and the 2D coordinates in the pixel coordinate system; then, calculate the 3D coordinates of the circle center in the world coordinate system, and project them onto the pixel coordinate system using the camera pose, camera intrinsic parameters and distortion coefficients in the world coordinate system.

[0073] b. Point Cloud Center Detection: First, extract the ROI point cloud near the ChArUco calibration board and segment the planar point cloud of the calibration board; then transform the point cloud into 2D planar points and sort the point cloud; second, find the row and column values ​​of the lower right corner point, delete the non-face points below, and calculate the initial values ​​of the four center points based on the lower right corner point and the size of the calibration board; finally, use a circle with a radius of 20cm to traverse the positions around the initial center values, and the average value of the center point of the circle with the fewest points inside the circle is the true center point, and then transform it to the LiDAR coordinate system.

[0074] c. Extrinsic parameter optimization: First, the 3D center point in the LiDAR coordinate system is projected onto the image using the initial extrinsic parameters. The 2D points detected in step a and the projected 2D points are then used to form optimized point pairs. The Euclidean distance between these point pairs is used to optimize the calibration parameters. In some embodiments, nonlinear optimization methods can be used for optimization, such as Newton's method, gradient descent method, quadratic programming algorithm, particle swarm optimization algorithm, etc. This invention does not limit the above nonlinear optimization methods.

[0075] In the above steps, the customized ChArUco calibration plate of this invention was used. The calibration plate contains four circular holes, which appear as circles in the image and also as circular holes in the point cloud. Therefore, it can be applied to the calibration of imaging results from cameras and radar. In this invention, by calculating the centers of circles in the image and point cloud, and then projecting the center of the point cloud circles onto the image, the calibration parameters are optimized based on the Euclidean distance between point pairs, providing preliminary calibration extrinsic parameter values.

[0076] 2) Further optimization of external parameters based on 4 ArUco calibration boards:

[0077] a. Segmenting ArUco calibration boards in the image: First, detect the corner points of the ArUco calibration boards and use PNP to solve the pose of each calibration board in the camera coordinate system; then, using the size and pose of the calibration boards, calculate the position of the four corners of each calibration board in the pixel coordinate system and segment the corresponding pixels.

[0078] b. ArUco calibration board in segmented point cloud: First, segment the ground point cloud and non-ground point cloud; second, cluster the non-ground point cloud and obtain the calibration board point cloud; then, based on each clustered point cloud, segment the planar point cloud of each calibration board; finally, project the calibration parameters in 1) onto the image segmented in a, and further optimize the extrinsic parameters of the camera and LiDAR based on the reprojection error.

[0079] In the above steps, using multiple ArUco calibration boards results in higher detection accuracy. By segmenting the ArUco calibration boards in the image and projecting them onto the planar point cloud, extrinsic parameter optimization is performed based on the reprojection error. Introducing multiple different calibration boards in the same scene can effectively reduce the statistical error of calibration extrinsic parameters.

[0080] Based on the above automatic calibration method, by extracting calibration board features from the image and point cloud respectively and optimizing feature point pairs, the entire process does not require manual intervention and achieves good results in both calibration speed and accuracy.

[0081] In some embodiments, the LIDAR and CAMERA joint calibration method provided by the present invention under weak constraints further includes:

[0082] Step S4: Verify the calibration effect, including: acquiring traffic sign features from the telephoto camera, fisheye camera, and LiDAR data, and determining whether the external parameters of the telephoto camera, fisheye camera, and LiDAR are abnormal based on the traffic sign features.

[0083] The extrinsic parameter calibration of telephoto or fisheye cameras is designed based on the traffic signs along the roads in the mining area, and can be performed during daily production and vehicle operation.

[0084] In some embodiments, the flowchart of the calibration extrinsic parameter verification algorithm is as follows: Figure 6 As shown, it includes:

[0085] After acquiring camera (telephoto camera or fisheye camera) and LiDAR data from the perception-driven module, the features of traffic signs on both sides of the road are extracted respectively:

[0086] 1) Image extraction sign:

[0087] a. Traditional Algorithm: First, based on RGB or HSV colors, blue, yellow, and red are segmented to extract corresponding regions; then, median filtering is used to reduce noise in the image, and morphological operations such as erosion and dilation are performed to further improve image quality; next, each contour in the image is calculated, and the minimum rectangle of each contour is calculated to determine whether there is an inclusion relationship between the rectangles. If there is an inclusion relationship, the smaller rectangle is removed; finally, rectangles that meet the requirements are further filtered based on the aspect ratio and contour area of ​​the traffic sign as the traffic sign features extracted from the image;

[0088] b. Deep learning algorithm: Semantic segmentation is performed based on the deeplabV3+ network trained on the currently publicly available cityscapes dataset to segment traffic sign features;

[0089] 2) Point cloud extraction of traffic signs: First, the original point cloud is segmented into ground to remove the influence of the ground; then, in order to remove the influence of retaining walls and other obstacles, horizontal hierarchical clustering and vertical clustering methods are used to cluster obstacles respectively; finally, the traffic sign point cloud features are further screened based on the reflection intensity of the lidar point cloud.

[0090] After extracting point cloud and image features, the original point cloud is projected onto the image based on the calibrated extrinsic parameters. The intersection-over-union (IoU) ratio of the two is calculated, and the extrinsic parameters are optimized based on the IoU ratio to obtain the optimal extrinsic parameters with the highest IoU. Then, to further improve the accuracy of the calibration, multiple frames of data are used for calibration. The optimal extrinsic parameters obtained from each frame are compared with the original extrinsic parameters. If the difference in any parameter exceeds a set threshold, the extrinsic parameters of that frame are considered abnormal. Finally, the number of frames with abnormal extrinsic parameters is counted. If it exceeds the set threshold, the extrinsic parameters of that sensor (telephoto camera or fisheye camera) are determined to be abnormal and need to be recalibrated; otherwise, the extrinsic parameters of that sensor (telephoto camera or fisheye camera) are considered normal.

[0091] By extracting the image features and point cloud features of the sign, calculating the intersection-union ratio of the two after projection, and performing multi-frame threshold judgment, it is possible to effectively estimate whether the external parameters are abnormal. At the same time, the recognition and detection of traffic signs can verify the calibration effect in the daily production environment without introducing new calibration markers, thus improving the convenience of calibration effect verification.

[0092] Based on the above parameter self-calibration algorithm, the calibration effect can be verified online in real time. When the sensor position shifts and the effect is not good, the calibration parameter abnormality can be reported and the automatic calibration can be re-performed, which greatly improves safety and production efficiency in application scenarios.

[0093] Using the above method, combined with vehicle-mounted perception sensors, the calibration extrinsic parameters of the fisheye camera and telephoto camera are first calculated based on the binocular camera extrinsic parameter calibration algorithm. Then, the calibration extrinsic parameters of the telephoto camera and M1 LiDAR are calculated. Finally, the extrinsic parameters of the telephoto camera, fisheye camera, and LiDAR are used for vehicle-mounted perception.

[0094] Please see Figure 8 A joint calibration device 800 for LiDAR and CAMERA under weak constraints includes the following units:

[0095] The determining unit 810 is used to determine the calibration site, including: setting a first calibration plate and a second calibration plate in front of the vehicle, wherein the first calibration plate is a checkerboard QR code calibration plate with round holes, and the second calibration plate is multiple QR code calibration plates, and the second calibration plate is placed on both sides of the first calibration plate;

[0096] The first calibration unit 820 is used to calibrate the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board.

[0097] The second calibration unit 830 is used to calibrate the telephoto camera and lidar of the vehicle, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board;

[0098] Please see Figure 9 , Figure 9A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. Device 900 can be used to implement the joint calibration apparatus 800 of the present disclosure. As shown, device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. Various programs and data required for the operation of device 900 may also be stored in RAM 903. CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904. Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as a keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as a disk, optical disk, etc.; and communication unit 909, such as a network card, modem, wireless transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] Processing unit 901 executes the various methods and processes described above. For example, in some embodiments, the process may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions or steps of the processes described above may be performed. Alternatively, in other embodiments, CPU 901 may be configured to execute the process by any other suitable means (e.g., by means of firmware).

[0100] The functions described above in this invention can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), Systems-on-Chip (SOCs), Load Programmable Logic Devices (CPLDs), and so on.

[0101] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0103] Furthermore, although the actions or steps are described in a specific order, this should be understood as requiring that such actions or steps be performed in the specific order shown or in sequential order, or requiring that all illustrated actions or steps be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0104] Although embodiments of this disclosure have been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms of implementing the claims.

Claims

1. A joint calibration method for LiDAR and CAMERA under weak constraints, comprising: Step S1: Determine the calibration site, including: setting up a first calibration board and a second calibration board in front of the vehicle. The first calibration board is a checkerboard QR code calibration board with round holes, and the second calibration board is multiple QR code calibration boards. The second calibration board is placed on both sides of the first calibration board. Step S2: Calibrate the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board. Step S3: Calibrate the telephoto camera and lidar of the vehicle, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board; The checkerboard QR code calibration plate with round holes is an adjustable-angle ChArUco calibration plate with four round holes. The four round holes are symmetrically arranged at the four corners of the ChArUco calibration plate. The multiple QR code calibration plates are four or eight adjustable-angle ArUco calibration plates. In step S2, the pose of the second calibration board is estimated, and based on the pose of the second calibration board, the calibration extrinsic parameters of the telephoto camera and the fisheye camera are obtained, specifically including: Step S2-1: Acquire images of the second calibration board using a telephoto camera and a fisheye camera respectively, detect the corner points and corner point IDs of the ArUco calibration board in the images, and distinguish the ArUco calibration board based on the corner point IDs; estimate the relative pose of the ArUco calibration board based on the pixel coordinates of the corner points on each ArUco calibration board and the three-dimensional coordinates of the corner points in the world coordinate system. Step S2-2: Based on the corner point IDs obtained in step S2-1, perform ArUco calibration board matching on the two second calibration board images. If more than two corner points are detected in the ArUco calibration board, it is determined as a valid matching calibration board. Then, calculate the extrinsic parameters of the telephoto camera and the fisheye camera based on the relative pose of each valid matching calibration board. Finally, take the average value as the final calibration result. In step S3, solving the calibration extrinsic parameters of the telephoto camera and the lidar based on the first calibration board includes: Step S3-1: Project the 3D center point in the lidar coordinate system onto the image captured by the telephoto camera using the initial extrinsic parameters; Step S3-2: Detect the 2D center point in the image, and form an optimized point pair with the projection point of the 3D center point and the detected 2D center point; Step S3-3: Using the Euclidean distance between the optimized point pairs as the optimization objective, optimize the initial extrinsic parameters using an optimization algorithm.

2. The LIDAR and CAMERA joint calibration method as described in claim 1, characterized in that: The 2D center point is the center of the circle corresponding to the hole in the image of the ChArUco calibration board; the 3D center point is the center of the circle corresponding to the hole in the point cloud of the ChArUco calibration board. The method for obtaining the 2D center point includes: detecting the corner points of the ChArUco calibration board; solving the transformation matrix from the world coordinate system to the camera coordinate system based on the 3D coordinates of the corner points in the world coordinate system and the 2D coordinates in the pixel coordinate system; calculating the 3D coordinates of the center point in the world coordinate system; and projecting the 3D coordinates of the center point into the pixel coordinate system using the camera pose, camera intrinsic parameters, and distortion coefficients in the world coordinate system. The method for obtaining the 3D center point includes: extracting multi-frame point clouds of interest from the area near the ChArUco calibration board; segmenting the planar point cloud of the ChArUco calibration board; transforming the planar point cloud into 2D planar points and sorting the point cloud; querying the row and column values ​​of the lower right corner point and deleting the non-face points below it; calculating the initial value of the center point corresponding to the circular hole in the four point clouds based on the lower right corner point and the size of the ChArUco calibration board; obtaining random points around the initial value of the center point; traversing the points within a circle with a radius of 20cm centered on the random points to obtain the number of points within the circle, and determining the center point of the circle with the fewest points within the circle; for the center point of the circle with the fewest points within each frame of the multi-frame point cloud of interest, calculating the average value of the center points of the multi-frame point cloud of interest, determining it as the final 3D center point, and transforming it to the lidar coordinate system.

3. The LIDAR and CAMERA joint calibration method as described in claim 2, characterized in that: In step S3, the calibration extrinsic parameters of the telephoto camera and LiDAR are optimized based on the second calibration board, including: Step S3-4: Detect the corner points of the ArUco calibration board in the image; use the corner point information and PNP algorithm to solve the pose of each ArUco calibration board in the camera coordinate system; determine the position of the four corners of each ArUco calibration board in the pixel coordinate system according to the size and pose of the calibration board, and segment the pixel region corresponding to each ArUco calibration board. Steps S3-5: Segment the ground point cloud and non-ground point cloud; perform clustering processing on the non-ground point cloud to identify the ArUco calibration board point cloud; for each clustered point cloud cluster, segment the corresponding ArUco calibration board planar point cloud; based on the calibration extrinsic parameters of the telephoto camera and LiDAR obtained by the solution, project the planar point cloud onto the pixel region corresponding to the segmented ArUco calibration board; optimize the calibration extrinsic parameters between the telephoto camera and LiDAR based on the reprojection error between the projection point and the image.

4. The LIDAR and CAMERA joint calibration method as described in claim 3, characterized in that... The method also includes: Step S4: Verify the calibration effect, including: acquiring traffic sign features from the telephoto camera, fisheye camera, and lidar data, and determining whether the external parameters of the telephoto camera, fisheye camera, and lidar are abnormal based on the traffic sign features.

5. The LIDAR and CAMERA joint calibration method as described in claim 4, characterized in that: Traffic sign features include traffic sign image features and traffic sign point cloud features; The process of obtaining traffic sign image features from the data includes: segmenting the image into blue, yellow, and red regions based on RGB or HSV colors to extract regions of the corresponding colors; applying medium-mean-weighted filtering to the segmented regions to reduce noise in the image; performing morphological operations, including erosion and dilation; calculating the contours and their minimum bounding boxes in the image, and removing smaller rectangles that contain each other; and selecting rectangles that meet the criteria based on the aspect ratio and contour area of ​​the traffic sign as traffic sign image features. Alternatively, semantic segmentation can be performed using a deeplabV3+ network trained on the cityscapes dataset to obtain traffic sign image features; The process of obtaining traffic sign point cloud features from the data includes: performing ground segmentation on the original point cloud and removing ground point clouds; using horizontal hierarchical clustering and vertical clustering methods to perform obstacle clustering on the non-ground point cloud; and filtering the point cloud based on the reflection intensity of the point cloud to obtain traffic sign point cloud features.

6. The LIDAR and CAMERA joint calibration method as described in claim 5, characterized in that: Determining whether the external parameters of the telephoto camera, fisheye camera, and LiDAR are abnormal based on the characteristics of the traffic sign includes: Verification is performed using multiple frames of data. The optimal extrinsic parameter and the original extrinsic parameter of each frame are compared. If the difference between the optimal extrinsic parameter and the original extrinsic parameter exceeds a preset threshold, the extrinsic parameter of that frame is marked as abnormal. The number of abnormal extrinsic parameter frames is counted. If it exceeds the preset threshold, it is determined that the extrinsic parameter of the corresponding sensor needs to be recalibrated; otherwise, the extrinsic parameter is determined to be normal. The optimal extrinsic parameters for each frame are obtained as follows: based on the calibrated extrinsic parameters, the point cloud is projected onto the image; the intersection-union ratio (IUR) of the point cloud and the image features is calculated; the extrinsic parameters are optimized according to the IUR, and the extrinsic parameter with the largest IUR is set as the optimal extrinsic parameter.

7. A joint calibration device for LiDAR and CAMERA under weak constraints, comprising: The determination unit is used to determine the calibration site, including: setting a first calibration board and a second calibration board in front of the vehicle, wherein the first calibration board is a checkerboard QR code calibration board with round holes, and the second calibration board is multiple QR code calibration boards; The first calibration unit is used to calibrate the telephoto camera and fisheye camera of the vehicle, including: estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and fisheye camera based on the pose of the second calibration board. The second calibration unit is used to calibrate the vehicle's telephoto camera and lidar, including: solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board, and optimizing the calibration extrinsic parameters of the telephoto camera and lidar based on the second calibration board; The checkerboard QR code calibration plate with round holes is an adjustable-angle ChArUco calibration plate with four round holes. The four round holes are symmetrically arranged at the four corners of the ChArUco calibration plate. The multiple QR code calibration plates are four or eight adjustable-angle ArUco calibration plates. The step of estimating the pose of the second calibration board, and obtaining the calibration extrinsic parameters of the telephoto camera and the fisheye camera based on the pose of the second calibration board, specifically includes: Step S2-1: Acquire images of the second calibration board using a telephoto camera and a fisheye camera respectively, detect the corner points and corner point IDs of the ArUco calibration board in the images, and distinguish the ArUco calibration board based on the corner point IDs; estimate the relative pose of the ArUco calibration board based on the pixel coordinates of the corner points on each ArUco calibration board and the three-dimensional coordinates of the corner points in the world coordinate system. Step S2-2: Based on the corner point IDs obtained in step S2-1, perform ArUco calibration board matching on the two second calibration board images. If more than two corner points are detected in the ArUco calibration board, it is determined as a valid matching calibration board. Then, calculate the extrinsic parameters of the telephoto camera and the fisheye camera based on the relative pose of each valid matching calibration board. Finally, take the average value as the final calibration result. The process of solving the calibration extrinsic parameters of the telephoto camera and lidar based on the first calibration board includes: Step S3-1: Project the 3D center point in the lidar coordinate system onto the image captured by the telephoto camera using the initial extrinsic parameters; Step S3-2: Detect the 2D center point in the image, and form an optimized point pair with the projection point of the 3D center point and the detected 2D center point; Step S3-3: Using the Euclidean distance between the optimized point pairs as the optimization objective, optimize the initial extrinsic parameters using an optimization algorithm.

Images