Calibration method for binocular camera and lidar based on hollowed-out elliptical calibrator

Abstract

The present application relates to a method for calibrating binocular camera and laser radar based on a hollowed-out elliptical calibrator, and belongs to the field of robot vision. A square aluminum plate of the hollowed-out elliptical calibrator is provided with Hamming code square grid marks in a clockwise direction on four corners of one side, has an elliptical through hole in the middle, and the center of the elliptical through hole coincides with the center of the square aluminum plate. The long axis of the elliptical through hole is in the vertical direction. The hardware system of the method for calibrating includes a binocular camera, an upper computer, a 16-line laser radar and a hollowed-out elliptical calibrator. The upper computer has an image processing program, a three-dimensional point cloud processing program and a mathematical calculation program. The operation steps of the method for calibrating are as follows: (1) image feature recognition of the hollowed-out elliptical calibrator, (2) point cloud feature recognition of the hollowed-out elliptical calibrator, (3) calculation of the external parameters of the camera and the 16-line laser radar, and (4) determination of the calibration result. The method for calibrating improves the calibration accuracy and robustness, and is suitable for practical application.

Description

Technical Field

[0001] This invention belongs to the field of robot vision inspection and calibration, specifically relating to a calibration method for a binocular camera and a 16-line lidar based on a hollowed-out ellipse calibrator. Background Technology

[0002] With the increasing demands for robustness and low latency in perception systems from autonomous driving and intelligent robots, multimodal sensor fusion has become a core paradigm for environmental perception. As mainstream perception units, cameras and LiDAR exhibit significant complementary characteristics. Cameras encode semantic information (such as texture and color) through dense pixel mapping, but are limited by distance accuracy and dynamic blur. LiDAR provides centimeter-level geometric perception through active ranging, but is constrained by point cloud sparsity (especially low-beam LiDAR) and lack of texture. Achieving data space alignment between the two through rigid body external parameter calibration is a prerequisite for building a reliable fusion system. However, existing calibration methods still have limitations in terms of adaptability to low-beam LiDAR and robustness without initial conditions, restricting their application on low-cost mobile platforms.

[0003] Binocular camera and LiDAR calibration methods are used in the field of robot vision inspection to estimate the extrinsic parameters of binocular cameras and LiDAR, and are responsible for determining the precise relative position information of the binocular cameras and LiDAR. Binocular cameras aim to acquire image texture information within the field of view, while 3D LiDAR aims to acquire precise positional information within the field of view, in order to analyze machine vision information in specific contexts. Existing binocular camera and LiDAR calibration methods are mainly based on target-specific calibration methods. Target-specific calibration methods typically employ specific calibrators, calibrating extrinsic parameters by extracting the geometric features of the calibrator. The main purpose of processing the calibrator in different ways is to obtain accurate corresponding features, relying on manual calibrators, such as checkerboard patterns, to provide strong geometric constraints.

[0004] The specific purpose of the calibration method is to establish the correspondence between common features in the field of view (“field of view features”), which consists of points, lines, surfaces, and vectors. However, it is difficult for the host computer to extract common view features from the point cloud. First, the point cloud acquired by the lidar is sparse, making it difficult to detect the corresponding features; second, the presence of point cloud noise affects the accuracy of feature fitting. Therefore, special calibration devices are usually used to assist in feature extraction. The method of calibrating the extrinsic parameters using a specific calibration device is called the target-specific method, which relies on a manual calibrator to provide strong geometric constraints. Taking the classic checkerboard method as an example, it uses the correspondence of plane normal vectors or the 2D-3D projection of corner points to solve for extrinsic parameters, achieving centimeter-level accuracy in a controlled environment. However, such methods face a double challenge: calibrator features are prone to degradation in low-line point clouds due to insufficient sampling, such as broken point clouds at the edge of the checkerboard and incomplete target features under 16-line radar; and having only a single feature constraint, such as only planes or corner points, can lead to optimization getting trapped in local optima. Summary of the Invention

[0005] For low-beam lidar applications, this invention provides a hollowed-out ellipse calibrator while ensuring calibration accuracy and stability. Simultaneously, it provides a calibration method for binocular cameras and lidar based on the hollowed-out ellipse calibrator.

[0006] The hollowed-out elliptical calibrator for binocular camera and lidar calibration includes a square aluminum plate 4; Hamming code grid (ArUco) markings are respectively provided on the four corners of one side of the square aluminum plate 4, and the four Hamming code grid markings distributed from top to bottom in a clockwise direction are the first marking 41, the second marking 42, the third marking 43 and the fourth marking 44; an elliptical through hole 45 is opened in the middle of the square aluminum plate 4, and the center of the elliptical through hole 45 coincides with the center of the square aluminum plate 4; the major axis of the elliptical through hole 45 is located in the vertical direction, and the minor axis of the elliptical through hole 45 is located in the horizontal direction.

[0007] The technical solution for a further hollowed-out ellipse calibrator is as follows:

[0008] The square aluminum plate 4 has dimensions of 1 meter × 1 meter; the Hamming code grid marker has dimensions of 0.1 meter × 0.1 meter, see [link / reference]. Figure 2 The internal black border of the Hamming code grid marking has a size of 0.08 meters × 0.08 meters; the major axis of the elliptical through hole 45 is 0.8 meters and the minor axis is 0.4 meters.

[0009] A calibration method for a binocular camera and a lidar based on a hollow elliptical calibrator is described. The hardware platform of the calibration method includes a binocular camera 1, a 16-line lidar 2, a hollow elliptical calibrator, a host computer, and a double-layer support 3. The host computer is a computer-controlled system. The binocular camera has a resolution of 2208×1242. The binocular camera 1 is located on the upper layer of the double-layer support 3, and the 16-line lidar 2 is located on the lower layer of the double-layer support 3. One side of the hollow elliptical calibrator with a Hamming code grid pattern faces the binocular camera 1 and the 16-line lidar 2. The distance between the binocular camera 1 and the hollow elliptical calibrator is 2.5 meters, forming a calibration test space.

[0010] The calibration procedure is as follows:

[0011] (1) Image feature recognition of the hollowed-out ellipse calibrator (1.1) Obtain the coordinates of the four center pixels

[0012] The left and right cameras of the binocular camera 1 simultaneously capture images of the hollow elliptical calibrator and transmit the captured left and right images to the host computer. The host computer uses the left camera as the pixel coordinate center to identify the center of the markers of the four Hamming code squares on the hollow elliptical calibrator in the left image. The host computer calculates the coordinates of the four center pixels corresponding to the first marker 41, the second marker 42, the third marker 43, and the fourth marker 44.

[0013] (1.2) Obtain the coordinates of the 3D points of the 4 cameras

[0014] Based on the principle of mimicking the human eye's judgment of object distance, the host computer constructs a depth image by comparing the differences between the left and right images. The depth image is an image that stores distance information. In the depth image, the left camera of the binocular camera 1 is used as the center of the three-dimensional coordinate system to establish the left camera three-dimensional coordinate system of the binocular camera 1. In the depth image, the coordinates of the four center pixels are converted into the coordinates of four camera three-dimensional points corresponding to the first mark 41, the second mark 42, the third mark 43, and the fourth mark 44 in the left camera three-dimensional coordinate system.

[0015] (2) Obtain the coordinates of four laser 3D points for point cloud features.

[0016] While photographing the hollow elliptical calibrator in step (1.1), the hollow elliptical calibrator is observed using a 16-line lidar 2 to obtain a three-dimensional point cloud of the calibration test space corresponding to the image on the left screen at the same time.

[0017] The host computer establishes a three-dimensional coordinate system for the 16-line lidar 2 with the 16-line lidar 2 as the three-dimensional coordinate center;

[0018] The following operations are performed in the 2D coordinate system of the 16-line lidar: a cuboid region containing the 3D point cloud of the hollow elliptical calibrator is set in the host computer, the 3D point cloud in the cuboid region is filtered to remove the messy 3D point cloud, and the 3D point cloud of the hollow elliptical calibrator is obtained.

[0019] The three-dimensional point cloud of the hollowed-out elliptical calibrator is subjected to plane fitting, and the plane-fitted three-dimensional point cloud that meets the plane fitting result is selected.

[0020] Edge extraction is performed on the planar fitted 3D point cloud to obtain the square edge 3D point cloud of the hollowed-out elliptical calibrator and the elliptical edge 3D point cloud of the elliptical through hole 45;

[0021] A straight line is fitted to the left edge of the three-dimensional point cloud of the square edge to obtain the left spatial straight line direction vector of the left edge straight line; an ellipse is fitted to the three-dimensional point cloud of the elliptical edge of the elliptical through hole 45 to obtain the three-dimensional coordinates of the center of the elliptical through hole 45 and the spatial straight line direction vector of the minor axis of the ellipse.

[0022] When the spatial straight line direction vector of the minor axis of the ellipse and the spatial straight line direction vector of the left space are orthogonal, according to the size of the hollowed-out ellipse calibrator, the coordinates of the four laser three-dimensional points corresponding to the first mark 41, the second mark 42, the third mark 43 and the fourth mark 44 in the three-dimensional coordinate system of the 16-line lidar 2 are obtained by the host computer based on the three-dimensional coordinates of the center of the ellipse through hole 45, the spatial straight line direction vector of the minor axis of the ellipse and the spatial straight line direction vector of the left space.

[0023] (3) Calculate the external parameters between the binocular camera and the 16-line lidar.

[0024] (3.1) Based on the coordinates of the four camera 3D points in the 3D coordinate system of the left camera of the binocular camera 1 obtained in step (1) and the coordinates of the four laser 3D points in the 3D coordinate system of the 16-line lidar 2 obtained in step (2), the coordinates of the four camera 3D points are defined as the feature point set of the left camera of the binocular camera 1. The coordinates of the four laser 3D points are defined as the feature point set of the 16-line lidar 2.

[0025] (3.2) Using the feature point set of the left camera of the binocular camera 1 Calculate the centroid coordinates μ of the left camera of stereo camera 1. c Utilizing the feature point set of 16-line lidar 2 Calculate the centroid coordinates μ of the 16-line lidar 2. l ;

[0026] (3.3) Using the centroid coordinates μ of the left camera of the binocular camera 1 c Calculate the decentralized coordinates of the left camera of stereo camera 1. Using the centroid coordinates μ of the 16-line lidar 2 l Calculate the decentralized coordinates of the 16-line lidar 2

[0027] (3.4) Calculate the external parameters between the binocular camera and the 16-line lidar based on singular value decomposition; the external parameters are the rotation matrix R and translation matrix t of the three-dimensional coordinate system of the 16-line lidar 2 relative to the three-dimensional coordinate system of the left camera of the binocular camera 1.

[0028] (4) Determine the calibration results

[0029] Based on the left-side image captured by the binocular camera 1 in step (1), the three-dimensional point cloud of the calibration test space observed by the 16-line lidar 2 in step 2, and the external parameters obtained in step 3, the three-dimensional point cloud of the calibration test space is reprojected onto the left-side image. When the edge of the hollowed-out elliptical calibrator in the left-side image coincides with the three-dimensional point cloud of the edge of the hollowed-out elliptical calibrator in the calibration test space, the calibration result meets the requirements, that is, the calculated external parameters between the binocular camera 1 and the 16-line lidar 2 are accurate. When the deviation between the edge of the hollowed-out elliptical calibrator in the left-side image and the three-dimensional point cloud of the edge of the hollowed-out elliptical calibrator in the calibration test space is more than 10 pixels, the calibration result does not meet the requirements, that is, the external parameters between the binocular camera 1 and the 16-line lidar 2 are inaccurate.

[0030] The further calibration operation technical solution is as follows:

[0031] In step (1.1), when the left and right cameras of the binocular camera 1 simultaneously capture images of the hollow elliptical calibrator, the left camera captures at least 10 left-view images and the right camera captures at least 10 right-view images; the 16-line lidar 2 observes the hollow elliptical calibrator and obtains the calibration test space three-dimensional point cloud corresponding to the left-view image at the same time, thus obtaining at least 10 frames of calibration test space three-dimensional point cloud.

[0032] When the left and right cameras of the binocular camera 1 simultaneously capture images of the hollow elliptical calibrator to obtain 10 left-side images and 10 right-side images, repeat steps (1.2) and (2) 10 times to obtain 10 sets of 3D point coordinates of 4 cameras and 3D point coordinates of 4 lasers.

[0033] In step (1.2), the principle of mimicking the human eye's judgment of object distance refers to using Hamming code grid markers as target features and constructing a depth constraint equation based on the parallax principle:

[0034] z = f·B / d (1)

[0035] In equation (1), f is the focal length, which refers to the distance from the optical center of the camera lens to the imaging plane; B is the baseline distance, which refers to the horizontal distance between the optical centers of the left and right lenses in a binocular camera; and d is the normalized parallax, which refers to the difference in pixel coordinates of the same object in the left and right images.

[0036] In step (2), the specific operation steps are as follows:

[0037] Delete other 3D point clouds in the environment, and retain the 3D point cloud information of the hollowed-out ellipse calibrator. The specific filtering operation is as follows:

[0038] Use the host computer to view the acquired 3D point cloud information of all hollow elliptical calibrators. Set a 1.5m × 1.5m × 1m cuboid area with the center of the hollow elliptical calibrator as the center, and the cuboid area contains the complete hollow elliptical calibrator. Delete the 3D point cloud of the calibration test space outside the cuboid area, and then perform statistical filtering to filter out the noisy point cloud that is 10 cm away from the 3D point cloud of the hollow elliptical calibrator.

[0039] The steps to obtain the 3D point cloud features of the hollowed-out ellipse calibrator are as follows:

[0040] The 3D point cloud of the hollowed-out elliptical calibrator is subjected to plane fitting using a random sampling consensus algorithm. This random sampling consensus algorithm is a model fitting algorithm used to estimate the parameters of a mathematical model in data containing noise or outliers. The resulting fitted plane 3D point cloud is then filtered, and the filtered fitted plane 3D point cloud is extracted using a rolling sphere method to obtain square edge 3D point clouds and elliptical edge 3D point clouds. A virtual sphere with a radius of 20 cm is rolled across the point cloud surface; the areas the virtual sphere cannot pass through are the edges. This ensures that the number of points in the elliptical edge 3D point cloud is greater than 20.

[0041] The left edge of the 3D point cloud of the square edge is fitted with a straight line using the random sampling consensus algorithm to obtain the spatial direction vector of the left edge straight line; the 3D point cloud of the ellipse edge is fitted with an ellipse using the random sampling consensus algorithm to obtain the 3D coordinates of the ellipse center and the spatial direction vector of the ellipse minor axis.

[0042] When the obtained ellipse minor axis spatial line direction vector and the left edge spatial line direction vector are orthogonal, the coordinates of the four laser three-dimensional points corresponding to the first mark 41, the second mark 42, the third mark 43, and the fourth mark 44 in the three-dimensional coordinate system of the 16-line lidar are calculated from the three-dimensional coordinates of the ellipse center, the ellipse minor axis spatial line direction vector, and the left edge spatial line direction vector.

[0043] In step (2), when the number of points in the obtained 3D point cloud of the ellipse edge is less than 20, the 3D points of the ellipse edge are manually selected in the host computer to ensure that the number of points in the obtained 3D point cloud of the ellipse edge is greater than 20.

[0044] In step (3), the specific operations for calculating the external parameters between the binocular camera and the 16-line lidar are as follows:

[0045] (3.1). Solving for the rotation and translation matrices using feature point sets.

[0046] The 3D point coordinates of the four cameras are defined as the feature point set of the left camera of the stereo camera 1, i.e. The coordinates of the four laser 3D points are defined as the feature point set of the 16-line lidar 2, i.e.

[0047] Formula (2) is used to solve for the rigid body transformation rotation matrix R∈SO(3) and translation matrix t∈R. 3 Formula (2) is as follows:

[0048]

[0049] In equation (2), ε i This is a Gaussian noise term. This represents the coordinates of the i-th corresponding 3D point in the feature point set of the left camera of binocular camera 1. Let R represent the coordinates of the i-th corresponding laser 3D point in the feature point set of the 16-line LiDAR 2, R represent the rotation matrix from the 3D coordinate system of the left camera of the binocular camera 1 to the 3D coordinate system of the 16-line LiDAR 2, and t represent the translation matrix from the 3D coordinate system of the left camera of the binocular camera 1 to the 3D coordinate system of the 16-line LiDAR 2.

[0050] (3.2). Calculate the centroid coordinates of the point set.

[0051] Using the feature point set of the left camera of the binocular camera 1 The centroid coordinates of the feature point set of the stereo camera are calculated using formula (3) as follows:

[0052]

[0053] In formula (3), μ c The centroid coordinates of the feature point set in the three-dimensional coordinate system of the left camera of binocular camera 1;

[0054] Utilizing the feature point set of 16-line lidar 2 The centroid coordinates of the feature point set of the 16-line lidar are calculated using formula (4) as follows:

[0055]

[0056] In formula (4), μ l The centroid coordinates of the feature point set in the 2D coordinate system of the 16-line lidar;

[0057] (3.3). Obtaining decentralized coordinates

[0058] Using the centroid coordinates μ c and μ l To eliminate the influence of the translation vector t on the estimation of the rotation matrix R, the decentralized coordinate formulas (5) for the left camera and (6) for the 16-line lidar are as follows:

[0059]

[0060] In formula (5), Let be the i-th decentralized coordinate of the feature point set of the left camera in the three-dimensional coordinate system of the binocular camera 1; in formula (6), The i-th decentralized coordinate of the feature point set in the 2D coordinate system of the 16-line lidar;

[0061] (3.4). Singular value decomposition calculates the rotation matrix R and translation vector t.

[0062] First, construct the covariance matrix H, and calculate the covariance matrix of the decentralized point set using the formula (7) as follows:

[0063]

[0064] Decompose the covariance matrix H of formula (7) into three matrices H = U∑V T The product of these two equations yields formula (8), which is the singular value decomposition formula. Formula (8) is as follows:

[0065] H=U∑V T (8)

[0066] In equation (8), U,V∈R 3×3 For each matrix, ∑=diag(σ1,σ2,σ3) is an orthogonal matrix (σ1≥σ2≥σ3≥0);

[0067] The transpose of the two orthogonal matrices obtained by decomposition using formula (8) is R = VU T Solve for the rotation matrix R. The formula for the rotation matrix R is as follows:

[0068] R = VU T (9)

[0069] Then use the centroid coordinates μ obtained from formula (5) c The centroid coordinates μ obtained from formula (6) l Solve for the translation vector t. The formula for the translation vector t (10) is as follows:

[0070] t = μ c -Rμ l (10)

[0071] Repeat steps (1) and (2) to obtain 10 sets of 3D point coordinates for 4 cameras and 3D point coordinates for 4 lasers. Through the operation in step (3), calculate the external parameters between 10 sets of binocular camera 1 and 16-line lidar 2.

[0072] Singular value decomposition is used to provide a closed-form global optimal solution to the 3D-3D point set registration problem. The solution process does not depend on the initial value of the iteration, avoiding the local convergence caused by the sensitivity of the initial value to gradient descent or random sampling consensus algorithms, and is suitable for the calibration scenario of the calibrator.

[0073] In step (4), the specific operations are as follows:

[0074] Input the intrinsic parameter matrix, distortion coefficients and extrinsic parameters obtained in step 3 into the host computer, transform the 3D point cloud in the 3D coordinate system of the 16-line LiDAR system 2 into 2D pixel coordinates in the left camera pixel coordinate system of the binocular camera 1, and then project it onto the left image; use 10 sets of extrinsic parameters between the binocular camera 1 and the 16-line LiDAR 2 to obtain 10 projected images.

[0075] When the edge of the hollowed-out elliptical calibrator in the left image coincides with the three-dimensional point cloud of the edge of the hollowed-out elliptical calibrator in the calibration test space, the calibration result meets the requirements, that is, the calculated external parameters between the binocular camera 1 and the 16-line lidar 2 are accurate; the image with the smallest pixel deviation among the 10 projected images is taken as the calibration result.

[0076] When the deviation of the three-dimensional point cloud between the edge of the hollowed-out elliptical calibrator in the left image and the edge of the hollowed-out elliptical calibrator in the calibration test space is more than 10 pixels, the calibration result does not meet the requirements, that is, the calculated external parameters between the binocular camera 1 and the 16-line lidar 2 are inaccurate.

[0077] The beneficial technical effects of this invention are reflected in the following aspects:

[0078] 1. The hollow elliptical calibrator of this invention contains distinct visual and geometric features, which is beneficial for feature extraction of point clouds from binocular cameras and 16-line LiDAR. The square aluminum plate has an elliptical hole, with the center of the square coinciding with the center of the ellipse. The elliptical hole provides clear three-dimensional point cloud features for the LiDAR. Compared to a circular hole, the elliptical hole has a clear directionality and can fit the correct ellipse even under larger, sparse point cloud conditions. The surface of the hollow elliptical calibrator uses a high-reflectivity material, making it easy to extract the LiDAR point cloud from the calibration device. Four Hamming code square markers with different IDs are assigned to the four corners of the square aluminum plate for visual feature extraction, making the calibration method simple, effective, and ensuring accurate results. The image features of the hollow elliptical calibrator are finely extracted using the four Hamming code square markers. Target features and corresponding feature point sets in the image are extracted based on the position of the Hamming code square markers. The three-dimensional coordinates of the Hamming code square markers in the binocular camera's left camera coordinate system are calculated using the parallax principle. This method can effectively extract target features from the image to obtain three-dimensional coordinate information, providing a specific feature point set for subsequent singular value decomposition calculations.

[0079] 2. The calibration algorithm of this invention is adapted to low-beam LiDAR. This invention fully considers the characteristics of low-beam LiDAR, where the 3D point cloud is relatively sparse. Traditional binocular camera and LiDAR calibration methods may not be suitable for low-beam LiDAR or may produce poor external parameter calibration results. Compared to traditional methods, this invention does not increase the complexity of the calibrator and can effectively extract the calibrator's feature constraints from the image and point cloud. It has good results for sparse point clouds. By utilizing the straight edge and elliptical feature constraints of the calibrator, the external parameters between sensors can be accurately determined, achieving centimeter-level accuracy and noise resistance under low-beam LiDAR conditions. Using the obtained feature point set, the external parameters of the camera and LiDAR are calculated based on singular value decomposition. Maintaining a single pose effectively increases the convenience and efficiency of the calibration process, while also providing high accuracy and stability. Singular value decomposition provides a closed-form global optimal solution for the 3D-3D point set registration problem. Its solution process does not depend on initial values ​​during iteration, avoiding the risk of local convergence due to initial value sensitivity in methods such as gradient descent or random sampling consensus algorithms. It is particularly suitable for calibration scenarios without prior extrinsic parameters. Using the calibration method of this invention, calibration was performed in a virtual simulation environment using 10 sets of point clouds and 10 sets of images corresponding to the same pose. The results were summed and divided by 10 to obtain the average value, which was then compared with the theoretical value. The average absolute error of the rotation matrix converted to Euler angles was 0.499°, and the average absolute error of the translation term was 0.007 meters. Attached Figure Description

[0080] Figure 1 This is a diagram of the actual calibration and testing space environment in an embodiment of the present invention;

[0081] Figure 2 This is a design drawing of the hollowed-out elliptical calibrator in an embodiment of the present invention;

[0082] Figure 3 This is a rigid body transformation diagram of the binocular camera and the 16-line lidar in an embodiment of the present invention;

[0083] Figure 4 This is a schematic diagram of the acquired visual features in an embodiment of the present invention, showing that the binocular camera correctly found the four Hamming code square markers and marked them with red boxes;

[0084] Figure 5 This is a flowchart of the feature point cloud processing for extracting the hollow elliptical calibrator in an embodiment of the present invention. (a) represents the observed three-dimensional point cloud of the calibration test space, (b) represents the extracted three-dimensional point cloud of the hollow elliptical calibrator, and (c) represents the three-dimensional point cloud of the square edge and the three-dimensional point cloud of the elliptical edge obtained after filtering.

[0085] Figure 6This is a diagram showing the effect of reprojecting a virtual simulation experiment using calculated external parameters in an embodiment of the present invention.

[0086] Figure 7 The calibration device used in the comparative examples of this invention, which is the MATLAB 2022b and Autoware method, is an 8×4 checkerboard with a single grid of 25 cm × 25 cm. Detailed Implementation

[0087] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0088] Example 1

[0089] See Figure 2 The hollowed-out elliptical calibrator for calibrating binocular cameras and lidar includes a square aluminum plate 4; Hamming code grid markings are provided on the four corners of one side of the square aluminum plate 4, and the four Hamming code grid markings distributed clockwise from top to bottom are the first marking 41, the second marking 42, the third marking 43 and the fourth marking 44; the first marking 41 represents the original Hamming code number 0, the second marking 42 represents the original Hamming code number 1, the third marking 43 represents the original Hamming code number 2, and the fourth marking 44 represents the original Hamming code number 3.

[0090] An elliptical through-hole 45 is formed in the center of the square aluminum plate 4, and the center of the elliptical through-hole 45 coincides with the center of the square aluminum plate 4. The major axis of the elliptical through-hole 45 is located in the vertical direction, and the minor axis of the elliptical through-hole 45 is located in the horizontal direction. The major axis of the elliptical through-hole 45 is 0.8 meters, and the minor axis is 0.4 meters.

[0091] The square aluminum plate 4 has a size of 1 meter × 1 meter; a plane coordinate system is formed by two adjacent sides, with the origin being the intersection of the two adjacent sides at the upper left corner when the square aluminum plate 4 is viewed from the front, the x-axis being the side horizontally to the right of the origin, and the y-axis being the side vertically downward from the origin.

[0092] The Hamming code grid markers are 0.1m x 0.1m in size. See [reference needed]. Figure 2 The inner black border of the Hamming code grid marker has a size of 0.08 meters × 0.08 meters.

[0093] The design of the hollowed-out ellipse calibrator consists of two steps: first, an image feature recognition module from a binocular camera, which can accurately identify Hamming code grid markings; and second, a three-dimensional point cloud feature recognition module from a 16-line lidar, which can accurately identify elliptical through-holes of 45.

[0094] (1) Image feature recognition module of binocular camera

[0095] The camera extracts image features using Hamming code grid markers. Since at least four different 3D point coordinates are required in singular value decomposition, four different Hamming code markers with different numbers are used as target features in the image. The white edges and black borders of the Hamming code grid are used for contrast to improve the recognition rate of the Hamming code grid markers.

[0096] In image feature extraction, binocular cameras obtain RGB images, which contain rich texture and color information but lack depth information. To ensure the accuracy and robustness of image features, Hamming code grid marking, which is widely used in the field of target recognition, is used in the calibration device.

[0097] (2) 3D point cloud feature recognition module of 16-line lidar

[0098] Under the observation of a 16-line lidar, the elliptical through-hole 45 of the hollow elliptical calibrator can still obtain more than 20 3D point clouds of elliptical edges, providing suitable point cloud features for recognition under sparse point cloud conditions. The surface of the hollow elliptical calibrator uses a high reflectivity material, which makes it easy to extract the lidar point cloud of the calibration device. The left vertical side of the square aluminum plate 4 is perpendicular to the minor axis of the elliptical through-hole 45, which constrains the ellipse fitting and improves the fitting accuracy. After ellipse fitting, the coordinates of four different 3D points at the same position as the image features can be calculated from the known dimensions of the hollow elliptical calibrator.

[0099] The spatial vector obtained by fitting the edge straight line is orthogonally verified with the axis vector obtained by fitting the ellipse. Since the major and minor axes of the ellipse have clear directions, the 16-line lidar 3D coordinate point set corresponding to the 3D coordinate point set of the binocular camera can be obtained from the coordinates of the ellipse center point, the size of the calibration device, and the size of the Hamming code mark.

[0100] Example 2

[0101] A method for calibrating binocular cameras and lidar based on a square-hollowed-out elliptical calibrator, see [link to relevant documentation]. Figure 1 The hardware system for implementing the calibration method includes a left camera, a right camera, a host computer, a 16-beam lidar, and the hollow ellipse calibrator of Embodiment 1, which are parallel to each other. The left camera and the right camera constitute a binocular camera 1, and the binocular camera 1 and the 16-beam lidar 2 are connected to the host computer.

[0102] See Figure 1 Based on the design of the hollowed-out ellipse calibrator calibrated by the binocular camera and lidar, a system was built. Figure 1 The calibration test space shown has a binocular camera 1 and a 16-line lidar 2 located on a double-layer support 3. The binocular camera 1 is 0.3 meters above the 16-line lidar, and the hollow elliptical calibrator is 2.5 meters away from the binocular camera 1.

[0103] The host computer contains image processing programs, 3D point cloud processing programs, and mathematical calculation programs.

[0104] The host computer, also known as the upper-level computer, contains MATLAB 2020b, Visual Studio Code, CloudCompare, and Visual Studio 2022 programs. A calibration test space is set up in the virtual simulation, with a distance of 250cm between the double-layer support 3 and the hollowed-out elliptical calibrator.

[0105] The distance between the lens of the left camera and the lens of the right camera is 120 mm; the field of view of the binocular camera 1 is 110° horizontal angle of view × 70° vertical angle of view × 120° diagonal angle of view.

[0106] The 16-line lidar 2 has a measurement range of 0.4 meters to 150 meters, an accuracy of ±2 centimeters, a vertical field of view of ±15°, a horizontal angular resolution of 0.1° to 0.4°, and 16 channels.

[0107] See Figure 3 , Figure 3 This represents the rigidity relationship between the calibration of the binocular camera and the 16-line LiDAR. The calibration process is to calculate the rotation matrix R and translation matrix t from the 16-line LiDAR coordinate system to the left camera coordinate system of the binocular camera.

[0108] The operation steps of the binocular camera and lidar calibration method based on the hollowed-out ellipse calibrator are as follows:

[0109] (1) Image feature recognition of the hollowed-out ellipse calibrator (1.1) Obtain the coordinates of the four center pixels

[0110] See Figure 4 The left and right cameras of the binocular camera 1 simultaneously capture images of the hollow elliptical calibrator. The left camera captures 10 frames of left-side images, and the right camera captures 10 frames of right-side images. The captured left-side and right-side images are then transmitted to the host computer. The host computer uses the left camera as the pixel coordinate center to identify the center of the markers of the four Hamming code squares on the hollow elliptical calibrator in the left-side image. The host computer calculates the 10 sets of four center pixel coordinates corresponding to the first marker 41, the second marker 42, the third marker 43, and the fourth marker 44.

[0111] (1.2) Obtain the coordinates of the 3D points of the 4 cameras

[0112] The host computer constructs a depth image by comparing the differences between the left and right images, based on the principle of human eyes judging the distance of objects. The depth image is an image that stores distance information.

[0113] The principle of mimicking the human eye's judgment of object distance refers to using Hamming code grid markers as target features and constructing a depth constraint equation based on the principle of parallax:

[0114] z = f·B / d (1)

[0115] In the depth image, with the left camera of the stereo camera 1 as the three-dimensional coordinate center, a three-dimensional coordinate system for the left camera of the stereo camera 1 is established. The average depth z = 2.4997 meters to the Hamming code grid markers in the 10 sets of images can be obtained, and the theoretical depth is z = 2.5 meters. In the depth image, the coordinates of the four center pixels are transformed into the coordinates of 10 sets of four camera three-dimensional points corresponding to the first marker 41, the second marker 42, the third marker 43, and the fourth marker 44 in the three-dimensional coordinate system of the left camera.

[0116] (2) Obtain the coordinates of four laser 3D points for point cloud features.

[0117] See Figure 5 In step (1.1), while photographing the hollow elliptical calibrator, a 16-line lidar 2 is used to observe the hollow elliptical calibrator to obtain the calibration test space three-dimensional point cloud corresponding to the left image at the same time, resulting in 10 frames of calibration test space three-dimensional point cloud.

[0118] The host computer establishes a three-dimensional coordinate system for the 16-line lidar 2 with the 16-line lidar 2 as the three-dimensional coordinate center;

[0119] See Figure 5 In step b, the following operations are performed in the 3D coordinate system of the 16-line LiDAR 2: Use the host computer to view the 3D point cloud information of all the hollowed-out elliptical calibrators. Set a cuboid region of 1.5m × 1.5m × 1m with the center of the hollowed-out elliptical calibrator as the center, and the cuboid region contains the complete hollowed-out elliptical calibrator; delete the 3D point cloud of the calibration test space outside the cuboid region, and then perform statistical filtering to filter out the noise point cloud that is 10cm away from the 3D point cloud of the hollowed-out elliptical calibrator.

[0120] The 3D point cloud of the hollowed-out elliptical calibrator is subjected to plane fitting using a random sampling consensus algorithm, and the 3D point clouds that conform to the plane fitting results are selected. The random sampling consensus algorithm is a model fitting algorithm used to estimate the parameters of a mathematical model in data containing noise or outliers.

[0121] See Figure 5In step c, edge extraction is performed on the planar fitted 3D point cloud. The filtered planar fitted 3D point cloud is extracted using a rolling ball method to obtain square edge 3D point clouds and elliptical edge 3D point clouds. A "virtual sphere" with a radius of 20 cm is rolled on the point cloud surface; the area that the virtual sphere cannot pass through is the edge. This ensures that the number of points in the obtained elliptical edge 3D point cloud is greater than 20. If the number of points in the obtained elliptical edge 3D point cloud is less than 20, the elliptical edge 3D points are manually selected in the host computer to ensure that the number of points in the obtained elliptical edge 3D point cloud is greater than 20. This yields square edge 3D point clouds of the hollowed-out elliptical calibrator and elliptical edge 3D point clouds of the elliptical through-hole 45.

[0122] The left edge of the three-dimensional point cloud of the square edge is fitted with a straight line using a random sampling consensus algorithm to obtain the left spatial straight line direction vector of the left edge straight line; the elliptical edge of the elliptical through hole 45 is fitted with an elliptical line using a random sampling consensus algorithm to obtain the three-dimensional coordinates of the center of the elliptical through hole 45 and the spatial straight line direction vector of the minor axis of the ellipse.

[0123] When the spatial straight line direction vector of the minor axis of the ellipse and the spatial straight line direction vector of the left space are orthogonal, according to the size of the hollowed-out ellipse calibrator, the three-dimensional coordinates of the center of the ellipse through hole 45, the spatial straight line direction vector of the minor axis of the ellipse, and the spatial straight line direction vector of the left space are calculated by the host computer to obtain the coordinates of 10 sets of 4 laser three-dimensional points corresponding to the first mark 41, the second mark 42, the third mark 43, and the fourth mark 44 in the three-dimensional coordinate system of the 16-line lidar 2.

[0124] (3) Calculate the external parameters between the binocular camera and the 16-line lidar (3.1). Solve for the rotation and translation matrices using feature point sets.

[0125] The 10 sets of 3D point coordinates from the four cameras are defined as the feature point set of the left camera of the stereo camera 1, i.e. The coordinates of 10 sets of 4 laser 3D points are defined as the feature point set of the 16-line lidar 2, i.e.

[0126] Formula (2) is used to solve for the rigid body transformation rotation matrix R∈SO(3) and translation matrix t∈R. 3 Formula (2) is as follows:

[0127]

[0128] In equation (2), ∈ i This is a Gaussian noise term. This represents the coordinates of the i-th corresponding 3D point in the feature point set of the left camera of binocular camera 1. Let R represent the coordinates of the i-th corresponding laser 3D point in the feature point set of the 16-line LiDAR 2, R represent the rotation matrix from the 3D coordinate system of the left camera of the binocular camera 1 to the 3D coordinate system of the 16-line LiDAR 2, and t represent the translation matrix from the 3D coordinate system of the left camera of the binocular camera 1 to the 3D coordinate system of the 16-line LiDAR 2.

[0129] (3.2). Calculate the centroid coordinates of the point set.

[0130] Using the feature point set of the left camera of the binocular camera 1 The centroid coordinates of the feature point set of the stereo camera are calculated using formula (3) as follows:

[0131]

[0132] In formula (3), μ c The centroid coordinates of the feature point set in the three-dimensional coordinate system of the left camera of binocular camera 1;

[0133] Utilizing the feature point set of 16-line lidar 2 The centroid coordinates of the feature point set of the 16-line lidar are calculated using formula (4) as follows:

[0134]

[0135] In formula (4), μ l The centroid coordinates of the feature point set in the 2D coordinate system of the 16-line lidar;

[0136] (3.3). Obtaining decentralized coordinates

[0137] Using the centroid coordinates μ c and μ l To eliminate the influence of the translation vector t on the estimation of the rotation matrix R, the decentralized coordinate formulas (5) for the left camera and (6) for the 16-line lidar are as follows:

[0138]

[0139] In formula (5), Let be the i-th decentralized coordinate of the feature point set of the left camera in the three-dimensional coordinate system of the binocular camera 1; in formula (6), The i-th decentralized coordinate of the feature point set in the 2D coordinate system of the 16-line lidar;

[0140] (3.4). Singular value decomposition calculates the rotation matrix R and translation vector t.

[0141] First, construct the covariance matrix H, and calculate the covariance matrix of the decentralized point set using the formula (7) as follows:

[0142]

[0143] Decompose the covariance matrix H of formula (7) into three matrices H = U∑V T The product of these two equations yields formula (8), which is the singular value decomposition formula. Formula (8) is as follows:

[0144] H=U∑V T (8)

[0145] Where U,V∈R 3×3 For each matrix, ∑=diag(σ1,σ2,σ3) is an orthogonal matrix (σ1≥σ2≥σ3≥0);

[0146] The transpose of the two orthogonal matrices obtained by decomposition using formula (8) is R = VU T Solve for the rotation matrix R. The formula for the rotation matrix R is as follows:

[0147] R = VU T (9)

[0148] Then use the centroid coordinates μ obtained from formula (5) c The centroid coordinates μ obtained from formula (6) l Solve for the translation vector t. The formula for the translation vector t (10) is as follows:

[0149] t = μ c -Rμ l (10)

[0150] Repeat steps (1) and (2) to obtain 10 sets of 3D point coordinates for 4 cameras and 4 3D point coordinates for 4 lasers. Through the operation in step (3), calculate the 10 sets of external parameters between the binocular camera 1 and the 16-line lidar 2. The calculation results are given in Tables 1 and 2 below. Singular value decomposition is used to provide a closed-form global optimal solution for the 3D-3D point set registration problem. The solution process does not depend on the initial value of the iteration, avoiding the local convergence caused by the sensitivity of the initial value to gradient descent or random sampling consensus algorithms, and is suitable for the calibration scenario of the calibrator.

[0151] Table 1. Calibration results of the rotation term

[0152] Unit ° roll Pitch yaw Mean Absolute Error Reference value 0 0 0 - Calibration value 0.592 -0.328 0.576 0.499

[0153] Table 2. Calibration results of the translation term

[0154] Unit m x y z Mean Absolute Error Reference value -0.004 -0.15 -0.2 - Calibration value -0.016 -0.157 -0.201 0.007

[0155] The reference value is the theoretical coordinate position set in the virtual simulation, and the calibration value is the calibration result obtained by using the calibration method of the present invention. Table 1 shows that the average absolute error of the rotation term calculated by formula (9) in this embodiment 2 is within 0.5°. Table 2 shows that the average absolute error of the translation term calculated by formula (10) in this embodiment 2 is within 1 cm. The method used has a good effect on low-beam lidar. This is because the hollow ellipse in the present invention can stably obtain feature constraints in low-beam lidar, overcoming the feature loss problem caused by the sparse point cloud in low-beam lidar, so that the calibration method of the present invention has higher accuracy.

[0156] (4) Determine the calibration results

[0157] See Figure 6 For the reprojected image in the virtual simulation environment, the intrinsic parameter matrix, distortion coefficients, and extrinsic parameters obtained in step 3 of the binocular camera 1 are input into the host computer. The intrinsic parameter matrix is ​​a matrix that maps 3D points in the camera coordinate system to 2D pixel coordinates on the image plane, and the distortion coefficients are values ​​proposed to correct nonlinear distortion caused by the lens. The 3D point cloud in the 3D coordinate system of the 16-line LiDAR system 2 is converted into 2D pixel coordinates in the left camera pixel coordinate system of the binocular camera 1, and then projected onto the left image. Ten projected images are obtained using ten sets of extrinsic parameters between the binocular camera 1 and the 16-line LiDAR system 2.

[0158] When the edge of the hollowed-out elliptical calibrator in the left image coincides with the three-dimensional point cloud of the edge of the hollowed-out elliptical calibrator in the calibration test space, the calibration result meets the requirements, that is, the calculated external parameters between the binocular camera 1 and the 16-line lidar 2 are accurate; the image with the smallest pixel deviation among the 10 projected images is taken as the calibration result.

[0159] The accuracy of the results obtained in Example 2 shows that the average absolute error of the rotation term is within 0.5° and the average absolute error of the translation term is within 1 cm. The above specific data and performance demonstrate that the binocular camera and lidar calibration method of the present invention has achieved significant advantages in improving calibration accuracy, robustness, and adaptability to practical applications.

[0160] Comparative Example

[0161] See Figure 7 The example is a checkerboard calibrator, which is a comparative experiment between the calibration method described in the virtual simulation environment and the MATLAB 2022b radar camera calibration program and the Autoware calibration program. The virtual simulation environment has known external parameter information available for comparison. The hardware system used in this comparative example is the same as that in Example 2.

[0162] (1) MATLAB 2022b Radar Camera Calibration

[0163] Using an 8×4 checkerboard grid with individual squares of 25 cm × 25 cm as the calibrator, the pose information of the checkerboard grid was acquired using a binocular camera and a 16-line LiDAR. For each pose change, a total of 10 sets of images from the binocular camera and 3D point cloud information from the 16-line LiDAR were acquired and input into the radar camera calibration program in MATLAB 2022b on the host computer. The average absolute error of the external parameter rotation matrix R converted into Euler angles by the program was 1.048°, and the average absolute error of the translation matrix t was 0.013 meters, which did not reach the sub-centimeter level accuracy.

[0164] (2) Autoware camera radar calibration

[0165] The program also uses an 8×4 checkerboard with each grid cell measuring 25 cm × 25 cm as the calibrator. It uses a binocular camera and a 16-line LiDAR to collect the checkerboard pose information. For each pose change, it collects a total of 10 sets of binocular camera images and 16-line LiDAR 3D point cloud information, which are then input into the Autoware calibration program on the host computer. The average absolute error of the external parameter rotation matrix R converted into Euler angles by the program is 0.988°, and the average absolute error of the translation matrix t is 0.030 meters, which does not reach the centimeter-level accuracy.

[0166] Comparing the external parameters obtained from the MATLAB 2022b calibration program with those obtained in Example 2, Example 2 shows a significant improvement in the accuracy of the rotation term (0.549°) and the accuracy of the translation term (0.006 meters). Similarly, comparing the external parameters obtained from the Autoware calibration program with those obtained in Example 2, Example 2 shows a significant improvement in the accuracy of the rotation term (0.489°) and the accuracy of the translation term (0.023 meters), with a particularly noticeable improvement in the translation term. It will be readily understood by those skilled in the art that Examples 1-3 are merely preferred embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for calibrating a hollowed-out elliptical calibrator for binocular camera and lidar calibration, characterized in that: The hollowed-out elliptical calibrator includes a square aluminum plate (4); Hamming code grid marks are provided on the four corners of one side of the square aluminum plate (4), and the four Hamming code grid marks distributed from top to bottom in a clockwise direction are the first mark (41), the second mark (42), the third mark (43) and the fourth mark (44); an elliptical through hole (45) is opened in the middle of the square aluminum plate (4), and the center of the elliptical through hole (45) coincides with the center of the square aluminum plate (4); the major axis of the elliptical through hole (45) is located in the vertical direction, and the minor axis of the elliptical through hole (45) is located in the horizontal direction; The hardware platform for the calibration method includes a binocular camera (1), a 16-line lidar (2), a hollow ellipse calibrator, a host computer, and a double-layer support (3). The host computer is the host computer. The binocular camera has a resolution of 2208×1242. The binocular camera (1) is located on the upper layer of the double-layer support (3), and the 16-line lidar (2) is located on the lower layer of the double-layer support (3). One side of the hollow ellipse calibrator with the Hamming code grid pattern faces the binocular camera (1) and the 16-line lidar (2). The distance between the binocular camera (1) and the hollow ellipse calibrator is 2.5 meters, forming a calibration test space. The calibration operation steps are as follows: (1) Image feature recognition of hollowed-out elliptical calibrator (1.1) Obtain the coordinates of the four center pixels The left and right cameras of the binocular camera (1) simultaneously capture images of the hollow elliptical calibrator and transmit the captured left and right images to the host computer. The host computer uses the left camera as the pixel coordinate center to identify the center of the four Hamming code squares on the hollow elliptical calibrator in the left image. The host computer calculates the four center pixel coordinates corresponding to the first mark (41), the second mark (42), the third mark (43), and the fourth mark (44). (1.2) Obtain the coordinates of the three-dimensional points of the four cameras The host computer constructs a depth image by comparing the differences between the left and right images, based on the principle of imitating the human eye to judge the distance of objects. The depth image is an image that stores distance information. In the depth image, the left camera of the binocular camera (1) is used as the three-dimensional coordinate center to establish the three-dimensional coordinate system of the left camera of the binocular camera (1). In the depth image, the coordinates of the four center pixels are converted into the coordinates of four camera three-dimensional points corresponding to the first mark (41), the second mark (42), the third mark (43), and the fourth mark (44) in the three-dimensional coordinate system of the left camera. (2) Obtain the coordinates of four laser 3D points for point cloud features While photographing the hollow elliptical calibrator in step (1.1), the hollow elliptical calibrator is observed by a 16-line lidar (2) to obtain the three-dimensional point cloud of the calibration test space corresponding to the image on the left at the same time. The host computer establishes a three-dimensional coordinate system for the 16-line lidar (2) with the 16-line lidar (2) as the three-dimensional coordinate center; The following operations are performed in the three-dimensional coordinate system of the 16-line lidar (2): a cuboid region containing the three-dimensional point cloud of the hollow elliptical calibrator is set in the host computer, the three-dimensional point cloud in the cuboid region is filtered to remove the messy three-dimensional point cloud, and the three-dimensional point cloud of the hollow elliptical calibrator is obtained. The three-dimensional point cloud of the hollowed-out elliptical calibrator is subjected to plane fitting, and the plane-fitted three-dimensional point cloud that meets the plane fitting result is selected. Edge extraction is performed on the three-dimensional point cloud fitted by the plane to obtain the three-dimensional point cloud of the square edge of the hollow elliptical calibrator and the three-dimensional point cloud of the elliptical edge of the elliptical through hole (45); A straight line is fitted to the left edge of the three-dimensional point cloud of the square edge to obtain the left spatial straight line direction vector of the left edge straight line; an ellipse is fitted to the three-dimensional point cloud of the elliptical edge of the elliptical through hole (45) to obtain the three-dimensional coordinates of the center of the elliptical through hole (45) and the spatial straight line direction vector of the minor axis of the ellipse. When the spatial straight line direction vector of the minor axis of the ellipse and the spatial straight line direction vector of the left space are orthogonal, according to the size of the hollow ellipse calibrator, the three-dimensional coordinates of the center of the ellipse through hole (45), the spatial straight line direction vector of the minor axis of the ellipse and the spatial straight line direction vector of the left space are calculated by the host computer to obtain the coordinates of the four laser three-dimensional points corresponding to the first mark (41), the second mark (42), the third mark (43) and the fourth mark (44) in the three-dimensional coordinate system of the 16-line laser radar (2); (3) Calculate the external parameters between the binocular camera and the 16-line lidar. (3.1) Based on the coordinates of the four camera three-dimensional points in the three-dimensional coordinate system of the left camera of the binocular camera (1) obtained in step (1) and the coordinates of the four laser three-dimensional points in the three-dimensional coordinate system of the 16-line lidar (2) obtained in step (2), the coordinates of the four camera three-dimensional points are defined as the feature point set of the left camera of the binocular camera (1). The four laser three-dimensional point coordinates are defined as a feature point set of the 16-line laser radar (2) ; (3.2) Using the feature point set of the left camera of the binocular camera (1) Calculate the centroid coordinates of the left camera of the stereo camera (1). Utilizing the feature point set of a 16-line lidar (2) Calculate the centroid coordinates of the 16-line lidar (2) ; (3.3) Using the binocular camera (1) the centroid coordinates of the left camera Calculate the decentralized coordinates of the left camera of the stereo camera (1). Using the centroid coordinates of a 16-line lidar (2) Calculate the decentralized coordinates of the 16-line lidar (2) ; (3.4) Calculate the external parameters between the binocular camera and the 16-line lidar based on singular value decomposition (SVD); the external parameters are the rotation matrix R and translation matrix t of the three-dimensional coordinate system of the 16-line lidar (2) relative to the three-dimensional coordinate system of the left camera of the binocular camera (1); (4) Determine the calibration results Based on the left-side image captured by the binocular camera (1) in step (1), the three-dimensional point cloud of the calibration test space observed by the 16-line lidar (2) in step (2), and the external parameters obtained in step (3), the three-dimensional point cloud of the calibration test space is reprojected onto the left-side image. When the edge of the hollow elliptical calibrator in the left-side image coincides with the three-dimensional point cloud of the edge of the hollow elliptical calibrator in the calibration test space, the calibration result meets the requirements, that is, the calculated external parameters between the binocular camera (1) and the 16-line lidar (2) are accurate. When the edge of the hollow elliptical calibrator in the left-side image deviates from the three-dimensional point cloud of the edge of the hollow elliptical calibrator in the calibration test space by more than 10 pixels, the calibration result does not meet the requirements.

2. The method of claim 1, wherein: The square aluminum plate (4) has a size of 1 meter × 1 meter; the Hamming code grid mark has a size of 0.1 meter × 0.1 meter; the elliptical through hole (45) has a major axis of 0.8 meters and a minor axis of 0.4 meters.

3. The method of claim 1, wherein: In step (1.1), when the left and right cameras of the binocular camera (1) simultaneously capture images of the hollow elliptical calibrator, the left camera captures at least 10 frames of left-side images and the right camera captures at least 10 frames of right-side images; the 16-line lidar (2) observes the hollow elliptical calibrator and obtains the calibration test space three-dimensional point cloud corresponding to the left-side image at the same time, thus obtaining at least 10 frames of calibration test space three-dimensional point cloud.

4. The calibration method for the binocular camera and lidar of the hollowed-out ellipse calibrator according to claim 3, characterized in that: When the left and right cameras of the binocular camera (1) simultaneously capture images of the hollow ellipse calibrator to obtain 10 left-side images and 10 right-side images, repeat steps (1.2) and (2) 10 times to obtain 10 sets of coordinates of 4 camera 3D points and 4 laser 3D points.

5. The method of claim 1, wherein: In step (1.2), the principle of mimicking the human eye's judgment of object distance refers to using Hamming code grid markers as target features and constructing a depth constraint equation based on the parallax principle: (1) In equation (1), f is the focal length, which refers to the distance from the optical center of the camera lens to the imaging plane; B is the baseline distance, which refers to the horizontal distance between the optical centers of the left and right lenses in a binocular camera; and d is the normalized parallax, which refers to the difference in pixel coordinates of the same object in the left and right images.

6. The method of claim 1, wherein: In step (2), the specific operation steps are as follows: Delete other 3D point clouds in the environment, and retain the 3D point cloud information of the hollowed-out ellipse calibrator. The specific filtering operation is as follows: Use the host computer to view the acquired 3D point cloud information of all hollow elliptical calibrators. Set a 1.5m × 1.5m × 1m cuboid area with the center of the hollow elliptical calibrator as the center, and the cuboid area contains the complete hollow elliptical calibrator. Delete the 3D point cloud of the calibration test space outside the cuboid area, and then perform statistical filtering to filter out the noisy point cloud that is 10 cm away from the 3D point cloud of the hollow elliptical calibrator. The steps to obtain the 3D point cloud features of the hollowed-out ellipse calibrator are as follows: The three-dimensional point cloud of the hollowed-out elliptical calibrator is subjected to random sampling consensus algorithm plane fitting to obtain a fitted plane three-dimensional point cloud. The fitted plane three-dimensional point cloud is then filtered, and the filtered fitted plane three-dimensional point cloud is extracted by rolling ball method to obtain square edge three-dimensional point cloud and elliptical edge three-dimensional point cloud; it is ensured that the number of points in the obtained elliptical edge three-dimensional point cloud is greater than 20. The left edge of the 3D point cloud of the square edge is fitted with a straight line using a random sampling consensus algorithm to obtain the spatial direction vector of the left edge straight line. The random sampling consensus algorithm is used to fit the ellipse edge 3D point cloud to obtain the 3D coordinates of the ellipse center and the spatial direction vector of the ellipse minor axis. When the obtained elliptical minor axis spatial line direction vector and the left edge spatial line direction vector are orthogonal, the coordinates of the four laser three-dimensional points corresponding to the first mark (41), the second mark (42), the third mark (43), and the fourth mark (44) in the three-dimensional coordinate system of the 16-line lidar (2) are calculated based on the three-dimensional coordinates of the elliptical center, the elliptical minor axis spatial line direction vector, and the left edge spatial line direction vector.

7. The binocular camera and lidar calibration method of hollowed-out ellipse calibrator according to claim 6, characterized in that: In step (2), when the number of points in the obtained 3D point cloud of the ellipse edge is less than 20, the 3D points of the ellipse edge are manually selected in the host computer to ensure that the number of points in the obtained 3D point cloud of the ellipse edge is greater than 20.

8. The method of claim 1, wherein: In step (3), the specific operation for calculating the external parameters between the binocular camera (1) and the 16-line lidar (2) is as follows: (3.1). Solving for the rotation and translation matrices using feature point sets. The three-dimensional point coordinates of the four cameras are defined as the feature point set of the left camera of the stereo camera (1), i.e. The coordinates of the four laser 3D points are defined as the feature point set of the 16-line lidar (2), i.e. ; The rigid body transformation rotation matrix is solved using equation (2) and translation matrix , equation (2) being as follows: （2） In equation (2), ϵ i This is a Gaussian noise term. This represents the coordinates of the i-th corresponding 3D point in the feature point set of the left camera of the binocular camera (1). This represents the coordinates of the i-th corresponding laser 3D point in the feature point set of the 16-line lidar (2). This represents the rotation matrix from the three-dimensional coordinate system of the left camera (1) of the binocular camera to the three-dimensional coordinate system of the 16-line lidar (2). This represents the translation matrix from the three-dimensional coordinate system of the left camera of the binocular camera (1) to the three-dimensional coordinate system of the 16-line lidar (2); (3.2). Calculate the coordinates of the centroid of the point set. The centroid coordinates of the feature point set of the left camera of the binocular camera (1) are calculated The centroid coordinates of the feature point set of the binocular camera are calculated, and the formula (3) is as follows: （3） In formula (3), is the centroid coordinate of the feature point set of the left camera of the binocular camera (1) in the three-dimensional coordinate system. Utilizing the feature point set of a 16-line lidar (2) The centroid coordinates of the feature point set of the 16-line lidar are calculated using formula (4) as follows: （4） In formula (4), is the centroid coordinate of the feature point set of the 16-line laser radar (2) three-dimensional coordinate system. (3.3). Obtain decentralized coordinates Using centroid coordinates and centroid coordinates Eliminate translation vector For rotation matrix The estimated impact, the decentralized coordinate formula (5) for the left camera and the decentralized coordinate formula (6) for the 16-line lidar are as follows: （5） （6） In formula (5), The i-th decentralized coordinate of the feature point set of the left camera in the three-dimensional coordinate system of the binocular camera (1); in formula (6), The i-th decentralized coordinate of the feature point set in the three-dimensional coordinate system of the 16-line lidar (2); (3.4). Singular value decomposition computes rotation matrix and translation vector ; First, construct the covariance matrix. The formula (7) for calculating the covariance matrix of the decentralized point set is as follows: （7） The covariance matrix of equation (7) is decomposed into the product of three matrices Equation (8), the singular value decomposition equation, is obtained as follows:​ （8） In equation (8), It is an orthogonal matrix. It is a diagonal matrix (σ1≥σ2≥σ3≥0); The transpose of the two orthogonal matrices obtained by decomposition using formula (8) Solve for the rotation matrix Rotation matrix Formula (9) is as follows: （9） Then use the centroid coordinates obtained from formula (5) The centroid coordinates obtained from formula (6) Solve for the translation vector Translation vector Formula (10) is as follows: （10） Repeat steps (1) and (2) to obtain 10 sets of 3D point coordinates of 4 cameras and 4 laser 3D point coordinates. Through the operation of step (3), calculate the external parameters between 10 sets of binocular cameras (1) and 16-line laser radar (2).

9. The calibration method for the binocular camera and lidar of the hollowed-out ellipse calibrator according to claim 1, characterized in that: In step (4), the specific operations are as follows: Input the intrinsic parameter matrix, distortion coefficients and external parameters obtained in step (3) of the binocular camera (1) into the host computer, and convert the three-dimensional point cloud in the three-dimensional coordinate system of the 16-line lidar system (2) into the 2D pixel coordinates of the left camera pixel coordinate system of the binocular camera (1), and then project it onto the left image; use the external parameters between 10 sets of binocular camera (1) and 16-line lidar (2) to obtain 10 projected images; When the edge of the hollow elliptical calibrator in the left image coincides with the three-dimensional point cloud of the edge of the hollow elliptical calibrator in the calibration test space, the calibration result meets the requirements, that is, the calculated external parameters between the binocular camera (1) and the 16-line lidar (2) are accurate; the image with the smallest pixel deviation among the 10 projected images is taken as the calibration result. When the deviation of the edge of the hollowed-out elliptical calibrator in the left image from the edge of the hollowed-out elliptical calibrator in the calibration test space in the three-dimensional point cloud exceeds 10 pixels, the calibration result does not meet the requirements.

Images