A Calibration Method for a Sparse LiDAR and a Visible Light / Infrared Imaging System

Abstract

The present invention relates to a calibration method for a sparse lidar and a visible light / infrared imaging system, comprising the following steps: obtaining the camera internal parameters; based on a calibration board, finding the coordinates of 9 center points corresponding to each group of data in the image and in the point cloud respectively; using a loss function to optimize the pixel coordinate system coordinates and the radar coordinate system coordinates of the feature points; obtaining the coordinates in the camera coordinate system according to the pixel coordinates; establishing 3D constraints in the camera coordinate system and the radar coordinate system, and obtaining the initial values of and ; using 3D-2D point iteration or 3D-3D point iteration for the initial values to obtain the optimized solutions of and. The present invention designs a calibration board and establishes geometric constraints between the dug-out feature points. The average reprojection error of the method of the present invention is within 3 pixels, achieving good results, and can also be used for the calibration of a sparse lidar and a visible light-infrared multi-band imaging system.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a calibration method for a sparse lidar and a visible light / infrared imaging system. Background Art

[0002] In recent years, autonomous driving technology has developed rapidly. This technology deploys an inertial navigation system, millimeter-wave radar, ultrasonic radar, lidar, imaging system, global positioning system, etc. on a vehicle, and cooperates with an in-vehicle high-performance computing platform for data fusion to automatically complete scene understanding, positioning, and obstacle avoidance during vehicle driving to ensure the safe driving of the vehicle.

[0003] According to the laser ranging principle, the lidar outputs lidar point cloud data, and the distance data of the target can be obtained. Commonly used imaging systems are divided into a visible light imaging system and an infrared imaging system according to different working bands and imaging principles.

[0004] Since the infrared imaging system can work all day long, it has gradually been applied to autonomous driving and is a useful supplement to visible light images. The fusion of lidar with visible light and infrared imaging systems can obtain three-dimensional information of the surrounding environment. Before these information are fused, it is necessary to first perform internal calibration on the imaging system. On this basis, the external calibration between the lidar and the imaging system is completed. The internal calibration obtains the internal parameter matrix of the imaging system, including the pixel principal point, pixel focal length, distortion parameters, etc. The internal parameter matrix reflects the relationship between image pixels and corresponding spatial points; the external calibration obtains the rigid body transformation matrix between the imaging system coordinate system and the lidar coordinate system, including the rotation matrix and translation vector, etc. The rigid body transformation matrix reflects the pose relationship between the imaging system and the lidar.

[0005] There are mainly two types of calibration methods between the lidar and the imaging system: offline calibration and online calibration.

[0006] Offline calibration is also called target calibration. In a non-task scenario, a specific target (usually a calibration board) is used to find the corresponding feature points of the specific target in the two systems respectively. In the early stage, due to the small number of lidar lines, such as 4 lines, 8 lines, and 16 lines, the design of the calibration board was usually relatively simple. Rodriguez et al. designed a calibration board with dug-out concentric circles, and the feature points were the centers of the concentric circles; Chen Dong et al. designed a calibration board combining dug-out circles and checkerboards, and the feature points were the corner points of the checkerboards.

[0007] In recent years, with the development of lidar technology, the number of lines of lidar has been increasing. The mechanical scanning lidar has reached 128 lines, and the solid-state lidar can also reach a level equivalent to more than 100 lines. Therefore, the calibration task has begun to shift to using a non-perforated checkerboard calibration board to complete. The main advantage of using a checkerboard calibration board is that it is convenient to automatically find the corner points of the checkerboard plane and establish constraints to solve the plane equation through algorithms in the dense lidar point cloud; the checkerboard can also be used for calibrating the internal parameter matrix of the imaging system. Zhou, Geiger, etc. obtained the rigid body transformation matrix based on the plane constraints and line constraints designed from the checkerboard found in the lidar point cloud.

[0008] Online calibration, also known as targetless calibration, does not require the use of a specific target and can be completed during the motion process of performing tasks. Online calibration is usually based on hand-eye calibration, that is, Ishikawa, etc. solved the equation by separately calculating the motion estimation of the lidar and the imaging system during the motion process.

[0009] In recent years, using convolutional neural networks (CNNs) for online calibration has become a research hotspot. Just by inputting the lidar point cloud and the corresponding image, the network can solve and obtain the rigid body transformation matrix. Schneider, etc. proposed RegNet, which uses a large amount of calibrated lidar point cloud and image data for training and directly calculates and obtains the rigid body transformation matrix when performing tasks; Ganesh, etc. proposed the self-supervised network CalibNet, which obtains the rigid body transformation matrix by maximizing the geometric and photometric consistency between the point cloud and the image.

[0010] Currently, the mainstream offline calibration method for checkerboards has good effects for lidars with 64 lines and above. However, when used for 16-line lidars, due to the sparse data, the error is relatively large, and since the checkerboard calibration board is a whole and there is no temperature difference, it cannot be used for infrared imaging systems. Summary of the Invention

[0011] To solve the above problems, the present invention proposes a calibration method for a sparse lidar and a visible light / infrared imaging system, and also relates to a calibration board, which can meet the calibration requirements of a 16-line sparse lidar, an infrared imaging system, and a visible light imaging system. The principle of the 16-line lidar is also laser ranging, but the point cloud information is less. Therefore, a perforated calibration board is designed, and geometric constraints are established between the perforated feature points; the imaging principle of the visible light imaging system is the reflection of light. To facilitate the identification of holes and the calibration board in the visible light image, the whole calibration board is designed to be black; the imaging principle of the infrared imaging system is thermal radiation, and there is a temperature difference between the holes and the calibration board of the perforated calibration board, which can be distinguished in the infrared image.

[0012] In view of the problem of sparse lidar point cloud data, this invention studies a method for simultaneously calibrating lidar with visible light and infrared imaging systems, designs a diamond-shaped nine-hole calibration plate, and proposes a geometric constraint loss function to optimize the coordinates of feature points. Finally, the infrared and visible light imaging systems are respectively calibrated with a 16-line lidar. The experimental results show that the average reprojection error is within 3 pixels, achieving good results. The method of this invention can also be used for the calibration of sparse lidar and visible light-infrared multi-band imaging systems.

[0013] The technical solution of this invention is as follows:

[0014] A calibration plate for sparse lidar and visible light / infrared imaging systems, including a plate body, on which there are 9 circular holes with the same radius. The circular holes correspond one by one to the coordinates of nine center points in the lidar coordinate system and the image pixel coordinate system. There are geometric constraint relationships among the nine center points, and the geometric constraint relationships include parallel constraints, perpendicular constraints, and midpoint constraints.

[0015] Furthermore, for the feature points in the image, the circles in the image are detected by Hough circle transformation, and the pixel coordinates (x, y) of nine center points are found in each image.

[0016] Furthermore, in the lidar point cloud, at least two laser scan lines pass through each circular hole, generating a distance mutation. The lidar coordinates (X l , Y l , Z l ) of nine center points are obtained by solving the center of the circumscribed circle.

[0017] Furthermore, the calibration plate is used to optimize the coordinates of feature points in two coordinate systems and solve the external parameters (rotation matrix and translation vector) of sparse lidar and visible light / infrared imaging systems.

[0018] This invention relates to a calibration method for sparse lidar and visible light / infrared imaging systems, including the following steps:

[0019] Step (1): Obtain the camera internal parameters;

[0020] Step (2): Based on the above calibration plate, find the coordinates of 9 center points corresponding to each group of data in the image and in the point cloud respectively;

[0021] Step (3): Use the loss function to optimize the coordinates of feature points in the pixel coordinate system and the lidar coordinate system;

[0022] Step (4): Obtain the coordinates in the camera coordinate system according to the pixel coordinates;

[0023] Step (5) Establish the 3D constraints in the camera and radar coordinate systems to obtain and initial values;

[0024] Step (6) Use 3D-2D point iteration or 3D-3D point iteration for the initial values to obtain and optimal solutions.

[0025] Furthermore, in step (2), the circles in the image are detected by Hough circle transform, and the pixel coordinates (x, y) of nine circle centers are found in each image. In the lidar point cloud, at least two laser scan lines pass through each circular hole, generating a distance mutation to construct a system of linear equations composed of unknowns. Solving this system of equations can obtain the circle center coordinates and radius, and then the coordinates of nine center feature points in 2D and 3D can be obtained.

[0026] Furthermore, in step (3), random search is used to continuously optimize the loss function to be close to 0 and reduce the error of the nine points.

[0027] Furthermore, in step (4), first convert the 2D feature points to 3D coordinates in the imaging system coordinate system, and then establish constraints with the 3D feature points in the lidar coordinate system. First, establish constraints to solve Use the direction vector of the line and the normal vector of the plane to establish constraints;

[0028] For the nine corresponding feature points in the two coordinate systems, find the direction vectors of 6 lines, ensuring that all 9 points are used twice and the weights are the same. 8 constraints can be established for 9 groups of corresponding points in one position. The initial value of can be solved by the method of singular value decomposition;

[0029] Use the obtained initial value as the initial value for calculating Use plane constraints, line constraints, and perpendicular constraints. For 9 groups of corresponding points in one position, 2 plane normal vectors and 6 line direction vectors are obtained. Use the centroid of the 9 points and the two plane normal vectors to obtain 2 equations; use the midpoints and direction vectors of 6 line segments to obtain 18 equations; use the projection points of four points in the radar coordinate system in the imaging system coordinate system to establish four perpendicular constraints to obtain 4 equations. A total of 24 equations are obtained, and the linear least squares solution of is used as the initial value.

[0030] Furthermore, after step (4), directly use the corresponding feature points in 2D and 3D to solve the PnP problem; use the initial value as the initial solution and use the LM iteration method to iteratively optimize the cost function to obtain and Iterative solutions

[0031] It can be seen that, in view of the calibration problem between the sparse lidar and the visible light and infrared imaging systems, the present invention designs a rhombic nine-hole calibration plate, and designs a geometric constraint loss according to the constraints of the calibration plate to optimize the measured coordinates of the feature points and reduce the error; the initial values of and are obtained according to the 3D geometric constraints, and the iterative solutions of and are obtained using the iterative method.

[0032] The present invention respectively uses the visible light imaging system, the infrared imaging system and the 16-line lidar for calibration experiments. The average reprojection error in the visible light system is within 3 pixels, and the average reprojection error in the infrared system is within 3 pixels.

[0033] The present invention is also applicable to the visible light-infrared multi-band imaging system. Only a set of lidar point cloud data needs to be collected, and it is used as a reference to calibrate multiple cameras of the multi-band imaging system respectively, which facilitates the fusion process of the multi-band imaging information and the lidar point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is the calibration plate of the embodiment of the present invention;

[0035] Figure 2 is the calculation of the center coordinates of the lidar data in the embodiment of the present invention;

[0036] FIG. 3(a) is the visible light image data of the embodiment of the present invention; (b) is the lidar point cloud data of the visible light system of the embodiment of the present invention; (c) is the experimental scene of the infrared system; (d) is the infrared image data;

[0037] FIG. 4 is the scatter plot of the feature point reprojection errors of two iterative methods in the embodiment of the present invention; (a) is 3D-3D iteration; (b) is 3D-2D iteration;

[0038] FIG. 5(a) is the projection result of the lidar points in the visible light system in the embodiment of the present invention; (b) is the projection result of the lidar points in the infrared system in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Next, the technical solutions in the embodiments will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present application.

[0040] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of this application shall have the ordinary meanings understood by those of ordinary skill in the art. The "first", "second" and similar terms used in this embodiment do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. "Installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two components. "Up", "down", "left", "right", "horizontal" and "vertical" are only relative to the orientation of the components in the drawings. These directional terms are relative concepts, which are used for relative description and clarification, and they can change accordingly with the change of the orientation of the components placed in the drawings.

[0041] The principle involved in the method of this embodiment is as follows:

[0042] The calibration of the lidar and the imaging system is essentially a conversion between coordinate systems. In this embodiment, (X l , Y l , Z l ) is used to represent the 3D point coordinates of the lidar-centered radar coordinate system; (X c , Y c , Z c ) is used to represent the 3D point coordinates in the imaging system coordinate system; (X w , Y w , Z w ) is used to represent the 3D point coordinates in the world coordinate system; (x, y) is used to represent the 2D point coordinates in the pixel coordinate system of the image.

[0043] The projection process of the camera can be represented by Equation (1), and the coordinate transformation between the lidar and the imaging system can be represented by Equation (2).

[0044] Among them, K is the internal parameter matrix of the imaging system, which can be obtained using the Zhang Zhengyou calibration method. and are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system. and are the rotation matrix and translation vector from the lidar coordinate system to the imaging system coordinate system. Combined, they are the rigid body transformation matrix.

[0045]

[0046]

[0047] By finding the coordinates (X l , Y l , Z l ) of the corresponding feature points in the lidar coordinate system and the coordinates (x, y) in the pixel coordinate system, and

[0048] The first method does not need to solve the coordinates in the imaging system coordinate system. Instead, it directly uses the 2D and 3D corresponding feature points to solve the PnP (Perspective N Points) problem.

[0049] Combining Equation (1) and Equation (2) and using homogeneous coordinates to combine the internal and external parameters together, Equation (3) is obtained. This is a system of equations with 12 unknowns, and each pair of feature points can provide two equations. Therefore, at least six pairs of feature points are required to solve the 12 unknowns, and then the initial values of the rotation matrix and translation vector can be solved.

[0050] Then, taking these two as the initial solutions, using the LM (Levenberg-Marquarelt) iterative method to iteratively optimize the cost function of Equation (4), the iterative solutions of and are obtained, where P li and p i are a pair of corresponding points of the point cloud and pixel coordinates, and N ≥ 6.

[0051]

[0052]

[0053] The second method requires finding the coordinates of the feature points in the imaging system coordinate system and using the 3D and 3D corresponding points for solution. According to Zhang Zhengyou's calibration method, when the physical size of the calibration board is known and assuming the world coordinates of the plane where the calibration board is located are Z w = 0, the projection process in Equation (1) can be converted into the process of solving the homography matrix, and then the internal parameter matrix K of the imaging system can be solved. Similarly, K can also be used to decompose the homography matrix to obtain and Then, according to Equation (1), the coordinates (X c , Y c , Z c ) of the feature points in the imaging system coordinate system are calculated. For multiple pairs of 3D feature points, by establishing geometric space constraints, the system of equations can be solved to obtain and The initial value is then iteratively solved using the Iterative Closest Point (ICP) algorithm. The spatial constraints and iterative methods vary depending on the calibration board and experimental equipment.

[0054] For sparse lidar, using the rectangular boundary points of a common checkerboard calibration board as feature points does not yield good results. Therefore, the internal information of a cutout calibration board needs to be used as feature points. Based on the above principle, the calibration board for sparse lidar and visible light / infrared imaging systems in this embodiment.

[0055] As Figure 1 shown, the calibration board of this embodiment consists of nine circular holes with equal radii, and the feature point pairs are the coordinates of the nine centers in the lidar coordinate system and the image pixel coordinate system. There are strict geometric constraint relationships among the nine centers, such as parallel constraints, perpendicular constraints, midpoint constraints, etc.; these constraints are used to complete the optimized calculation of the feature point coordinates and and the calculation of the initial value.

[0056] For the feature points in the image, the circles in the image are detected through Hough circle transformation, and the pixel coordinates (x, y) of the nine centers can be found in each image.

[0057] In the lidar point cloud, each laser scan line passes through the circular hole, resulting in a sudden change in distance, as Figure 2 shown. Suppose there are two lidar scan lines passing through the circle, and each of these two scans will generate a section of sudden distance change. The two endpoints of the sudden change are on the circle, namely the four points P1, P2, P3, and P4 in the figure; also, since three points on the circle can determine a circle, the problem can be converted into finding the circumcircle of a triangle, and the center coordinates can be solved using the three points P1, P2, and P3. Suppose the coordinates of the three points P1, P2, and P3 in the lidar coordinate system are (X l1 , Y l1 , Z l1 ), X l2 , Y l2 , Z l2 ), (X l3 , Y l3 , Z l3 ). The plane equation determined by the three points P1, P2, and P3 is A0x + B0y + C0z + D0 = 0, and the coordinates of the center P0 are (X l , Y l , Z l ), and the radius is R (R > 0).

[0058] It can be obtained from X l , Y l , Z l, a linear equation system composed of four unknowns R, as shown in Equation (5). Solving this equation system can obtain the center coordinates and the radius. To make the result more accurate, in this paper, any three points are selected from the four points P1, P2, P3, and P4, calculated four times, and the average value of the results is used as the center coordinates.

[0059]

[0060] Due to errors in the measurement process, after obtaining the coordinates of the nine center feature points in 2D and 3D, it is also necessary to optimize the coordinates of the feature points in the two coordinate systems using the geometric constraints of the calibration board.

[0061] In this embodiment, a loss function (loss function) is designed to optimize the loss so that the error of the coordinates is as small as possible.

[0062] As Figure 1 shown, there are midpoint constraints in the calibration board of this embodiment, such as E being the midpoint of AB, etc.; perpendicular constraints, such as EB⊥BH, etc.

[0063] The calibration method of the sparse lidar and visible light / infrared imaging system in this embodiment includes the following steps:

[0064] Step (1): Obtain the camera internal parameters;

[0065] Step (2): Based on the above calibration board, find the coordinates of 9 center points corresponding to each group of data in the image and the point cloud respectively;

[0066] Step (3): Use the loss function to optimize the coordinates of the points;

[0067] Step (4): Obtain the coordinates in the camera coordinate system according to the pixel coordinates;

[0068] Step (5): Establish 3D constraints in the camera and lidar coordinate systems to obtain the and initial values;

[0069] Step (6): Use 3D-2D point iteration or 3D-3D point iteration to obtain the and optimized solutions.

[0070] Specifically, it is carried out as follows:

[0071] This embodiment uses loss mid to represent the midpoint constraint error, as shown in Equation (6), that is, the distance between point E and the midpoint of AB; this paper uses loss vertical to represent the perpendicular constraint error, as shown in Equation (7), that is, the absolute value of the dot product between two vectors; the overall loss is as shown in Equation (8), and each point is used 4 times, with equal weights to avoid introducing new errors.

[0072] In this embodiment, the method of random search is used to continuously optimize the loss to be close to 0 and reduce the error of nine points.

[0073] loss mid (E, A, B) = ||E - mid(A, B)|| (6)

[0074]

[0075]

[0076] After optimizing the 2D feature point coordinates and 3D feature point coordinates respectively, the second method introduced in the previous section of this paper is used to find and the initial values. First, the 2D feature points are converted into 3D coordinates in the imaging system coordinate system, and then constraints are established with the 3D feature points in the lidar coordinate system.

[0077] Compared with the first method of solving the PnP problem to directly obtain the external parameter matrix, the second method solves the rotation matrix and the translation vector separately, which can better reduce the error between the two, so the effect is better.

[0078] First, establish constraints to solve Compared with point coordinates, the spatial vector is only subject to constraints. Therefore, the direction vector of the line and the normal vector of the plane are used to establish constraints, as shown in Eqs. (9) and (10). Among them and are the direction vectors of a certain line in two coordinate systems; and are the normal vectors of the calibration plate plane in two coordinate systems.

[0079]

[0080]

[0081] For the nine corresponding feature points in two coordinate systems, find the direction vectors of 6 lines: AB, BC, CD, DA, EF, HG to establish constraints, and use the three points E, I, H and the three points F, I, G to solve the plane normal vector twice to establish constraints; this can ensure that all 9 points are used twice and the weights are the same. 8 constraints can be established for 9 groups of corresponding points in one position, and the initial value of can be solved by the method of Singular Value Decomposition (SVD). Using multiple groups of corresponding points in multiple positions and establishing more constraints can obtain a more accurate initial value result.

[0082] Use the initial value obtained above as the initial value for calculation , and the constraints used include plane constraint, line constraint, and perpendicular constraint, as shown in Eqs. (11), (12), and (13). Among them and d c are the normal vector of the plane and the constant term of the plane equation in the imaging system coordinate system, and P l is a point in the lidar coordinate system, and P c is the corresponding point in the imaging system coordinate system, I is the 3×3 identity matrix, is the direction vector of the line where P c is located, is the direction vector of the line perpendicular to the line where P c is located.

[0083]

[0084]

[0085]

[0086] For 9 groups of corresponding points at a position, 2 plane normal vectors and 6 line direction vectors are obtained. Substitute the centroid of the 9 points and the two plane normal vectors into Eq. (11) to obtain 2 equations; substitute the midpoints and direction vectors of the 6 line segments into Eq. (12) to obtain 18 equations; use the projection points of the four points A, B, C, and D in the lidar coordinate system in the imaging system coordinate system, substitute them into Eq. (13) respectively, and establish four perpendicular constraints of AB⊥BC, BC⊥CD, CD⊥DA, and DA⊥AB to obtain 4 equations, and a total of 24 equations are obtained. Since the translation vector has three unknowns, the 24 equations form an overdetermined linear system of equations, and the linear least squares solution can be obtained as the initial value.

[0087] After obtaining the initial values of and , the 3D-2D point iteration or 3D-3D point iteration method can be used to solve the and optimal solutions.

[0088] Simulation experiment

[0089] In actual situations, and the true values are unknown, and usually the reprojection error of feature points is used to evaluate and Accuracy. However, this metric assumes that there is no error in the feature points. To better verify the effectiveness of the method proposed in this paper, a simulation experiment is conducted in this embodiment.

[0090] First, a calibration board is generated with random dimensions, and the lidar coordinates of the feature points are randomly generated, and Using a fixed intrinsic matrix K, the coordinates in the imaging system coordinate system and the pixel coordinate system are obtained, and the 3D constraint and the PnP method are used to solve and the initial values, which are compared with the actual values. The results of multiple simulations are shown in Table 1. It is represented by the unit vector of the rotation axis and the rotation angle, and is represented by a vector.

[0091] In Table 1, a represents the result using the 3D constraint, b represents the initial value result using the PnP method. The rotation axis error and the translation error are both the L1 errors of the rotation axis vector and the translation vector relative to the actual ones, and the angular error is the error between the rotation angle and the actual rotation angle, with the unit of radians. It can be seen that the error of using the 3D constraint method to find the initial value is several orders of magnitude smaller than that of the PnP method. Therefore, using the 3D constraint to find the initial value can greatly reduce the number of iterations required for subsequent iterative solutions, while the advantage of the PnP method lies in its simplicity.

[0092] Table 1 Error comparison between the two methods

[0093]

[0094] By simulating the results of finding the initial value in an ideal situation, the effectiveness of the method proposed in this paper is verified. In actual measurement, there are certain errors in the obtained feature point coordinates. It is necessary to randomly add errors to the lidar coordinate system data and the imaging system coordinate system data generated in the previous section to simulate the measured values, and then use the method of this embodiment to obtain and the results, and the simulation results are shown in Table 2.

[0095] Table 2 Results of the method of this embodiment

[0096]

[0097]

[0098] As can be seen from Table 2, the method of this embodiment can greatly reduce the loss of the feature points, and the and errors solved by this method are very small compared with the actual errors. The simulation experiment verifies the effectiveness of the method proposed in this paper, and actual experiments will be carried out in the next section.

[0099] Experimental equipment

[0100] Calibration experiments were carried out using lidar with visible light imaging system and infrared imaging system respectively to verify the effectiveness of the method in this embodiment.

[0101] The visible light imaging system uses a high-definition industrial camera of LT-USB1080P model, with an imaging resolution of 1920×1080 pixels and a focal length of 6mm; the infrared imaging system uses a thermal imager of IR-Pilot640 model from Axisir Optoelectronics, with an imaging resolution of 640×512 pixels, a focal length of 6.9mm, and a working band of 8-14μm; the lidar uses a 16-line sparse lidar of Velodyne VLP-16 model, with a vertical field of view of -15°-15° and a resolution of 2°; the horizontal field of view is 360°, and the highest resolution is 0.1°.

[0102] Experimental process

[0103] According to the previous principle introduction, at least two laser scan lines need to pass through each round hole of the calibration board. Combining the parameters and working distance of the lidar and the imaging system, the size of the calibration board is set to 1.2m×1.35m, and the radius of each round hole is 9cm.

[0104] Place the calibration board in the overlapping area of the fields of view of the lidar and the imaging system, about 2m away from the system. Use the experimental system to collect multiple groups of data and conduct experiments according to the process in Table 1. Figures 3(a) and (b) show a set of corresponding data of visible light images and laser point clouds. Figures 3(c) and (d) show the experimental scene of the infrared experimental system and the infrared images obtained by the infrared imaging system respectively. In the experiment, the camera calibration toolbox of Matlab is used to obtain the internal parameter matrix K of the imaging system, and the corresponding functions in Opencv are used to complete the circle detection and distortion correction in the image. The other parts of the method are written in Python.

[0105] According to the distance of the calibration board and the horizontal resolution of the lidar, it can be calculated that the distance between adjacent laser scan points is about 0.5cm; that is, the maximum error in judging points on the circle is 0.5cm under ideal conditions. Considering that some laser points may not receive return values, as well as the errors of the lidar and the algorithm, it is considered that the maximum error of the center point in the lidar coordinate system is 5cm, and the maximum error of the center point in the pixel coordinate system is 10 pixels. Optimize the loss according to these two parameters, take multiple groups of data to establish the initial value by 3D constraints, and then use the ICP algorithm of 3D-3D and the LM iterative method of 3D-2D to obtain the optimized solution respectively.

[0106] Test results

[0107] In the experiments of the visible light and infrared systems, the same number of feature points are used in this paper, and the same number of iterations are performed to obtain the final results.

[0108] The experimental results of the visible light system and the infrared system are shown in Tables 3 and 4. Here, a represents the 3D-3D iteration using the ICP algorithm, and b represents the results of the 3D-2D iteration using the L-M iteration method. The average reprojection errors of the 3D to 2D feature points in the x and y directions (in pixels) are used to evaluate the accuracy of the method.

[0109] For the visible light system, it can be seen that in the case of using 3D-3D iteration, the average reprojection error of the method in this embodiment is about 4 pixels, and in the case of using 3D-2D iteration, the average reprojection error is less than 3 pixels, achieving very good results.

[0110] For the infrared system, in the case of using 3D-3D iteration, the average reprojection error of the method in this embodiment is within 4 pixels, and in the case of using 3D-2D iteration, the average reprojection error is within 3 pixels, with very good results.

[0111] Figure 4 shows the scatter plot of the feature point reprojection errors of the two iteration methods. It can be seen that except for some feature points, the reprojection errors are relatively small. The points with larger errors may be due to measurement errors introduced by lidar, image algorithms, etc.

[0112] Table 4 Experimental Results of the Visible Light System

[0113]

[0114] Table 5 Experimental Results of the Infrared System

[0115]

[0116] Figures 5(a) and (b) respectively show the experimental results and the fusion results of projecting the point cloud data onto the image. It can be seen that except for a little deviation at some edge points with large distance mutations, the 16 laser scan lines are very coherent, indicating that the calibration effect is very good, and the obtained fusion data can be used in subsequent other tasks.

[0117] In this embodiment, aiming at the calibration problem between the sparse lidar and the visible light and infrared imaging systems, a diamond-shaped nine-hole calibration plate is designed. According to the constraints of the calibration plate, a geometric constraint loss is designed to optimize the measured coordinates of the feature points and reduce the error; according to the 3D geometric constraints, the and initial values are obtained, and the iterative method is used to obtain the and Iterative solution

[0118] In this embodiment, a visible light imaging system, an infrared imaging system, and a 16-line lidar are respectively used for calibration experiments. The average reprojection error is within 3 pixels in the visible light system and within 3 pixels in the infrared system.

[0119] The method of this embodiment is also applicable to a visible light-infrared multi-band imaging system. Only a set of lidar point cloud data needs to be collected, and based on this, calibration is respectively performed with multiple cameras of the multi-band imaging system, facilitating the fusion process of multi-band imaging information and lidar point cloud data.

[0120] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A calibration method for a sparse lidar and a visible light or infrared imaging system, characterized in that: Including the following steps: Step (1): Obtain the camera internal parameters; Step (2): Based on the calibration board, find the coordinates of the 9 center points corresponding to each group of data in the image and the point cloud respectively; the calibration board includes a board body, and the board body is provided with 9 circular holes with the same radius. The circular holes correspond one-to-one to the coordinates of the nine center points in the lidar coordinate system and the image pixel coordinate system. There are geometric constraint relationships among the nine center points, and the geometric constraint relationships include parallel constraints, perpendicular constraints, and midpoint constraints; Step (3): Use the loss function to optimize the pixel coordinate system coordinates and radar coordinate system coordinates of the feature points; Step (4): Obtain the coordinates in the camera coordinate system according to the pixel coordinates; Step (5) establishes 3D constraints in the camera and radar coordinate systems to obtain and initial values, and are the rotation matrix and translation vector from the lidar coordinate system to the imaging system coordinate system; Step (6) uses 3D-2D point iteration or 3D-3D point iteration on the initial value to obtain and the optimized solution.

2. The calibration method of a sparse lidar and a visible light or infrared imaging system according to claim 1, wherein: In step (2), for the feature points in the image, the circles in the image are detected by Hough circle transformation, and the pixel coordinates of nine circle centers are found in each image. .

3. A calibration method for a sparse lidar and a visible light or infrared imaging system according to claim 1, characterized in that: In step (2), in the lidar point cloud, at least two laser scan lines pass through each circular hole, resulting in a distance mutation. The lidar coordinates of the nine centers are obtained by solving the center of the circumscribed circle. .

4. A calibration method for a sparse lidar and a visible light or infrared imaging system according to claim 1, characterized in that: In step (2), the calibration board is used to optimize the feature point coordinates in the two coordinate systems and solve the external parameters of the sparse lidar and the visible light / infrared imaging system.

5. A calibration method for a sparse lidar and a visible light or infrared imaging system according to claim 1, characterized in that: In step (3), use random search to continuously optimize the loss function to be close to 0 and reduce the error of the nine points.

6. A calibration method for a sparse lidar and a visible light or infrared imaging system according to claim 1, characterized in that, In step (4), first convert the 2D feature points into 3D coordinates in the imaging system coordinate system, and then establish constraints with the 3D feature points in the lidar coordinate system. First, establish constraints to solve , and use the direction vector of the line and the normal vector of the plane to establish constraints; For the nine feature points corresponding to two coordinate systems, find the direction vectors of six lines, ensuring that all nine points are used twice with consistent weights. Eight constraints can be established for the nine groups of corresponding points in one position, and the initial value can be solved by the method of singular value decomposition. of the initial value; Use the initial value obtained above as , calculate the initial value of . Using planar constraints, linear constraints, and perpendicular constraints, for 9 groups of corresponding points at a position, 2 planar normal vectors and 6 linear direction vectors are obtained. Using the centroid of 9 points and two planar normal vectors, 2 equations are obtained; using the midpoints and direction vectors of 6 line segments, 18 equations are obtained; using the projection points of four points in the radar coordinate system in the imaging system coordinate system, four perpendicular constraints are established to obtain 4 equations. A total of 24 equations are obtained, and the linear least squares solution of is used as the initial value.

7. A calibration method for a sparse lidar and a visible light or infrared imaging system according to claim 1, characterized in that: After step (4), directly use the corresponding feature points in 2D and 3D to solve the PnP problem; take the initial value as the initial solution, and use the LM iterative method to iteratively optimize the cost function to obtain and the iterative solution of.

Images