A joint calibration method for single-line lidar and camera

Abstract

The present invention discloses a joint calibration method for a single-line laser radar and a camera, and belongs to the technical field of multi-sensor calibration. The calibration method comprises the following steps: (1) adjusting the position and height of a calibration plate, placing a calibration plate with a hollow area and a non-hollow area in the field of view of a camera and a laser radar; (2) the laser radar scans the calibration plate, takes a photo with a camera, and obtains a series of point cloud information by scanning. The point cloud at the intersection of the scanning line and the boundary line of the non-hollow and hollow areas of the calibration plate is used as a feature point, and the point cloud close to the feature point is selected from a plurality of point clouds. After optimizing the point cloud by the LM algorithm, a straight line fitting is performed; (3) the coordinates in the calibration plate coordinate system are calculated by the straight line fitting equation and the boundary line equation; (4) the coordinates of the pixel points corresponding to the laser radar feature points are found in the camera image; (5) the pose matrix between the laser radar and the camera is solved by parameter fitting.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor calibration, and in particular to a joint calibration method for a single-line laser radar and a camera. Background Art

[0002] New technologies that organically combine artificial intelligence with the automotive industry have begun to attract people's attention and attention. Cars equipped with camera sensors and combined with the overall car can complete the collection, analysis and decision-making of surrounding environment data. However, under harsh environmental conditions, radar sensors are more robust to environmental conditions than camera sensors. Therefore, the joint application of lidar and camera is more extensive. Obtaining the relationship between the lidar coordinate system and the camera coordinate system is the prerequisite for data fusion between the two.

[0003] There are many methods for joint calibration of sensors, and the feature point method is the most commonly used and relatively simple calibration method. The calibration error mainly exists in the design of the calibration plate, the determination of the feature points and the corresponding pixel points, and the calibration method. There is currently no unified standard for the calibration plate for the joint calibration of LiDAR and cameras, and its design depends on the choice of calibration method. For the calibration of the camera, a circular calibration plate can be used, and the calibration and distortion correction are performed using the detected center point of the circle. The two-dimensional LiDAR scanning line is invisible, and the position and coordinates of the corresponding feature points cannot be accurately obtained in the image captured by the camera. Therefore, the present invention designs a calibration plate with a hollow area and a non-hollow area to solve the calibration between the two-dimensional LiDAR and the camera. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a new target design and joint calibration method for single-line laser radar and camera. The present invention can feedback the distance information of the point cloud line segment by scanning the calibration plate with a two-dimensional laser radar, so as to obtain the coordinates of the point cloud. The intersection of the scanning line and the boundary of the hollow area and non-hollow area of ​​the calibration plate is the feature point. The point cloud close to the feature point is optimized by the LM algorithm, and then a straight line fitting is performed, the feature points are extracted to establish a constraint equation, and the feature points that meet the requirements are screened. Finally, the parameter fitting method is used to calculate the transformation matrix of the posture relationship between the camera and the laser radar.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A target design and joint calibration method for a single-line laser radar and a camera, comprising the following steps:

[0007] (1) Adjust the position and height of the calibration plate and place a calibration plate with hollow areas and non-hollow areas in the field of view of the camera and lidar;

[0008] (2) The laser radar scans the calibration plate and takes a photo with a camera. The laser radar scans to obtain a series of point cloud information. The point cloud at the intersection of the scanning line and the boundary line of the non-hollowed and hollowed areas of the calibration plate is used as a feature point. The point cloud close to the feature point is selected from the multiple point clouds, and after being optimized by the LM algorithm, a straight line fitting is performed;

[0009] (3) Calculate the coordinates in the calibration plate coordinate system by using the straight line fitting equation and the boundary line equation;

[0010] (4) Find the coordinates of the pixel points corresponding to the lidar feature points in the camera image;

[0011] (5) Parameter fitting is used to solve the pose matrix between the lidar and the camera.

[0012] A further technical solution is that the calibration plate includes a rectangular substrate, a first square hollow area arranged on the substrate, a second hollow area and a third hollow area in the shape of an isosceles right triangle arranged on the substrate, the right side length of the first hollow area serves as the waist of the second hollow area, the third hollow area is axially symmetrical with the second hollow area, and the third hollow area is connected to the bottom corners of the second hollow area.

[0013] In a further technical solution, the equation of the straight line fitting is as shown in formula (1):

[0014]

[0015] Where xi,yi are the coordinates of the point cloud.

[0016] In a further technical solution, in the step (3), the specific method for calculating the coordinates of the feature points in the calibration plate coordinate system is as follows: the intersection points A, B, C, and D of the scanning line and the boundary lines of the non-hollow and hollow areas of the calibration plate are set as feature points, the distance between the installation position of the laser radar and the bottom of the calibration plate is set as height h, and its installation position is adjusted to find the initial point cloud falling on the left side of the calibration plate. in The coordinate system is established with the lower left corner of the calibration plate as the origin and the horizontal direction as the x-axis. The straight-line distance L from the fixed position of the laser radar to the calibration plate is:

[0017]

[0018] Where, β is the radar elevation angle;

[0019] Then the coordinates of each feature point in the calibration plate coordinate system are:

[0020]

[0021] Where h is the ordinate length of point cloud E, is the angle from the laser radar signal emission point to the feature point; α is the radar roll angle; ρ i The distance from the laser radar to the scanning point.

[0022] In a further technical solution, the specific method of step (4) is:

[0023] Find the pixel point corresponding to the laser radar feature point in the camera image, and set the coordinates of the pixel point corresponding to the feature point to be A′(u A ,v A ), B′(u B ,v B ), C′(u C ,v C ), D′(u D ,v D ), then we get 4 sets of coordinate point pairs corresponding to the lidar data and image data, and the coordinate system sequence corresponding to the lidar data points and the camera image pixel points is as follows:

[0024] {(x A ,y A ,u A ,v A ),(x B ,y B ,u B ,v B ),(x C ,y C ,u C ,v C ),(x D ,y D ,u D ,v D )}

[0025] According to a further technical solution, step (5) comprises:

[0026] Within the common field of view of the LiDAR and the camera, each LiDAR data point (x l ,y l ) has a unique corresponding pixel point (u,v) in the camera's image; the positional relationship between the laser radar and the camera is:

[0027]

[0028] Assuming that there are n sets of feature points satisfying the above equation, according to formula (4), we can get:

[0029]

[0030] In the formula, A 9×1 =(a 11 ,a12 ,a 13 ,a 21 ,a 22 ,a 23 ,a 31 ,a 32 ,a 33 ) T , input the coordinate point pair corresponding to the laser radar data and image data of the feature point, and then calculate A 9×1 , that is, the pose matrix between the lidar and the camera is obtained.

[0031] Beneficial Effects

[0032] 1. The present invention designs the calibration plate into hollow areas and non-hollow areas. The hollow areas are selected with special geometric shapes to make the calculation simple and convenient. At the same time, the feature points are selected on the boundary line, which is conducive to marking the feature points, reducing the influence of the laser radar scanning frequency on the point cloud distribution, and ensuring that the pixel coordinates extracted from the camera image are closer to the true value. The squares and isosceles right triangles cut in the hollow areas can be used to measure the roll angle and pitch angle of the laser radar; the square part can complete the calibration of the camera. The size of the calibration plate can be adjusted according to the design requirements to ensure accuracy.

[0033] 3. The calibration method of the present invention adopts the feature point method, obtains the precise coordinates of the feature points through mathematical deduction, and then extracts the corresponding image pixels to ensure the accuracy of the feature point pairs.

[0034] 4. When fitting a straight line, the present invention takes into account that if there are discrete point clouds in the data, there will be large errors in the fitting process. Therefore, the present invention first uses the LM least squares method to process the point cloud and then fits the straight line to it, which reduces the error and makes the fitting straight line closer to the real laser radar scanning line, ensuring the accuracy of the calibration result. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of the target design and joint calibration method for single-line laser radar and camera of the present invention;

[0036] Figure 2 It is the structural diagram of the calibration board;

[0037] Figure 3 Schematic diagram of the calibration board (with target board)

[0038] Figure 4 Coordinate system relationship diagram

[0039] Figure 5 Roll angle calibration diagram

[0040] Figure 6 Pitch angle calibration diagram

[0041] Figure 7 This is the joint calibration diagram.

[0042] Markings in the figure: 1, substrate, 2, first hollow area, 3, second hollow area, 4, third hollow area, 5, target plate, 6, target point. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the present invention, a more detailed and complete description of the working process and functional implementation of the embodiments of the present invention will be provided in conjunction with the accompanying drawings provided with the embodiments of the present invention.

[0044] Example

[0045] A target design and joint calibration method for a single-line laser radar and a camera, comprising the following steps:

[0046] (1) Adjust the position and height of the calibration plate and place a calibration plate with hollow areas and non-hollow areas in the field of view of the camera and lidar;

[0047] (2) The laser radar scans the calibration plate and takes photos with a camera. The laser radar scans to obtain a series of point cloud information. The point cloud at the intersection of the scanning line and the boundary line between the non-hollowed area and the hollowed area of ​​the calibration plate is used as a feature point. The point cloud close to the feature point is selected from the multiple point clouds, optimized by the LM algorithm, and then a straight line fitting is performed;

[0048] (3) Calculate the coordinates in the calibration plate coordinate system by using the straight line fitting equation and the boundary line equation;

[0049] (4) Find the pixel coordinates corresponding to the lidar feature points in the camera image; (5) Parameter fitting is used to solve the pose matrix between the lidar and the camera.

[0050] like Figure 1 As shown, the calibration plate includes a rectangular substrate 1, a first square hollow area 2 arranged on the substrate, a second hollow area and a third hollow area in the shape of an isosceles right triangle arranged on the substrate, the right side length of the first hollow area 2 serves as the waist of the second hollow area 3, the third hollow area 4 is axially symmetrical with the second hollow area 3, and the third hollow area 4 is connected to the bottom corner of the second hollow area 3.

[0051] The calibration plate further comprises a target plate 5 matched with the first hollow area 2, and the target plate 5 is provided with a plurality of rows of circular target points 6 arranged at equal intervals.

[0052] The calibration plate of the present invention can meet the needs of individual calibration and joint calibration of multiple sensors, save costs, and make the calibration process more convenient to operate on the basis of reducing errors. In order to improve the accuracy of data collection, the size of the calibration plate should be increased as much as possible so that there is enough distance between the feature points.

[0053] Take the calibration plate size of 86cm×40cm as an example:

[0054] The calibration plate is made of PVC board and is divided into a hollow area and a non-hollow area. The first hollow area is a square ABCD with a side length of 22 cm, the second hollow area is an isosceles right triangle BCE with a right-angle side length of 22 cm, and the third hollow area is an isosceles right triangle FEG. The first hollow area can complete the calibration of the laser radar; when it is a non-hollow area, placing the target plate on the first hollow area can complete the calibration of the internal and external parameters of the camera and the calibration of the laser radar roll angle. At the same time, the first hollow area can complete the calibration of the laser radar roll angle. The isosceles right triangle can complete the calibration of the laser radar pitch angle, and the special triangle facilitates the extraction and verification of the laser radar feature points.

[0055] The calibration plate is designed to be divided into a hollow area and a non-hollow area. In an environment where there are almost no extra obstacles within the laser radar scanning range, the laser radar can obtain a series of point cloud information by scanning the non-hollow area of ​​the calibration plate, while there is no point cloud information when scanning the hollow area, which can meet the extraction of feature points.

[0056] Since the point cloud distribution of the laser radar may not be on a straight line, it presents a curved distribution state. Therefore, in order to make the final fitted straight line closer to the accurate shape, the LM algorithm is used to optimize the initial point cloud distribution first. The steps are as follows:

[0057] Assuming that the curve equation of the initial point cloud distribution is f(x), the LM algorithm optimization problem is defined as follows:

[0058] For f(x) at x n The first-order Taylor expansion is:

[0059] f(x+x n )≈f(x n )+J(x n )Δx

[0060] The damping method is used to iterate the above formula to find the optimal solution.

[0061]

[0062] Let the derivative of Δx equal to 0, and the optimal solution is:

[0063] Δx * =-(J(x n ) T J(x n )+μI) -1 J(x n ) T f(xn )

[0064] Where I is the unit matrix, μ is the damping parameter, and J(x n ) is the Jacobian matrix of f(x).

[0065] According to the optimal solution, the iterative increment Δx is calculated, the gain rate ρ is calculated through the iterative increment, and the damping coefficient is adjusted according to the gain rate:

[0066]

[0067] The gain rate ρ refers to the ratio of the first-order Taylor expansion approximate function to the true function. The closer it is to 1, the more accurate the approximation is.

[0068] The next iteration increment is solved in the same way. After each iteration increment is solved, the optimized point cloud is solved by the basic recursive formula. After all point clouds are optimized, straight line fitting is performed.

[0069] After optimization, a straight line fitting is performed to obtain the straight line fitting equation, as shown in formula (1):

[0070]

[0071] In the formula, x i ,y i are the coordinates of the point cloud.

[0072] The position of the characteristic point of the present invention is on the boundary between the hollow area and the non-hollow area, that is, finding the characteristic point on the edges ad, be, fe and fg respectively. The laser radar scans the calibration plate in the largest possible range to improve the accuracy of the experiment, and its scanning line passes through the selected boundary line.

[0073] The specific method for calculating the coordinates of the feature points in the calibration plate coordinate system is as follows: set the intersection points A, B, C, and D of the scanning line and the boundary lines of the non-hollow and hollow areas of the calibration plate as feature points, the distance between the installation position of the laser radar and the bottom of the calibration plate as height h, adjust its installation position to find the initial point cloud that falls on the left side of the calibration plate in The coordinate system is established with the lower left corner of the calibration plate as the origin and the horizontal direction as the x-axis. The straight-line distance L from the fixed position of the laser radar to the calibration plate is:

[0074]

[0075] Where, β is the radar elevation angle;

[0076] like Figure 6As shown, adjust the laser radar position, send out laser signals O0E and O0B, and the angle between the two lines is θ, where BB0 and O0E are perpendicular to O0B0. When i takes the value of 2, the coordinates of point B in the laser radar coordinate system can be obtained by formula (16), and the distance from point B to the plane where the radar is installed can be obtained by the Pythagorean theorem, that is, the length of BF, so that the coordinates of point B in the calibration plate coordinate system can be obtained. Similarly, when i takes the values ​​of 1, 3, and 4, the coordinates of points A, C, and D can be obtained. Therefore, the coordinates of the four required feature points in the calibration plate coordinate system can be calculated from a series of point clouds obtained by scanning the calibration plate by the laser radar:

[0077] Then the coordinates of each feature point in the calibration plate coordinate system are:

[0078]

[0079] Where h is the ordinate length of point cloud E, is the angle from the laser radar signal emission point to the feature point; α is the radar roll angle; ρ i The distance from the laser radar to the scanning point.

[0080] The calculation method of the roll angle is:

[0081] like Figure 3 As shown in the figure, the roll angle of the square cut from the first hollow area is calculated, and the points where the laser radar emission scanning line falls on the left and right boundary lines of the square are point E and point F respectively, ∠EOF, L OE , L OF It can be measured by laser radar and L can be obtained by the cosine theorem EF The numerical value of the roll angle α is:

[0082]

[0083] In the above formula, L AB Represents the side length of the first hollow area;

[0084] The calculation method of the pitch angle is:

[0085] like Figure 4 As shown, two isosceles right triangle plates cut from the second hollow area and the third hollow area are used to measure the pitch angle, the first isosceles right triangle plate is marked as △A1B1C1, the second isosceles right triangle plate is separated from the first isosceles right triangle plate by a certain distance and placed in the same posture, marked as △A2B2C2, the laser radar is located in front of the two triangle plates for scanning, the intersection points of the scanning line with △A1B1C1 are marked as E1 and F1, and the intersection points with △A2B2C2 are marked as E2 and F2, ∠E1OF1, L OE1 , LOF1 It can be measured by laser radar, and L can be obtained by the cosine theorem E1F1 , draw a straight line E1Z1 perpendicular to the straight line B1C1, and we get We get E1Z1=Z1C1, so Similarly, we can obtain but Then the solution formula for the pitch angle is:

[0086]

[0087] An embodiment of the present invention provides a method for calibrating a laser radar deflection angle.

[0088] Specifically, the straight line fitting equation of the feature point is approximately the laser radar scanning line, and the arc tangent value of the slope of the straight line is the deflection angle of the laser radar.

[0089] θ=tan -1 k (8)

[0090] In the above formula, k represents the slope of the pixel line.

[0091] The laser radar data points are invisible. In order to obtain the pixel coordinates in the image corresponding to the feature points, the series of points obtained by scanning the laser radar calibration plate are screened out to select four points on the boundary line or near the boundary line, and then the linear fitting is performed to obtain the linear equation. The linear equation L can be obtained through the dimension data designed by the calibration plate. ad , L be , L ef , L fg , the simultaneous equations are solved to obtain the coordinates of the feature points on the boundary line. The positions of the feature points are marked on the calibration plate, and the coordinates of the corresponding feature points are extracted in the camera image to obtain the pixel coordinates.

[0092] Find the pixel point corresponding to the laser radar feature point in the camera image, and set the coordinates of the pixel point corresponding to the feature point to be A′(u A ,v A ), B′(u B ,v B ), C′(u C ,v C ), D′(u D ,v D ), then we get 4 sets of coordinate point pairs corresponding to the lidar data and image data, and the coordinate system sequence corresponding to the lidar data points and the camera image pixel points is as follows:

[0093] {(x A ,y A ,u A ,v A ),(x B ,yB ,u B ,v B ),(x C ,y C ,u C ,v C ),(x D ,y D ,u D ,v D )}

[0094] In a further technical solution, the specific method of step (5) is:

[0095] like Figure 2 and Figure 6 As shown in the figure, the conversion relationship between the laser radar coordinate system and the reference coordinate system is:

[0096]

[0097] In the above formula, the reference coordinate system O w -x w y w z w ; Laser radar coordinate system coordinate O w -x l y l

[0098] The conversion relationship between the reference coordinate system and the camera coordinate system is:

[0099]

[0100] The coordinate transformation from two-dimensional laser radar to camera is obtained by equations (8) and (9):

[0101]

[0102] The homogeneous coordinates of a point in the three-dimensional space in the camera coordinate system are expressed as p = (x c ,y c ,z c ,1), the coordinates of the point in the imaging plane are expressed as p′=(u,v,1), then

[0103]

[0104] In the above formula, f x and f y They represent the focal length of the camera in the horizontal direction of the x-axis and the horizontal direction of the y-axis respectively; (u0,v0) represents the coordinates of the center of the image pixel; the image coordinate system is O-uv.

[0105] The extrinsic calibration problem between a two-dimensional lidar and a camera is the problem of solving the mapping matrix between lidar data points and two-dimensional image pixels.

[0106] Within the common field of view of the LiDAR and the camera, each LiDAR data point (x l ,y l ) has a unique corresponding pixel point (u, v) in the camera's image; the positional relationship between the lidar and the camera is:

[0107]

[0108] Assuming that there are n groups of feature points satisfying the above equation, we can get:

[0109]

[0110] In the formula, A 9×1 =(a 11 ,a 12 ,a 13 ,a 21 ,a 22 ,a 23 ,a 31 ,a 32 ,a 33 ) T , input the coordinate point pair corresponding to the laser radar data and image data of the feature point, and then calculate A 9×1 .

Claims

1. A joint calibration method for a single-line laser radar and a camera, characterized in that: The following steps are involved: (1) Adjust the position and height of the calibration plate and place a calibration plate with hollow areas and non-hollow areas in the field of view of the camera and lidar; (2) The laser radar scans the calibration plate and takes a photo with a camera. The laser radar scans to obtain a series of point cloud information. The point cloud at the intersection of the scanning line and the boundary line of the non-hollowed and hollowed areas of the calibration plate is used as a feature point. The point cloud close to the feature point is selected from the multiple point clouds, and after being optimized by the LM algorithm, a straight line fitting is performed; (3) Calculate the coordinates in the calibration plate coordinate system by using the straight line fitting equation and the boundary line equation; (4) Find the coordinates of the pixel points corresponding to the lidar feature points in the camera image; (5) Parameter fitting to solve the pose matrix between the lidar and the camera; The calibration plate includes a rectangular substrate, a first square hollow area arranged on the substrate, a second hollow area and a third hollow area in the shape of an isosceles right triangle arranged on the substrate, the right side length of the first hollow area serves as the waist of the second hollow area, the third hollow area is axially symmetrical with the second hollow area, and the third hollow area is connected to the bottom corners of the second hollow area.

2. The joint calibration method of a single-line laser radar and a camera according to claim 1, characterized in that: The calibration plate also includes a target plate matched with the first hollow area, and the target plate is provided with a plurality of rows of circular target points arranged at equal intervals.

3. The joint calibration method of a single-line laser radar and a camera according to claim 1, characterized in that: The equation of the straight line fitting is shown in formula (1): (1) Where xi, yi are the coordinates of the point cloud.

4. The joint calibration method of a single-line laser radar and a camera according to claim 1, characterized in that: In the step (3), the specific method for calculating the coordinates of the feature points in the calibration plate coordinate system is as follows: the intersection points A, B, C, and D of the scanning line and the boundary lines of the non-hollow and hollow areas of the calibration plate are set as feature points, the distance between the installation position of the laser radar and the bottom of the calibration plate is the height h, and its installation position is adjusted to find the initial point cloud Q that falls on the left side of the calibration plate. ,in ; Take the lower left corner of the calibration plate as the coordinate origin, and the horizontal direction is x The coordinate system is established by the axis, and the straight-line distance L from the fixed position of the laser radar to the calibration plate is: (2) In the formula, is the radar elevation angle; Then the coordinates of each feature point in the calibration plate coordinate system are: (3) Where h is the ordinate length of point cloud E, is the angle from the laser radar signal emission point to the feature point; is the radar roll angle; The distance from the laser radar to the scanning point.

5. The joint calibration method of a single-line laser radar and a camera according to claim 1, characterized in that: The step (5) comprises: Assume that each point cloud of the laser radar The corresponding pixel point exists and is unique in the camera image ; Then the positional relationship between the laser radar and the camera is: (4) Assuming that there are n groups of feature points satisfying the above equation, we can get: (5) In the formula, , is the external parameter of the pose matrix. By inputting the coordinate point pair data corresponding to the laser radar data and image data of the feature points, the external parameter can be calculated. .

Images