Calibration board, calibration method and system for joint calibration of laser radar and depth camera

Abstract

The present invention discloses a calibration plate, a calibration method and a system for joint calibration of a laser radar and a depth camera, and belongs to the technical field of autonomous mobile robots. The present invention uses the specific shape of the calibration plate to obtain point and line constraint pairs of observation points in the laser radar coordinate system and the depth camera coordinate system. Six constraint pairs and twelve constraint equations are obtained by at least two observations, and the nine unknown variables of the 2D laser radar and depth camera coordinate transformation are solved by the least squares method, thereby realizing the joint calibration of the laser radar and the depth camera. The present invention solves the problems of the cumbersome calibration process and the asynchronous data frame time of the existing joint calibration method.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous mobile robots, and more specifically, relates to a calibration board, a calibration method and a system for joint calibration of a laser radar and a depth camera. Background Art

[0002] The fusion of 2D LiDAR and depth camera can effectively make up for the problem that the robot has few environmental perception features and is affected by ambient light, ensuring the high-precision operation of the mobile robot and greatly improving the robustness of the mobile robot system. The joint calibration of 2D LiDAR and depth camera aims to determine the coordinate transformation relationship between the 2D LiDAR and depth camera. However, since the scanning plane of the 2D LiDAR is difficult to find in the depth camera, it is difficult to directly obtain the corresponding relationship between the observation point in the LiDAR coordinate system and the depth camera coordinate system. It is necessary to use a specific calibration plate or a certain constraint relationship to obtain the corresponding position of the observation point in the two coordinate systems.

[0003] There are methods that can realize the joint calibration of 2D lidar and camera. A typical method is to capture the scanning feature information of 2D lidar through a specific hollow calibration plate, place the calibration plate on the display, and reproduce the laser points on the display through a computer. Then the camera observes the calibration plate with a laser scanning line background to obtain the corresponding points in the camera coordinate system and the lidar coordinate system, and use the least squares method to solve the coordinate transformation relationship between the camera and lidar. Another method is to use a structured environment, through wall columns or corner features, and use the points scanned by the lidar on the wall columns or corner lines observed by the camera as constraints, and use the least squares method to solve the rotation matrix and translation matrix between the lidar coordinate system and the camera coordinate system. Some existing calibration methods have a strong dependence on the environment, and observations from multiple angles are required during the calibration process, which puts higher requirements on the openness of the environment; some observations are made more frequently, requiring more than twelve observations at different angles and positions to determine the coordinate transformation relationship between the lidar and the camera. The calibration process is cumbersome and inefficient; some calibration methods achieve the calibration effect while the calibration model is moving, which brings about the problem of time synchronization of the depth camera and 2D lidar observation data. Time asynchrony will result in data frame mismatch. Summary of the invention

[0004] In view of the defects of the related art, the purpose of the present invention is to provide a calibration board, calibration method and system for the joint calibration of lidar and depth camera, aiming to solve the problems of cumbersome calibration process and data frame time asynchrony in the existing joint calibration method.

[0005] To achieve the above object, one aspect of the present invention provides a calibration plate for joint calibration of a 2D laser radar and a depth camera, comprising a first plate, a second plate, and a third plate;

[0006] The first plate and the second plate are both triangles, the three sides of the first plate are AB, BC and AC, the three sides of the first plate are AD, DC and AC, the AC sides of the first plate and the second plate are equal in length and overlap, and the first plate and the second plate are not coplanar;

[0007] The AB side of the first plate and the AD side of the second plate share a vertex, and the AB side and the AD side are located in the plane where the third plate is located.

[0008] Optionally, a fourth plate is also included;

[0009] The BC side of the first flat plate and the DC side of the second flat plate share a vertex, and the BC side and the DC side are located in the plane where the fourth flat plate is located.

[0010] Another aspect of the present invention further provides a joint calibration method based on the above calibration plate, characterized in that it comprises the following steps:

[0011] S1. Place the calibration plate in the field of view of the 2D laser radar and the depth camera to obtain observation data of the 2D laser radar and the depth camera;

[0012] S2, fitting three intersection lines between the scanning plane of the 2D laser radar and the first plate, the second plate and the third plate, obtaining three intersection points of the three intersection lines, and projecting them into the camera coordinate system, wherein the three intersection points are obtained by intersecting the three intersection lines in pairs;

[0013] S3, identifying and extracting the depth camera observation data on the calibration plate, fitting the three-dimensional coordinate points of the observation data in the camera coordinate system to obtain the plane equations of the three planes where the first plate, the second plate and the third plate are located respectively, and then obtaining the spatial straight line equations of the three intersection lines AB, AC, and AD, wherein the three intersection lines AB, AC, and AD are obtained by the intersection of two of the three planes where the first plate, the second plate and the third plate are located respectively;

[0014] S4, determine whether the coordinate conversion parameters are known at this time; if so, continue to step S5, if not, proceed to step S6;

[0015] S5, perform error check, if the error check passes, the calibration process ends; if not, execute step S6;

[0016] S6. Using the constraint relationship between the three intersection points and the spatial straight line equations of the three intersection lines AB, AC, and AD, a constraint equation is established and solved to obtain coordinate transformation parameters; then, the position and posture of the calibration model relative to the calibration plate is adjusted, and the process returns to step S1.

[0017] Furthermore, in step S2, fitting three intersection lines between the scanning plane of the 2D laser radar and the first flat plate, the second flat plate and the third flat plate, and obtaining three intersection points of the three intersection lines includes:

[0018] Identify and extract the observation data of the 2D laser radar falling on the calibration plate, use the distance and slope between two adjacent points to identify and segment the straight line features, and fit the segmented observation points into three straight lines; combine any two of the three straight lines to obtain the coordinates of the three intersection points in the 2D laser radar coordinate system.

[0019] Furthermore, in step S2, the three intersection points are projected into the camera coordinate system according to the following formula:

[0020]

[0021] In the formula, Represents the rotation matrix from the 2D lidar coordinate system to the camera coordinate system, Represents the translation matrix from the 2D lidar coordinate system to the camera coordinate system, is the point coordinate in the 2D laser radar coordinate system, is the point coordinate in the camera coordinate system.

[0022] Furthermore, the step S5 specifically includes:

[0023] Change the posture of the calibration model or the calibration plate, solve the straight line equations of the three intersection points and the three intersection lines AB, AC, and AD, calculate the distances from the three intersection points to the three intersection lines AB, AC, and AD, and if they are less than or equal to the set threshold, it is determined that the error check has passed and no correction is required; if they are greater than the set threshold, execute step S6.

[0024] Furthermore, in step S6, the constraint equation is solved using the least squares method.

[0025] Furthermore, in step S1, the left camera coordinate system of the depth camera is set as the reference coordinate system.

[0026] Furthermore, in step S3, identifying and extracting the depth camera observation data on the calibration plate includes:

[0027] Pixel points of the first plate, the second plate and the third plate observed by the depth camera are extracted, and the three-dimensional coordinate values ​​of the corresponding pixel points in the reference coordinate system are obtained according to the SDK function provided by the depth camera.

[0028] Another aspect of the present invention further provides a calibration system for joint calibration of a 2D laser radar and a depth camera, comprising: a computer-readable storage medium and a processor;

[0029] The computer-readable storage medium is used to store executable instructions;

[0030] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the above-mentioned joint calibration method.

[0031] Compared with the prior art, the above technical solution conceived by the present invention can achieve the following beneficial effects:

[0032] 1. The calibration process is simple and efficient. You only need to place the calibration plate where the 2D LiDAR and the depth camera can observe together, and the coordinate transformation relationship between the 2D LiDAR and the depth camera can be solved through the observation data at two different positions and angles.

[0033] 2. After each posture adjustment in the joint calibration method of the present invention, the calibration model and the calibration plate are stationary, thus avoiding the problem of time asynchrony between the 2D laser radar and the depth camera data frames.

[0034] 3. The calibration plate is easy to make. The calibration plate only needs to ensure that two intersecting triangular faces with a certain angle intersect with another plane at the same time, and the three faces intersect each other to form three intersecting straight lines. There is no specific angle requirement for verticality and the angle of the triangular faces. Ordinary 3D printing can complete the production of the calibration plate.

[0035] 4. No restrictions on installation conditions and environmental conditions. During the calibration process, there are no restrictions on the horizontal and tilted installation of the 2D laser radar and depth camera, and there is no requirement for whether the ground of the calibration environment is level. There is no requirement for the placement of the calibration plate, as long as the scanning point of the 2D laser radar can fall on the key feature surface.

[0036] 5. Strong anti-interference ability. The solution obtained from the first two observations is used as the initial value, and multiple observations are made again. The online correction of the 2D lidar and depth camera calibration results is realized based on the distance from the point to the line being less than a certain threshold.

[0037] 6. Wide application range. In addition to realizing the joint calibration of 2D laser radar and depth camera, the calibration board and calibration method of the present invention can also realize the installation posture calibration of 2D laser radar. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a calibration plate used in an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a calibration system in an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of a calibration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] One aspect of the present invention provides a calibration plate for joint calibration of a 2D laser radar and a depth camera, comprising a first plate, a second plate, and a third plate;

[0043] The first plate and the second plate are both triangles, the three sides of the first plate are AB, BC and AC, the three sides of the first plate are AD, DC and AC, the AC sides of the first plate and the second plate are equal in length and overlap, and the first plate and the second plate are not coplanar;

[0044] The AB side of the first plate and the AD side of the second plate share a vertex, and the AB side and the AD side are located in the plane where the third plate is located.

[0045] Optionally, a fourth plate is also included;

[0046] The BC side of the first flat plate and the DC side of the second flat plate share a vertex, and the BC side and the DC side are located in the plane where the fourth flat plate is located.

[0047] Another aspect of the present invention further provides a joint calibration method based on the above calibration plate, characterized in that it comprises the following steps:

[0048] S1. Place the calibration plate in the field of view of the 2D laser radar and the depth camera to obtain observation data of the 2D laser radar and the depth camera;

[0049] S2, fitting three intersection lines between the scanning plane of the 2D laser radar and the first plate, the second plate and the third plate, obtaining three intersection points of the three intersection lines, and projecting them into the camera coordinate system, wherein the three intersection points are obtained by intersecting the three intersection lines in pairs;

[0050] S3, identifying and extracting the depth camera observation data on the calibration plate, fitting the three-dimensional coordinate points of the observation data in the camera coordinate system to obtain the plane equations of the three planes where the first plate, the second plate and the third plate are located respectively, and then obtaining the spatial straight line equations of the three intersection lines AB, AC, and AD, wherein the three intersection lines AB, AC, and AD are obtained by the intersection of two of the three planes where the first plate, the second plate and the third plate are located respectively;

[0051] S4, determine whether the coordinate conversion parameters are known at this time; if so, continue to step S5, if not, proceed to step S6;

[0052] S5, perform error check, if the error check passes, the calibration process ends; if not, execute step S6;

[0053] S6. Using the constraint relationship between the three intersection points and the spatial straight line equations of the three intersection lines AB, AC, and AD, a constraint equation is established and solved to obtain coordinate transformation parameters; then, the position and posture of the calibration model relative to the calibration plate is adjusted, and the process returns to step S1.

[0054] It is understandable that steps S2 and S3 in the present invention can be interchanged or performed simultaneously.

[0055] The contents involved in the above embodiment are described below in conjunction with a preferred embodiment.

[0056] Figure 1 It is a schematic diagram of the calibration plate used in the embodiment of the present invention. The main feature of the calibration plate is that it is composed of two intersecting triangular faces with a certain angle that intersect with two intersecting rectangular faces at the same time. There is no restriction on the angle between the two triangular faces and the two rectangular faces. It is advisable to have as many 2D laser radar observation points as possible fall on the triangular faces and one rectangular face. The other rectangular face can be used as the support surface or working surface of the calibration plate. The size of the calibration plate mainly depends on the angular resolution of the 2D laser radar. By increasing the length of the working rectangular face and the angle between the triangular faces, more observation points can be obtained, thereby improving the calibration accuracy. The calibration system used in the embodiment of the present invention is as follows: Figure 2 As shown, it includes the above calibration board, depth camera and 2D laser radar. Figure 1 The rectangular surface in the image does not necessarily have to be a rectangle, but may be a trapezoid or other irregular shapes, as long as it is convenient to obtain the straight line EI during scanning.

[0057] When starting calibration, place the calibration board at a position where the 2D laser radar and depth camera can observe at the same time, start the 2D laser radar and depth camera, and collect the observation data of the 2D laser radar and depth camera. The data of the 2D laser radar is based on the radar coordinate system Ox l y l zl It is observed that the observation data of the depth camera is based on the left camera coordinate system Ox c y c z c (hereinafter referred to as the camera coordinate system) is observed.

[0058] To process the data of 2D laser radar, first identify and extract the observation data of 2D laser radar on the calibration plate. After extracting the observation data on the calibration plate, use the distance and slope change between two adjacent points to identify and segment the straight line features, and fit the segmented data points into three straight lines, such as Figure 2 As shown. Fit the points on the rectangular plane PBDQ into the straight line EI, the points on the plane ABC into the straight line FG, and the points on the plane ACD into the straight line GH. Combine any two of the three straight lines to find the intersection point. The intersection point is projected into the camera coordinate system according to formula (1).

[0059]

[0060] In the formula, Represents the rotation matrix from the 2D lidar coordinate system to the camera coordinate system, T l c Represents the translation matrix from the 2D lidar coordinate system to the camera coordinate system.

[0061] After projection:

[0062]

[0063] Where l represents the 2D lidar coordinate system, and i represents the i-th intersection point under 2D lidar observation.

[0064] From the projection results, we know that we only need to solve nine unknowns to obtain the rotation matrix and translation matrix for the 2D lidar and depth camera coordinate system transformation. This is done by simply combining nine independent sets of equations.

[0065] To process the data of the depth camera, first extract the pixel points of planes PBDQ, plane ABC, and plane ACD under the observation of the depth camera, and obtain the three-dimensional coordinate values ​​of the corresponding pixel points based on the camera coordinate system of the left camera according to the SDK function provided by the depth camera. Fit the three-dimensional coordinate values ​​corresponding to more than four pixel points on each plane to the plane equation, and combine any two plane equations of planes PBDQ, plane ABC, and plane ACD to obtain the spatial straight line equations of straight line AB, straight line AC, and straight line AD.

[0066] Assume that the plane equations of plane PBDQ, plane ABC, and plane ACD are:

[0067]

[0068] Then the spatial straight line equations of straight line AB, straight line AC, and straight line AD are:

[0069]

[0070] In the formula, A1, B1, C1, D1, A2, B2, C2, D2, A3, B3, C3, and D3 are known.

[0071] Since the coordinate system projection of the point does not change the geometric relationship, after the point in the 2D lidar coordinate system is projected into the camera coordinate system, its three points should be on the straight line AB, the straight line AC, and the straight line AD, that is:

[0072]

[0073] Arranged:

[0074]

[0075] Change the position and posture of the calibration model or calibration plate, and obtain the six constraint equations through the same steps again. Combine the equations of the first observation to form a hyperstatic equation group, and use the least squares method to solve the parameters of the rotation matrix and translation matrix, and then determine the coordinate transformation of the 2D lidar and depth camera.

[0076] Error check: After solving the rotation matrix and translation matrix from the 2D lidar coordinate system to the camera coordinate system, change the position of the calibration model or calibration board, solve the intersection of F, G, H and the straight line equation under the observation of the depth camera, solve the distance from the point to the straight line, set the distance threshold, if the solution result is less than or equal to the set threshold, retain the current coordinate transformation parameters and the calibration ends; if the solution result is greater than the set threshold, the check fails, and the point-line constraint equation is reconstructed using the previous method, and the coordinate transformation parameters are re-solved using the least squares method in the previous set of equations.

[0077] An embodiment of the present invention further provides a calibration system for joint calibration of a 2D laser radar and a depth camera, comprising: a computer-readable storage medium and a processor;

[0078] The computer-readable storage medium is used to store executable instructions;

[0079] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the above-mentioned joint calibration method.

[0080] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A laser radar and depth camera joint calibration method, based on a 2D laser radar and depth camera joint calibration plate, characterized in that: The calibration plate includes a first plate, a second plate and a third plate; The first plate and the second plate are both triangles, the three sides of the first plate are AB, BC and AC, the three sides of the first plate are AD, DC and AC, the AC sides of the first plate and the second plate are equal in length and overlap, and the first plate and the second plate are not coplanar; The AB side of the first plate and the AD side of the second plate share a vertex, and the AB side and the AD side are located in the plane where the third plate is located; The joint calibration method comprises the following steps: S1. Place the calibration plate in the field of view of the 2D laser radar and the depth camera to obtain observation data of the 2D laser radar and the depth camera; S2, fitting three intersection lines between the scanning plane of the 2D laser radar and the first plate, the second plate and the third plate, obtaining three intersection points of the three intersection lines, and projecting them into the camera coordinate system, wherein the three intersection points are obtained by intersecting the three intersection lines in pairs; S3, identifying and extracting the depth camera observation data on the calibration plate, fitting the three-dimensional coordinate points of the observation data in the camera coordinate system to obtain the plane equations of the three planes where the first plate, the second plate and the third plate are located respectively, and then obtaining the spatial straight line equations of the three intersection lines AB, AC, and AD, wherein the three intersection lines AB, AC, and AD are obtained by the intersection of two of the three planes where the first plate, the second plate and the third plate are located respectively; S4, determine whether the coordinate conversion parameters are known at this time; if so, continue to step S5, if not, proceed to step S6; S5, perform error check, if the error check passes, the calibration process ends; if not, execute step S6; S6. Using the constraint relationship between the three intersection points and the spatial straight line equations of the three intersection lines AB, AC, and AD, a constraint equation is established and solved to obtain coordinate transformation parameters; then, the position and posture of the calibration model relative to the calibration plate is adjusted, and the process returns to step S1.

2. The joint calibration method according to claim 1, characterized in that: In the step S2, fitting three intersection lines between the scanning plane of the 2D laser radar and the first plate, the second plate and the third plate, and obtaining three intersection points of the three intersection lines comprises: Identify and extract the observation data of the 2D laser radar falling on the calibration plate, use the distance and slope between two adjacent points to identify and segment the straight line features, and fit the segmented observation points into three straight lines; combine any two of the three straight lines to obtain the coordinates of the three intersection points in the 2D laser radar coordinate system.

3. The joint calibration method according to claim 2, characterized in that: In step S2, the three intersection points are projected into the camera coordinate system according to the following formula: In the formula, Represents the rotation matrix from the 2D lidar coordinate system to the camera coordinate system, T l c Represents the translation matrix from the 2D lidar coordinate system to the camera coordinate system, is the point coordinate in the 2D laser radar coordinate system, is the point coordinate in the camera coordinate system.

4. The joint calibration method according to claim 1, characterized in that: The step S5 specifically includes: Change the posture of the calibration model or the calibration plate, solve the straight line equations of the three intersection points and the three intersection lines AB, AC, and AD, calculate the distances from the three intersection points to the three intersection lines AB, AC, and AD, and if they are less than or equal to the set threshold, it is determined that the error check has passed and no correction is required; if they are greater than the set threshold, execute step S6.

5. The joint calibration method according to claim 1, characterized in that: In step S6, the constraint equation is solved using the least squares method.

6. The joint calibration method according to claim 1, characterized in that: In the step S1, the left camera coordinate system of the depth camera is set as the reference coordinate system.

7. The joint calibration method according to claim 6, characterized in that: In step S3, identifying and extracting the depth camera observation data on the calibration plate includes: Pixel points of the first plate, the second plate and the third plate observed by the depth camera are extracted, and the three-dimensional coordinate values ​​of the corresponding pixel points in the reference coordinate system are obtained according to the SDK function provided by the depth camera.

8. A calibration system for joint calibration of 2D laser radar and depth camera, characterized in that: include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the joint calibration method described in any one of claims 1 to 7.

Images