Accurate calibration method for LiDAR and visible light camera

Abstract

The present invention discloses a precise calibration method for a laser radar and a visible light camera, which uses two near and far targets as calibration tools, wherein the near target is a chessboard, which is placed in the field of view of the camera and is used to estimate the camera's posture; wherein the far target is a trihedron in a structured scene, which is placed in the field of view of the laser radar and is used to estimate the laser radar's posture. During the calibration process, the spatial relative position of the trihedron and the chessboard is kept unchanged, and the posture of the laser radar and camera (LiDAR-Camera) device is adjusted, multiple groups of observation data are collected, and the relative posture of the laser radar and the camera is obtained by solving the hand-eye calibration equation. The present invention avoids the problem of large matching errors in cross-modal data feature extraction in single-target calibration by introducing near and far dual targets to estimate the postures of the laser radar and the camera respectively, and has higher calibration accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of space environment perception technology, and relates to cross-modal data fusion between a three-dimensional laser radar and a visual system, and in particular to a high-precision joint calibration method between a laser radar and a visible light camera. Background Art

[0002] LiDAR can quickly and accurately obtain distance information in the environment and has high robustness to weak texture areas. Cameras can obtain dense and rich texture information, but cannot obtain depth. The two types of data are highly complementary (sparse vs. dense, geometry vs. texture). Fusion of the two modal data can provide a more comprehensive perception of the surrounding environment. In recent years, with the increasing cost of LiDAR equipment, the technical threshold for obtaining three-dimensional point clouds has been reduced year by year. Equipment that integrates visible light images and radar depth data has been widely used in autonomous driving, industrial robot navigation, enterprise digital twins, aerial photography and other related fields. A key core technical issue in environmental perception based on radar and visible light camera equipment is the high-precision joint calibration between the two devices. From a technical point of view, the point cloud data obtained by the lidar and the visible light images obtained by the camera belong to data of different modalities. Due to the sparsity of radar point cloud data and certain systematic measurement errors (the measurement error range of the lidar in the navigation and positioning scenario is on the order of centimeters), it is not easy to find observable and reliable geometric feature correspondence between the two types of data. The joint calibration of lidar and camera is a research direction that scientific and technological workers at home and abroad are focusing on exploring.

[0003] The existing joint calibration methods of lidar and visible light camera can be roughly divided into the following two categories according to the difference in target morphology: two-dimensional target calibration method and three-dimensional target calibration method. The commonly used two-dimensional target is a checkerboard. By fitting the edges, corners or plane features of the checkerboard in the point cloud, the feature correspondence between the three-dimensional lidar and the camera is found. The literature (Y. Park, S. Yun, C. S. Won, K. Cho, K. Um, and S. Sim. Calibration between color camera and 3D lidar instruments with a polygonal planar board. Sensors, 14 (3): 5333–5353, 2014.) proposed that the black and white color difference in the checkerboard will pollute the point cloud data in the lidar and affect the accuracy of distance measurement. In the literature (Pusztai Z and Hajder L. Accurate calibration of Lidar-camera systems using ordinary boxes. In: 2017 international conference on computer vision, Venice, Italy, 22 October 2017–29 October 2017, pp. 394–402. IEEE.), Zoltan Pusztai et al. used a carton with known size as a three-dimensional target, detected the carton plane from the point cloud, and further completed the calibration between the radar and the camera. The common shortcomings of this method are: the feature extraction and matching of cross-modal data makes it difficult to achieve high accuracy in the calibration result; the distance between the target and the LiDAR-Camera device is difficult to coordinate (in order to improve the calibration accuracy of the camera, the target should be placed at a closer position in the camera field of view, and in order to suppress the system noise of the LiDAR, the target should be placed at a farther position in the LiDAR field of view). Summary of the invention

[0004] The main purpose of the present invention is to provide a precise calibration method for a laser radar and a visible light camera to overcome the deficiencies in the prior art.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0006] An embodiment of the present invention provides a calibration method for a laser radar and a visible light camera, the calibration method comprising the following steps:

[0007] A first target and a second target are set as calibration tools, and the first target is farther from the LiDAR-Camera device to be calibrated than the second target, wherein the first target is a trihedron in a structured scene, is placed in the field of view of the LiDAR in the LiDAR-Camera device, and is used to estimate the position and posture of the LiDAR, and the second target is a checkerboard, is placed in the field of view of the camera in the LiDAR-Camera device, and is used to estimate the position and posture of the camera;

[0008] Calibrate the LiDAR-Camera device, keep the spatial relative position of the first target and the second target unchanged during the calibration process, and collect multiple groups of observation data by adjusting the posture of the LiDAR-Camera device; and

[0009] The relative position and posture of the laser radar and the camera are obtained by solving the hand-eye calibration equation.

[0010] Compared with the prior art, the advantages of the present invention include: in the calibration problem of laser radar and visible light camera, the difficulty of cross-modal data feature extraction and matching is avoided. By introducing the dual targets of near and far, this joint calibration problem is converted into a camera pose estimation based on a chessboard and a laser radar pose estimation problem based on a trihedron in a structured scene. By solving the classic hand-eye calibration equation, the relative pose of the laser radar and the camera is obtained, and a higher calibration accuracy can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the installation location of the laser radar and camera;

[0012] Figure 2 is an example of a trihedron in a structured scene;

[0013] Figure 3 It is a schematic diagram of the plane fitting of the trihedral point cloud in the structured scene;

[0014] Figure 4 This is a schematic diagram of the estimated camera pose on a chessboard;

[0015] Figure 5 This is a schematic diagram of the principle of long-range and short-range dual-target calibration. DETAILED DESCRIPTION

[0016] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0017] An embodiment of the present invention provides a calibration method for a laser radar and a visible light camera, the calibration method comprising the following steps:

[0018] A first target and a second target are set as calibration tools, and the first target is farther from the LiDAR-Camera device to be calibrated than the second target, wherein the first target is a trihedron in a structured scene, is placed in the field of view of the LiDAR in the LiDAR-Camera device, and is used to estimate the position and posture of the LiDAR, and the second target is a checkerboard, is placed in the field of view of the camera in the LiDAR-Camera device, and is used to estimate the position and posture of the camera;

[0019] Calibrate the LiDAR-Camera device, keep the spatial relative position of the first target and the second target unchanged during the calibration process, and collect multiple groups of observation data by adjusting the posture of the LiDAR-Camera device; and

[0020] The relative position and posture of the laser radar and the camera are obtained by solving the hand-eye calibration equation.

[0021] Furthermore, the method further comprises the following steps:

[0022] 1) estimating the intrinsic parameters of the camera;

[0023] 2) Select a trihedron as the first target in the structured scene, adjust the position and observation angle of the laser radar so that the trihedron is placed in the field of view of the laser radar, collect a frame of point cloud data of the laser radar, obtain the three plane equations of the trihedron through the plane fitting algorithm, calculate the coordinates of the intersection of the three planes, and use them as the origin of the world coordinate system. W , and further calculate the position and posture of the lidar

[0024] 3) Place the chessboard as the second target in the camera field of view, collect a frame image of the chessboard in the camera, and calculate the camera's position and posture

[0025] 4) Keeping the spatial relative position of the trihedron and the chessboard unchanged, by adjusting the posture of the LiDAR-Camera device, repeating steps 2) to 3) to collect N groups of observation data, where N ≥ 2;

[0026] 5) Calculate the relative position R of the LiDAR-Camera device LC ,T LC ;

[0027] 6) Obtain the relative position and posture of the laser radar and the camera by solving the hand-eye calibration equation.

[0028] Furthermore, in the step 1), the Zhang Zhengyou calibration algorithm is used to estimate the intrinsic parameters of the camera.

[0029] Furthermore, the structured scene is an indoor scene, and the trihedron is two mutually perpendicular walls and a floor.

[0030] Furthermore, the distance between the trihedron and the laser radar is greater than 5 meters.

[0031] Furthermore, the LiDAR-Camera device includes a laser radar and a pair of stereo cameras, wherein the angle between the laser radar and the camera ranges from 60 to 90 degrees.

[0032] Furthermore, the step 6) includes: solving the hand-eye calibration equation based on the calibrateHandeye() function of OpenCV. The calibrateHandeye() function is a standard library function based on OpenCV to solve the hand-eye calibration equation.

[0033] Furthermore, the step 5) specifically includes:

[0034] Define the origin of the world coordinate system as the intersection of the trihedron, p W is a point in the world coordinate system. When the LiDAR-Camera device is in position 1, the relationship (1) exists:

[0035]

[0036] When the LiDAR-Camera device is in posture 2, there is a relationship (2):

[0037]

[0038] From equations (1) and (2), we can get equation (3):

[0039]

[0040] In formula (3), when p W When is a 3×1 column vector, T LC , is a column vector of the same dimension, and when p W When is a 3×3 matrix, T LC , are matrices of the same dimension, respectively

[0041] in similar;

[0042] When p W for When , formula (3) is simplified to:

[0043]

[0044] When p W for When , formula (3) is simplified to:

[0045]

[0046] Expand equations (4) and (5) separately and subtract them to obtain:

[0047]

[0048] Multiply the left and right sides of the equation (6) by take have to:

[0049]

[0050] Simplify equation (7) to AX = XB and solve it. LC Substitute into equation (4) and solve for T LC .

[0051] Furthermore, the camera is a visible light camera.

[0052] Furthermore, the method also includes obtaining the plane equation of the trihedron by a plane fitting algorithm.

[0053] The technical solution, its implementation process and principle will be further explained below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the LiDAR-Camera device to be calibrated in this embodiment includes a laser radar and a pair of stereo cameras, wherein the laser radar and the camera form a 90-degree angle. The calibration method involved in the present invention does not involve a stereo camera, and the purpose of introducing a stereo camera is to quantitatively evaluate the calibration results; the angle between the laser radar and the camera does not have to be 90 degrees, and can be flexibly adjusted according to the placement of the trihedron and the chessboard selected in the scene.

[0055] The specific implementation steps are as follows:

[0056] 1) Use Zhang Zhengyou's calibration algorithm to estimate the intrinsic parameters of the visible light camera, Where u0, v0 are the coordinates of the center of the camera plate in the pixel coordinate system, α, β are the scale factors on the two image axes, and γ is the skewness of the two image axes;

[0057] 2) Select a trihedron in the structured scene. Taking the indoor scene as an example, the trihedron can be two perpendicular walls and a floor, such as Figure 2As shown. Adjust the position and observation angle of the laser radar so that the trihedron is placed in the field of view of the laser radar. The laser radar in the navigation and positioning scene has a centimeter-level measurement error. In order to reduce the impact of the system measurement error on the calibration accuracy, the distance between the trihedron and the laser radar is greater than 5 meters. Collect a frame of point cloud data from the laser radar, and use the plane fitting algorithm to obtain the three plane equations of the trihedron, as shown in Figure 3 As shown. Calculate the coordinates of the intersection of the three planes as the origin O of the world coordinate system W , and further calculate the position and posture of the lidar

[0058] 3) Place the chessboard as the second target in the camera field of view and collect a frame of the chessboard in the camera, such as Figure 4 As shown, calculate the camera pose

[0059] 4) During the calibration process, the spatial relative position of the trihedron and the chessboard is kept unchanged. By adjusting the position and posture of the LiDAR-Camera device, re-execute steps 2) to 3) to collect N (N ≥ 2) sets of observation data.

[0060] 5) The relative position of the lidar and the camera is obtained by solving the hand-eye calibration equation. For the specific solution process, please refer to the calibrateHandeye() function of OpenCV.

[0061] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for calibrating a laser radar and a visible light camera, characterized in that include: 1) Use Zhang Zhengyou calibration algorithm to estimate the intrinsic parameters of the camera in the LiDAR-Camera device; 2) Select a trihedron as the first target in the structured scene, place it in the field of view of the laser radar in the LiDAR-Camera device, adjust the position and observation angle of the laser radar in the LiDAR-Camera device so that the trihedron is placed in the field of view of the laser radar, and collect a frame of point cloud data from the laser radar. Use the plane fitting algorithm to obtain the three plane equations of the trihedron, calculate the coordinates of the intersection of the three planes, and use them as the origin of the world coordinate system. W , and further calculate the position and posture of the lidar 3) Place the checkerboard as the second target in the camera field of view, and make the first target farther from the LiDAR-Camera device to be calibrated than the second target, and collect a frame image of the checkerboard in the camera to calculate the camera's position and posture 4) Calibrate the LiDAR-Camera device, keep the spatial relative position of the trihedron and the chessboard unchanged, adjust the posture of the LiDAR-Camera device, repeat steps 2) to 3), and collect N groups of observation data, where N ≥ 2; 5) Calculate the relative position R of the LiDAR-Camera device LC ,T LC ; 6) Obtain the relative position and posture of the laser radar and the camera by solving the hand-eye calibration equation.

2. The method according to claim 1, characterized in that: The structured scene is an indoor scene, and the trihedron is two mutually perpendicular walls and a floor.

3. The method according to claim 1, characterized in that: The distance between the trihedron and the laser radar is greater than 5 meters.

4. The method according to claim 1, characterized in that: The LiDAR-Camera device includes a laser radar and a pair of stereo cameras, wherein the angle between the laser radar and the camera ranges from 60 to 90 degrees.

5. The method according to claim 1, characterized in that Step 6) includes: solving the hand-eye calibration equation based on the calibrateHandeye() function of OpenCV.

6. The method according to claim 1, characterized in that Step 5) specifically includes: Define the origin of the world coordinate system as the intersection of the trihedron, p W is a point in the world coordinate system. When the LiDAR-Camera device is in position 1, the relationship (1) exists: When the LiDAR-Camera device is in posture 2, there is a relationship (2): From equations (1) and (2), we can get equation (3): In formula (3), when p W When is a 3×1 column vector, T LC , is a column vector of the same dimension, and when p W When is a 3×3 matrix, T LC , are matrices of the same dimension, respectively in similar; When p W for When , formula (3) is simplified to: When p W for When , formula (3) is simplified to: Expand equations (4) and (5) separately and subtract them to obtain: Multiply the left and right sides of the equation (6) by take have to: Simplify equation (7) to AX = XB and solve it. LC Substitute into equation (4) and solve for T LC .

7. The method according to claim 1, characterized in that: The camera is a visible light camera.

8. The method according to claim 1, characterized in that: The method also includes obtaining the plane equation of the trihedron by a plane fitting algorithm.

Images