All-round panoramic camera and laser radar full-automatic calibration method based on natural environment

Abstract

The invention relates to a full-automatic calibration method for an all-round panoramic camera and a laser radar based on a natural environment, belongs to the technical field of data fusion of an environment sensing system, and aims to get rid of limitation of calibration equipment and automatically obtain high-precision external parameters according to an internal parameter of the all-round panoramic camera and a feature point matching relation between panoramic image frames. Estimating the motion of the panoramic camera, and estimating the motion of the laser radar by using the feature matching relationship of the point cloud data of the laser radar; through pose estimation, the initial external parameters between the panoramic camera and the laser radar are calculated in a hand-eye calibration mode, and the external parameters are further optimized by using an edge matching method, so that full-automatic external parameter calibration of the panoramic camera and the laser radar is realized, manual intervention is not needed, and the calibration efficiency is improved. And the line feature matching relationship between the panoramic image and the point cloud information is further utilized to optimize the external parameter, so that higher-precision external parameter solving is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of environmental perception system data fusion. Background Art

[0002] With the rapid development of science and technology and manufacturing, mobile robots play an increasingly important role in human social life. Mobile robots can accurately reconstruct the surrounding environment map by integrating laser data sensors and visual sensors, and have stronger anti-interference ability for changes in the external environment. At the same time, they can obtain more accurate information about the environment, and can realize dynamic target detection and relocation functions on this basis. How to reasonably integrate the digital information obtained by various types of sensors has become an important development direction in the research of mobile robot environmental perception systems. Among them, joint calibration through laser radar combined with camera is a reasonable and feasible calibration method.

[0003] Existing calibration methods based on calibration equipment, such as the Chinese patent with publication number CN105678783B, discloses a calibration method for data fusion of a catadioptric panoramic camera and a laser radar. The data fusion calibration structure includes a laser radar and a single-viewpoint catadioptric panoramic camera installed on the body of the environment perception system; the joint calibration method includes: 1. calibration of the camera internal parameter K; 2. calibration of the refraction point parameter X of the catadioptric mirror surface; m， Y m， Z m Solution; 3. Panoramic camera world coordinate point parameter X w， Y w， Z w Solution; 4. Measurement of world coordinate point parameters of LiDAR; 5. Joint calibration of panoramic camera and LiDAR. The above patent can effectively calibrate the internal parameters of the panoramic camera, and thus provide a reasonable, fast and effective solution to the ranging and positioning problem in the environmental perception system.

[0004] However, the above patent cannot automatically establish the corresponding feature relationship between the panoramic image and the lidar point cloud, and requires manual intervention to establish the corresponding relationship. Therefore, it is impossible to achieve high-precision solution of external parameters. The calibration of panoramic cameras and lidars is an important part of the fusion of the two. Accurate external parameters often play a positive role in the fusion of the two information. The calibration method based on the calibration equipment needs to rely on special calibration targets and special layout of the scene, which makes the calibration not flexible enough.

[0005] Calibration methods based on natural environments, such as those based on edge feature matching, do not need to rely on specially made calibration targets. However, they require manual measurement to obtain the initial values ​​of external parameters, cannot achieve automatic calibration, and the accuracy of external parameters is difficult to reach the pixel level. Summary of the invention

[0006] In order to get rid of the limitation of calibration equipment and automatically obtain high-precision external parameters, the present invention proposes a "fully automatic calibration method for surround view panoramic cameras and lidar based on natural environment".

[0007] like Figure 1 As shown, the fully automatic calibration method of the surround view camera and the lidar based on the natural environment includes the following steps:

[0008] 1) Calibrate the intrinsic parameters of the surround-view panoramic camera: Using the panoramic camera spherical model, u = π(·)P describes the mapping relationship between the spatial point P of the surround-view panoramic camera and the corresponding pixel point u, where π(·) represents the generalized projection process. By calibrating the intrinsic parameters of the surround-view panoramic camera, the specific parameters of the projection process are obtained, and the mapping relationship between the spatial point P and the corresponding pixel point u is accurately established;

[0009] 2) Perform feature extraction and feature matching on panoramic images and lidar point cloud data respectively;

[0010] 3) Estimate the motion of the surround view camera based on the internal parameters of the surround view camera and the matching relationship between the feature points of the panoramic image frames, and estimate the motion of the lidar using the feature matching relationship of the lidar point cloud data;

[0011] 4) The surround view camera and the LiDAR are rigidly connected. During the movement, the relative posture remains unchanged. Through posture estimation, the initial external parameters between the surround view camera and the LiDAR are calculated using the hand-eye calibration method:

[0012] The constraint between the motion pose matrix and the external parameter matrix of the surround view panoramic camera and the laser radar is AX=XB. Matrix A is the motion pose transformation matrix of the surround view panoramic camera, matrix B is the motion pose transformation matrix calculated by the laser radar odometer, X is the external parameter matrix between the surround view panoramic camera and the laser radar, R, t is used to represent the rotation and translation relationship between the surround view panoramic camera and the laser radar, and the motion pose of the laser radar is represented by T L =[R L ,t L ], the motion pose of the panoramic camera is represented by T C =[R C ,t C ], according to the hand-eye calibration model, the following relationship is satisfied:

[0013] R L R=RR C ,

[0014] R L t+t L =Rt C +t,

[0015] When using a monocular panoramic camera to estimate the camera's pose, the calculated pose information has scale uncertainty. Therefore, the pose information calculated by the panoramic camera and the lidar has a certain scale uncertainty. C and t L There is a difference of size s between them, which can be expressed using the formula:

[0016] R L t+t L =Rt C ·s+t,

[0017] Using the above constraints, we construct the objective function Optimize the external parameters R, t and s, and obtain the initial external parameters between the surround view camera and the lidar through iteration;

[0018] 5) In order to further optimize the external parameters, the edge matching method is used to establish the constraint relationship between the panoramic image and the lidar point cloud.

[0019] In order to extract edge features in the LiDAR point cloud, the dense point cloud is converted into an intensity map and a depth map according to the intensity and elevation values ​​contained in the LiDAR point cloud. Each pixel in the depth map represents the spatial distance between the measured point (x, y, z) and the LiDAR, and the depth value

[0020] The pixels in the intensity map represent the echo intensity value of the measured object, and the calculation formula is as follows:

[0021] I=R / D 2 , where R represents the reflectivity of the object in the laser radar measurement information, D represents the depth value, and I represents the echo intensity value; using the initial external parameter R,t and the internal parameter of the surround panoramic camera, the coordinates of the point cloud projected to the pixel plane of the surround panoramic image are calculated, and the point cloud {P i , i=1,2,…n}corresponding to the depth value {D i} and intensity value {I i}After the projection processπ(·), the corresponding depth map and intensity map are obtained.

[0022] 6) extracting edge features from the panoramic image, the depth map obtained in step 5), and the intensity map, respectively, and matching edge features in the panoramic image and the lidar point cloud;

[0023] 7) According to the matching relationship between the panoramic image and the laser radar point cloud features obtained in step 6), construct objective function 2 and optimize the initial external parameters:

[0024] When constructing the second objective function, factor a is constructed according to the angle of the line characteristic normal vector. Factor a is represented by a piecewise function:

[0025]

[0026] θ is the angle between the normal vector of the line feature in the point cloud projection point and the normal vector of the line feature in the matched panoramic image. ε is the threshold of the angle between the two vectors. When the angle θ is greater than the threshold, the group of line features is considered to be mismatched. The factor a controls the objective function 2 and sets it to 0 to eliminate the impact on the optimization result. The final objective function 2 is:

[0027]

[0028] Where k is the number of line feature matches, j is the number of times a is set to 0 when a mismatch is detected, and p is the number of times a is set to 0 when a mismatch is detected. i Represents the line features in the point cloud projection, q i Represents the feature points of each line in the panoramic image, It represents the initial extrinsic parameters between the surround camera and the lidar. Finally, the initial extrinsic parameters are optimized according to objective function 2 to obtain more accurate extrinsic parameters.

[0029] Technical effects:

[0030] The present invention does not need to perform external parameter calibration on the surround-view panoramic camera and the laser radar in a specific scenario. In the outdoor environment, without relying on special calibration targets, the useful information in the natural environment is used to calibrate the spatial position relationship between the surround-view panoramic camera and the laser radar, so that the calibration is flexible. The surround-view panoramic camera and the laser radar both have the ability to perceive the scene in a 360-degree surround, and fully display the image information within a 360-degree range, so that the information acquisition of the environment is more accurate. By respectively performing feature extraction and feature point matching on the panoramic image and the laser radar point cloud data, and estimating the odometer information of the surround-view panoramic camera and the laser radar, the corresponding feature relationship between the panoramic image and the laser radar point cloud is automatically established, and the initial value of the external parameter is calculated based on the hand-eye calibration method.

[0031] Then, the edge features in the panoramic image and lidar point cloud are used to convert the dense point cloud into an intensity map and a depth map according to the intensity and elevation values ​​contained in the lidar point cloud. The panoramic image, depth map, and intensity map are further optimized and solved using edge matching. Since line features are prone to mismatching when line features intersect and have strong textures, which affects the optimization of external parameters, the factor a is constructed according to the angle of the line feature normal vector when constructing the objective function two to avoid the influence of mismatching on the initial external parameter optimization results and further improve the accuracy of the external parameters.

[0032] The present invention realizes the fully automatic external parameter calibration of the surround-view panoramic camera and the laser radar without manual intervention, and further optimizes the external parameters by using the line feature matching relationship between the panoramic image and the point cloud information to achieve a higher-precision external parameter solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.

[0035] In step 2), the ORB feature extraction algorithm can be used to obtain key points and descriptors between image frames in the panoramic image sequence, and then the feature points are matched to obtain the matching relationship between feature points between frames; the lidar point cloud data is denoised and classified, and edge points and plane points are extracted according to the classification results, and these feature points are matched between frames;

[0036] In step 6), the canny operator can be used to extract edge features from the panoramic image, the depth map obtained in step 5), and the intensity map, respectively, and the KD tree algorithm can be used to match the edge features in the panoramic image and the lidar point cloud.

Claims

1. A fully automatic calibration method for surround view cameras and lidar based on natural environment, characterized in that: The steps include: 1) Calibrate the intrinsic parameters of the surround-view panoramic camera: Using the panoramic camera spherical model, u = π(·)P describes the mapping relationship between the spatial point P of the surround-view panoramic camera and the corresponding pixel point u, where π(·) represents the generalized projection process. By calibrating the intrinsic parameters of the surround-view panoramic camera, the specific parameters of the projection process are obtained, and the mapping relationship between the spatial point P and the corresponding pixel point u is accurately established; 2) Perform feature extraction and feature matching on panoramic images and lidar point cloud data respectively; 3) Estimate the motion of the surround view camera based on the internal parameters of the surround view camera and the matching relationship between the feature points of the panoramic image frames, and estimate the motion of the lidar using the feature matching relationship of the lidar point cloud data; 4) The surround view camera and the LiDAR are rigidly connected. During the movement, the relative posture remains unchanged. Through posture estimation, the initial external parameters between the surround view camera and the LiDAR are calculated using the hand-eye calibration method: The constraint between the motion pose matrix and the external parameter matrix of the surround view panoramic camera and the laser radar is AX=XB. Matrix A is the motion pose transformation matrix of the surround view panoramic camera, matrix B is the motion pose transformation matrix calculated by the laser radar odometer, X is the external parameter matrix between the surround view panoramic camera and the laser radar, R, t is used to represent the rotation and translation relationship between the surround view panoramic camera and the laser radar, and the motion pose of the laser radar is represented by T L =[R L ,t L ], the motion pose of the panoramic camera is represented by T C =[R C ,t C ], according to the hand-eye calibration model, the following relationship is satisfied: R L R=RR C , R L t+t L =Rt C +t, The pose information calculated by the surround panoramic camera and lidar is translated into C and t L There is a difference of size s between them, which can be expressed using the formula: R L t+t L =Rt C ·s+t, Using the above constraints, we construct the objective function Optimize the external parameters R, t and s, and obtain the initial external parameters between the surround view camera and the lidar through iteration; 5) Extract edge features from the LiDAR point cloud. According to the intensity and elevation values ​​contained in the LiDAR point cloud, convert the dense point cloud into an intensity map and a depth map. Each pixel in the depth map represents the spatial distance between the measured point (x, y, z) and the LiDAR. The pixels in the intensity map represent the echo intensity value of the measured object, and the calculation formula is as follows: I=R / D 2 , where R represents the reflectivity of the object in the laser radar measurement information, D represents the depth value, and I represents the echo intensity value; using the initial external parameter R,t and the internal parameter of the surround panoramic camera, the coordinates of the point cloud projected to the pixel plane of the surround panoramic image are calculated, and the point cloud {P i , i=1,2,…n}corresponding to the depth value {D i } and intensity value {I i }After the projection process π(·), the corresponding depth map and intensity map are obtained; 6) extracting edge features from the panoramic image, the depth map obtained in step 5), and the intensity map, respectively, and matching edge features in the panoramic image and the lidar point cloud; 7) According to the matching relationship between the panoramic image and the laser radar point cloud features obtained in step 6), construct objective function 2 and optimize the initial external parameters: When constructing the second objective function, factor a is constructed according to the angle of the line characteristic normal vector. Factor a is represented by a piecewise function: θ is the angle between the normal vector of the line feature in the point cloud projection point and the normal vector of the line feature in the matched panoramic image. ε is the threshold of the angle between the two vectors. When the angle θ is greater than the threshold, the group of line features is considered to be mismatched. The factor a controls the objective function 2 and sets it to 0 to eliminate the impact on the optimization result. The final objective function 2 is: Where k is the number of line feature matches, j is the number of times a is set to 0 when a mismatch is detected, and p is the number of times a is set to 0 when a mismatch is detected. i Represents the line features in the point cloud projection, q i Represents the feature points of each line in the panoramic image, It represents the initial extrinsic parameters between the surround camera and the lidar. Finally, the initial extrinsic parameters are optimized according to objective function 2 to obtain more accurate extrinsic parameters.

2. The method for fully automatic calibration of surround view camera and laser radar based on natural environment according to claim 1 is characterized in that: In step 2), the ORB feature extraction algorithm is used to obtain the key points and descriptors between image frames in the panoramic image sequence, and then the feature points are matched to obtain the matching relationship between the feature points between frames; the lidar point cloud data is denoised and classified, and the edge points and plane points are extracted according to the classification results, and these feature points are matched between frames.

3. The method for fully automatic calibration of surround view camera and laser radar based on natural environment according to claim 1 is characterized in that: In step 6), the canny operator is used to extract edge features from the panoramic image, the depth map obtained in step 5), and the intensity map, respectively, and the KD tree algorithm is used to match the edge features in the panoramic image and the lidar point cloud.

Images