A vertical feature-based lidar and infrared camera extrinsic parameter optimization method

Abstract

The application belongs to the field of multi-sensor fusion, and proposes a laser radar and infrared camera external parameter optimization method based on vertical features. The indoor environment information scanned by the laser radar and the infrared camera is analyzed at the same time, the vertical features in the two kinds of data are extracted and matched, the residual equation and gradient descent are introduced to evaluate and update the matching results, and finally the external parameter calibration results with high precision and strong robustness are obtained, which provides accurate relative pose relationship for the subsequent joint use of the two.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion and relates to a method for optimizing the extrinsic parameters of a lidar and infrared camera based on vertical features. Background Technology

[0002] LiDAR and infrared cameras are widely used in autonomous vehicles and robotics in many scenarios, such as autonomous navigation, 2D or 3D mapping, object classification, and obstacle avoidance. LiDAR can provide precise 3D geometric information but lacks sufficient environmental feature information. Conversely, infrared cameras can provide rich environmental representations such as thermal radiation values ​​but struggle to provide accurate distance information. Therefore, to provide more comprehensive and accurate perception information and make the system more reliable in complex and changing environments, fusing these two sensors is essential. Obtaining accurate external parameters between them through a calibration process is a prerequisite for efficient fusion of the two sensors.

[0003] Most current automatic calibration methods rely on specially designed calibration objects, such as checkerboard patterns or specific image patterns. These objects detect, extract, and match feature points from images and point clouds, transforming the original problem into a least-squares equation, which is then solved. However, targeted calibration methods have inherent drawbacks. For example, when these calibration targets are placed near the sensor suite, external errors may be amplified in long-distance scenes. Furthermore, it is difficult to guarantee the availability of suitable calibration equipment every time calibration is needed, resulting in relatively low feasibility and flexibility. Currently, many automatic targetless calibration methods have emerged, primarily achieving extrinsic parameter calibration through multi-sensor feature extraction of environmental characteristic information. These methods significantly reduce calibration conditions while ensuring high matching accuracy. For instance, the paper (J. Kümmerle and T. Kühner, “Unified Intrinsic and Extrinsic Camera and LiDAR Calibration under Uncertainties,” in 2020 IEEE International Conference on Robotics and Automation, May 2020) proposes a method for extrinsic parameter calibration of LiDAR and camera in uncertain environments based on edge features of LiDAR and continuous depth features of color camera. This method performs well when the initial extrinsic parameters are relatively accurate. However, when the initial extrinsic parameter error is large, the concentrated point cloud edge features extracted using depth continuity often exhibit drift that is difficult to correct during matching. Currently, there is no suitable target-free extrinsic parameter calibration method for LiDAR and infrared cameras. Therefore, there is an urgent need for a target-free extrinsic parameter calibration method for LiDAR and infrared cameras that can still fit accurate extrinsic parameter results using sufficient feature information even when the initial extrinsic parameter deviation is large, and has stronger robustness. Summary of the Invention

[0004] To overcome the extrinsic parameter optimization problems existing in current technologies for LiDAR and infrared cameras, this invention proposes a targetless calibration method based on vertical features. This method extracts vertical line features from the scanning results of radar and infrared camera sensors on the same scene, then performs feature matching based on the current extrinsic parameters, and formulates a residual function to evaluate the rationality of the extrinsic parameters until the value of this function is within the allowable error range.

[0005] The specific solution of this invention is a method for optimizing the extrinsic parameters of a lidar and infrared camera based on vertical features, comprising the following steps:

[0006] Step 1: Collect the same environmental information using an infrared camera and a lidar, and extract features from each separately;

[0007] The infrared camera acquires environmental information to obtain infrared images. The LSD line detection algorithm (Rafael Grompone von Gioi, a Line Segment Detector, 2012) is used to obtain line feature information in the grayscale image of the infrared image. The least squares method is used to fit the line feature information into multiple line equations. Based on the parameters of these equations, the equations of vertical lines perpendicular to the xy-plane are selected, yielding the vertical line feature information P corresponding to these equations. i C ;

[0008] The lidar acquires environmental information to obtain multiple frames of raw point cloud data, and then performs the following operations:

[0009] The original point cloud of multiple frames is downsampled to reduce the amount of data that needs to be processed in subsequent calculations, while also reducing the interference of outliers and noise on the results; feature regions are obtained by Euclidean clustering, and then the feature regions are fitted into a multi-frame planar point cloud using the RANSAC method.

[0010] Three consecutive planar point cloud frames, f1, f2, and f3, are selected from the multi-frame planar point cloud and stitched together to achieve temporal alignment of the three consecutive planar point cloud frames and improve point cloud density. This process requires calculating the transformation matrix from the last two frames to f1 in these three point cloud frames. The specific steps are as follows:

[0011] First, use formula (1) to find the midpoint coordinates q of the first frame planar point cloud f1. i ;

[0012]

[0013] Where x i Let f1 be the coordinates of the i-th point in f1, and f1 has a total of n points.

[0014] Establish the evaluation function as shown in equation (2);

[0015]

[0016] Where M is the transformation matrix, x' i Let f2 be the coordinates of the i-th point in the second frame planar point cloud f1 to be matched with f1; given an initial value of M, transform f2 to f1 according to M, calculate the evaluation function, and continuously update the value of the transformation matrix M using Newton's method until the value of the evaluation function is less than a given threshold, thus obtaining the transformation matrix M1, and transforming f2 to the f1 coordinate system to align the two frame point clouds in time;

[0017] Then, also based on equation (2), x' i Replace with x″ i ,x″ iThe coordinates of the i-th point in the third frame planar point cloud f3 to be matched with f1 are given. The transformation matrix M2 is obtained using the same steps, and f3 is transformed into the f1 coordinate system to align the two frame point clouds in time, thus completing the time alignment of the three frame point cloud information.

[0018] The depth value of each pixel in each of the three frame point clouds is calculated, and then the time-aligned three frame point clouds are converted into an image. The three channels of each pixel in the image store the depth value of the corresponding pixel in the three frame point clouds. In the image, the distance between the current pixel and its neighboring pixels is compared, and outliers with a distance greater than a threshold are removed, thus extracting line feature information.

[0019] The eigenvectors of the line feature information are obtained using Principal Component Analysis (PCA). Eigenvectors with an angle less than a given threshold to the positive z-axis are selected, and the corresponding vertical feature point cloud P is found. i L ;

[0020] Step 2: Optimize extrinsic parameters based on feature points and initial extrinsic parameters;

[0021] Feature regions are selected from the environmental information collected in step 1. Based on the infrared image pixel coordinates corresponding to the vertical line feature information within the feature regions and the laser point coordinates corresponding to the vertical feature point cloud, the initial external parameter matrix is ​​obtained using the direct linear transformation method. The specific steps are as follows:

[0022] The pixel coordinates [uv] of the infrared image are known. T and the coordinates of the laser point [X w Y w Z w ] T We establish a linear transformation equation containing external parameters, as shown in equation (3):

[0023]

[0024] Among them, z c The parameters used to convert the pixel coordinates of an infrared image to normalized coordinates, where K is a known intrinsic parameter of the infrared camera. This is the initial extrinsic parameter matrix [R|t]. 4*3 Where R and t are the rotation and translation components in the linear transformation, respectively, and P W =[X w Y w Z w 1] T Solving equation (3) yields the constraint conditions shown in equation (4);

[0025]

[0026] Where t1, t2, and t3 are the initial extrinsic parameter matrices. The three row vectors;

[0027] The least-squares solution of equation (4) obtained by using the SVD method is the initial extrinsic parameter matrix. The value of .

[0028] Based on the external parameter matrix, the position of the laser point in the infrared camera coordinate system is obtained using rigid body transformation. Then, the point at the above position is projected onto the infrared image plane using a pinhole model to obtain the position of the laser point on the infrared image. The following relationship is established:

[0029]

[0030] In the formula, q l To fit the vertical feature point cloud into a point on a straight line using a kd-tree, For point q l The transpose of the direction vector, e L To extract the noise error of the laser point, e C To extract noise errors from infrared images, P i L The vertical feature point cloud obtained in step 1, The true value of the external parameter matrix;

[0031]

[0032] in,

[0033]

[0034]

[0035]

[0036] In the formula r i , e i These represent the deviation between the laser point and the corresponding feature point in the infrared image, the descent gradient of the residual equation, and the noise interference term, respectively. δT is the residual amount, and p = f(πP). i C );

[0037] The solution for the residual δT is obtained by equation (6), and the solution is iteratively solved using gradient descent until δT is less than the threshold. Then the optimized extrinsic parameter matrix is ​​obtained.

[0038] The beneficial effects of this invention are that the method requires infrared grayscale images, point cloud information and initial external parameters at the same time as input, and uses vertical line features to automatically calculate the transformation matrix between these sensors. Compared with commonly used edge features, vertical line features have strong applicability to both indoor and outdoor scenes. They are relatively dispersed in most indoor and outdoor scenes, and can still fit accurate external parameter results with sufficient feature information when the initial external parameter deviation is large, thus having stronger robustness. Attached Figure Description

[0039] Figure 1 Point cloud data information.

[0040] Figure 2 Infrared camera image.

[0041] Figure 3 Initial extrinsic parameter matching results.

[0042] Figure 4 Optimize the extrinsic parameter matching results. Detailed Implementation

[0043] The technical solution of the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0044] A method for optimizing extrinsic parameters of a lidar and infrared camera based on vertical features includes the following steps:

[0045] In this specific implementation, a 128-line 3D laser sensor is used to generate 3D point clouds at a frequency of 10Hz. The horizontal scanning range is 120° (-60° to +60°), and the vertical scanning range is 25°. The angular resolution for both horizontal and vertical directions is 0.2°. A TE-EV1 infrared camera with a resolution of 640*480 pixels is used. Data is acquired simultaneously using two sensors, and point cloud data and image information from the same frame are extracted from a ROS bag using existing functions. The acquired infrared images are shown below. Figure 1 As shown.

[0046] Step 1: Collect the same environmental information using an infrared camera and a lidar, and extract features from each separately;

[0047] The infrared camera acquires environmental information to obtain infrared images. The LSD line detection algorithm (Rafael Grompone von Gioi, a Line Segment Detector, 2012) is used to obtain line feature information in the grayscale image of the infrared image. The least squares method is used to fit the line feature information into multiple line equations. Based on the parameters of these equations, the equations of vertical lines perpendicular to the xy-plane are selected, yielding the vertical line feature information P corresponding to these equations.i C ;

[0048] The lidar acquires environmental information to obtain multiple frames of raw point cloud data. Vertical line features are extracted using the continuity of distance, and then the following operations are performed:

[0049] The original point cloud from multiple frames is subjected to voxel filtering, and the point cloud is downsampled at a grid size of 5mm. The downsampled global point cloud is shown below. Figure 2 As shown. This reduces the amount of data that needs to be processed in subsequent calculations, while also reducing the interference of outliers and noise on the results; feature regions are obtained using Euclidean clustering, and then the RANSAC method is used to fit the feature regions into a multi-frame planar point cloud;

[0050] Three consecutive planar point cloud frames, f1, f2, and f3, are selected from the multi-frame planar point cloud and stitched together to achieve temporal alignment of the three consecutive planar point cloud frames and improve point cloud density. This process requires calculating the transformation matrix from the last two frames to f1 in these three point cloud frames. The specific steps are as follows:

[0051] First, use formula (1) to find the midpoint coordinates q of the first frame planar point cloud f1. i ;

[0052]

[0053] Where x i Let f1 be the coordinates of the i-th point in f1, and f1 has a total of n points.

[0054] Establish the evaluation function as shown in equation (2);

[0055]

[0056] Where M is the transformation matrix, x' i Let f2 be the coordinates of the i-th point in the second frame planar point cloud f1 to be matched with f1. Given an initial value of M, transform f2 to f1 according to M, calculate the evaluation function, and continuously update the value of the transformation matrix M using Newton's method until the value of the evaluation function is less than the given threshold 0.2, thus obtaining the transformation matrix M1. Transform f2 to the f1 coordinate system to align the two frame point clouds in time.

[0057] Then, also based on equation (2), x' i Replace with x″ i ,x″ i The coordinates of the i-th point in the third frame planar point cloud f3 to be matched with f1 are given. The transformation matrix M2 is obtained using the same steps, and f3 is transformed into the f1 coordinate system to align the two frame point clouds in time, thus completing the time alignment of the three frame point cloud information.

[0058] The depth value of each pixel in each of the three frame point clouds is calculated, and then the time-aligned three frame point clouds are converted into an image. The three channels of each pixel in the image store the depth value of the corresponding pixel in the three frame point clouds. In the image, the distance between the current pixel and its neighboring pixels is compared, and outliers with a distance greater than a threshold are removed, thus extracting line feature information.

[0059] The eigenvectors of the line feature information are obtained using Principal Component Analysis (PCA). Eigenvectors with an angle less than a given threshold to the positive z-axis are selected, and the corresponding vertical feature point cloud P is found. i L .

[0060] Step 2: Optimize extrinsic parameters based on feature points and initial extrinsic parameters;

[0061] Feature regions are selected from the environmental information collected in step 1. Based on the infrared image pixel coordinates corresponding to the vertical line feature information within the feature regions and the laser point coordinates corresponding to the vertical feature point cloud, the initial extrinsic parameter matrix is ​​obtained using the Direct Linear Transform (DLT) method. The specific steps are as follows:

[0062] The pixel coordinates [uv] of the infrared image are known. T and the coordinates of the laser point [X w Y w Z w ] T We establish a linear transformation equation containing external parameters, as shown in equation (3):

[0063]

[0064] Among them, z c The parameters used to convert the pixel coordinates of an infrared image to normalized coordinates, where K is a known intrinsic parameter of the infrared camera. This is the initial extrinsic parameter matrix [R|t]. 4*3 Where R and t are the rotation and translation components in the linear transformation, respectively, and P W =[X w Y w Z w 1] T Solving equation (3) yields the constraint conditions shown in equation (4);

[0065]

[0066] Where t1, t2, and t3 are the initial extrinsic parameter matrices, respectively. The three row vectors have a total of twelve dimensions, while the number of matching point pairs is much greater than six. Therefore, the least squares solution of equation (4) can be obtained using the SVD method, which is the initial extrinsic parameter matrix. The values ​​of the initial external parameter matrix are obtained as follows:

[0067]

[0068] The laser point cloud was projected onto the infrared image using this initial extrinsic parameter, and the result is as follows: Figure 3 As shown.

[0069] Based on the external parameter matrix, the position of the laser point in the infrared camera coordinate system is obtained using rigid body transformation. Then, the point at the above position is projected onto the infrared image plane using a pinhole model to obtain the position of the laser point on the infrared image. The following relationship is established:

[0070]

[0071] In the formula, q l To fit the vertical feature point cloud into a point on a straight line using a kd-tree, For point q l The transpose of the direction vector, e L To extract the noise error of the laser point, e C To extract noise errors from infrared images, P i L The vertical feature point cloud obtained in step 1, The true value of the external parameter matrix;

[0072]

[0073] in,

[0074]

[0075]

[0076]

[0077] In the formula r i , e i These represent the deviation between the laser point and the corresponding feature point in the infrared image, the descent gradient of the residual equation, and the noise interference term, respectively. δT is the residual amount, and p = f(πP). i C );

[0078] The solution for the residual δT is obtained by equation (6), and then iteratively solved using gradient descent until δT is less than the threshold. This yields the optimized extrinsic parameter matrix. The optimized extrinsic parameter matrix values ​​are as follows:

[0079]

[0080] The laser point cloud is projected onto the infrared image using the optimized extrinsic matrix, with the following effect: Figure 4 As shown. Compare the matching results with the initial extrinsic parameter matrix. Figure 3 It can be clearly seen that the effect of projection using optimized extrinsic parameters is that the depth information of the laser can be more accurately matched with the infrared image, and this effect is particularly prominent on some edges with discontinuous depth.

[0081] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for optimizing extrinsic parameters of a lidar and infrared camera based on vertical features, characterized in that, The steps include the following: Step 1: Collect the same environmental information using an infrared camera and a lidar, and extract features from each separately; The infrared camera acquires environmental information to obtain infrared images. The LSD line detection algorithm is used to obtain line feature information in the grayscale image of the infrared image. The least squares method is used to fit the line feature information into multiple line equations. Based on the parameters of these equations, the equations of vertical lines perpendicular to the xy-plane are selected, resulting in the vertical line feature information P corresponding to the vertical line equations. i C ; The lidar acquires environmental information to obtain multiple frames of raw point cloud data, and then performs the following operations: The original point cloud of multiple frames is downsampled, and feature regions are obtained by Euclidean clustering. Then, the feature regions are fitted into a multi-frame planar point cloud using the RANSAC method. Three consecutive planar point clouds are selected from the multi-frame planar point clouds and stitched together to achieve temporal alignment of the three consecutive planar point clouds. The depth value of each pixel in each of the three point clouds is calculated, and then the time-aligned three point clouds are converted into an image. The three channels of each pixel in the image store the depth value of the corresponding pixel in the three point clouds. In the image, the distance between the current pixel and its neighboring pixels is compared, and outliers with a distance greater than a threshold are removed, thus extracting line feature information. The eigenvectors of the line feature information are obtained using Principal Component Analysis (PCA). Eigenvectors with an angle less than a given threshold to the positive z-axis are selected, and the corresponding vertical feature point cloud P is found. i L ; Step 2: Optimize extrinsic parameters based on feature points and initial extrinsic parameters; Feature regions are selected from the environmental information collected in step 1. Based on the infrared image pixel coordinates corresponding to the vertical line feature information within the feature regions and the laser point coordinates corresponding to the vertical feature point cloud, the initial external parameter matrix is ​​obtained using the direct linear transformation method. Based on the external parameter matrix, the position of the laser point in the infrared camera coordinate system is obtained using rigid body transformation. Then, the point at the above position is projected onto the infrared image plane using a pinhole model to obtain the position of the laser point on the infrared image. The following relationship is established: In the formula, q l To fit the vertical feature point cloud into a point on a straight line using a kd-tree, For point q l The transpose of the direction vector, e L To extract the noise error of the laser point, e C To extract noise errors from infrared images, P i L The vertical feature point cloud obtained in step 1, The true value of the external parameter matrix; in, In the formula r i , e i These represent the deviation between the laser point and the corresponding feature point in the infrared image, the descent gradient of the residual equation, and the noise interference term, respectively. δT is the residual amount, and p = f(πP). i C ); The solution for the residual δT is obtained by equation (6), and then iteratively solved using gradient descent until δT is less than the threshold. This yields the optimized extrinsic parameter matrix.

2. The method for optimizing extrinsic parameters of a lidar and infrared camera based on vertical features according to claim 1, characterized in that, In step 1, the specific steps for achieving temporal alignment of the three consecutive planar point clouds are as follows: First, use formula (1) to find the midpoint coordinates q of the first frame planar point cloud f1. i ; Where x i Let f1 be the coordinates of the i-th point in f1, and f1 has a total of n points. Establish the evaluation function as shown in equation (2); Where M is the transformation matrix, x' i Let f2 be the coordinates of the i-th point in the second frame planar point cloud f1 to be matched with f1; given an initial value of M, transform f2 to f1 according to M, calculate the evaluation function, and continuously update the value of the transformation matrix M using Newton's method until the value of the evaluation function is less than a given threshold, thus obtaining the transformation matrix M1, and transforming f2 to the f1 coordinate system to align the two frame point clouds in time; Then, also based on equation (2), x' i Replace with x” i ,x” i The coordinates of the i-th point in the third frame planar point cloud f3 to be matched with f1 are given. The transformation matrix M2 is obtained using the same steps, and f3 is transformed into the f1 coordinate system to align the two frame point clouds in time, thus completing the time alignment of the three frame point cloud information.

3. The method for optimizing extrinsic parameters of a lidar and infrared camera based on vertical features according to claim 1, characterized in that, In step 2, the initial external parameter matrix The steps to obtain it are as follows: The pixel coordinates [uv] of the infrared image are known. T and the coordinates of the laser point [X w Y w Z w ] T We establish a linear transformation equation containing external parameters, as shown in equation (3): Among them, z c The parameters used to convert the pixel coordinates of an infrared image to normalized coordinates, where K is a known intrinsic parameter of the infrared camera. This is the initial extrinsic parameter matrix [R|t]. 4*3 Where R and t are the rotation and translation components in the linear transformation, respectively, and P W =[X w Y w Z w 1] T Solving equation (3) yields the constraint conditions shown in equation (4); Where t1, t2, and t3 are the initial extrinsic parameter matrices, respectively. The three row vectors; The least-squares solution of equation (4) obtained by using the SVD method is the initial extrinsic parameter matrix. The value of .

4. The method for optimizing extrinsic parameters of a lidar and infrared camera based on vertical features according to claim 1, characterized in that, The downsampling described in step 1 is specifically performed using voxel filtering.

Images