Camera and solid-state laser radar online calibration method for mobile robot

Abstract

The application discloses a kind of camera and solid-state laser radar online calibration method of mobile robot, first by solid-state laser radar and camera acquisition point cloud and image data, using specific function optimization to subpixel level accuracy, then transform into least square problem and obtain offline external parameter solution.When the speed of mobile robot is lower than threshold value, radar and camera collect current scene point cloud and image original data, project point cloud on camera imaging plane by offline external parameter, generate point cloud reflectivity map with point cloud reflectivity as pixel, convert camera original image to gray scale image, extract the outline pixel of reflectivity map and gray scale image, construct error equation according to the distance of paired points and convert to least square problem, finally solve and obtain online external parameter update.Not only can the camera and solid-state laser radar of mobile robot be calibrated offline external parameter, but also can update external parameter calibration online during robot movement, so as to ensure the stability of subsequent operation of mobile robot.

Description

TECHNICAL FIELD

[0001] The application relates to a sensor joint calibration method, in particular to an online calibration method for a camera and a solid-state laser radar of a mobile robot. BACKGROUND

[0002] In the fields of automatic driving, robots, AR / VR, as commonly used sensors, laser radars and cameras have their own unique advantages, but they also have their own shortcomings and deficiencies. More and more schemes begin to adopt multi-camera and multi-laser radar configurations to achieve the purpose of multi-sensor fusion. Multi-sensor calibration is a prerequisite for the normal operation of these systems, but the current multi-modal sensor calibration scheme is still very troublesome and requires a lot of manual intervention. The most common one is the calibration problem between the camera and the radar. Therefore, calibrating the external parameters of the camera and the laser radar and obtaining the coordinate conversion relationship of the two sensors are crucial for the fusion of the camera imaging data and the point cloud data of the laser radar.

[0003] The current calibration process of the camera and the laser radar is to calibrate the external parameters before the operation of the environment perception system, and the external parameter values are no longer adjusted during the operation of the system. There are many mature calibration tools for cameras and laser radars at present, such as the automatic driving software Autoware, the calibration tool kit lidar_camera_calibration, the Apollo system of Baidu, etc. However, most of these open-source calibration tools are for traditional mechanical laser radars and do not support solid-state laser radars, which makes the above-mentioned existing calibration tools unable to be directly used for the calibration of solid-state laser radars. In addition, once the external parameters are calibrated by the existing offline calibration method, the external parameters are difficult to update during the operation of the system, and a specific calibration board and an open and interference-free calibration environment are required to recalibrate. However, when the autonomous mobile robot moves in a complex scene, factors such as bumps and collisions will change the external parameters of the camera and the laser radar. Due to the long-distance detection characteristics of the laser radar, a small angle deviation in the external parameters of the camera and the laser radar will cause a large matching error in a long-distance scene. Therefore, how to provide a method for offline external parameter calibration of the camera and the solid-state laser radar of a mobile robot, and online update of the external parameter calibration during the movement of the robot to ensure the stability of the subsequent operation of the mobile robot is one of the research directions in the industry. SUMMARY

[0004] In view of the problems existing in the prior art, the application provides an online calibration method for a camera and a solid-state laser radar of a mobile robot, which can calibrate the offline external parameters of the camera and the solid-state laser radar of the mobile robot, and can update the online external parameter calibration during the movement of the robot, thereby ensuring the stability of the subsequent operation of the mobile robot.

[0005] In order to achieve the above object, the technical scheme adopted by the present application is: a camera and solid-state laser radar online calibration method for a mobile robot, and the specific steps are:

[0006] (1) using the camera factory calibration of the internal parameter coefficient, or by Zhang Zhengyou calibration method and Matlab toolbox calibration camera internal parameter coefficient;

[0007] (2) in offline calibration, solid-state laser radar and camera collect environmental data respectively, and convert the point cloud collected by solid-state laser radar into pcd format, and extract the four corner point coordinates of the calibration board in the point cloud pcd and camera image respectively and optimize the accuracy;

[0008] (3) through the initial external parameter, the point cloud corner point coordinates in step (3) are converted to the camera imaging plane through the initial external parameter and the internal parameter coefficient in step (1);

[0009] (4) the distance error equation between the matching points is constructed, and the least square problem is converted to solve the optimal offline external parameter of the camera and the laser radar;

[0010] (5) the mobile robot starts normal movement, and when the speed of the mobile robot is less than the set speed threshold, it enters the online calibration state, and the solid-state laser radar and the camera collect the original point cloud and image data of the current environment respectively; the point cloud is projected onto the imaging plane of the camera through the offline initial external parameter, and the reflectivity value of the point cloud is used as the pixel to generate the reflectivity graph, and the original image is converted into a gray scale graph;

[0011] (6) the reflectivity graph and the gray scale graph obtained in step (5) are subjected to contour extraction;

[0012] (7) for each contour pixel in the reflectivity graph, the nearest neighbor pixel coordinates in the gray scale contour graph are searched, and the pixel pairs with the Euclidean distance less than the set threshold are matched;

[0013] (8) after all the contours are matched, the error equation can be constructed according to the distance error of the matching point pairs and the optimal online external parameter coefficient is solved;

[0014] (9) the online external parameter coefficient obtained in step (8) is used to update and calibrate the offline external parameter coefficient obtained in step (4).

[0015] Further, the specific process of step (1) is:

[0016] Let O-xyz be the camera coordinate system, and let the point P coordinate in the world coordinate system be (X, Y, Z) T , the point P' coordinate on the physical imaging plane is (X', Y') T , then according to the similar triangle, we have:

[0017]

[0018] Where f represents the camera focal length, because the imaging is inverted, a negative sign is generated, in order to eliminate the negative sign, the camera image is symmetrical to the front of the camera, then it can be written as:

[0019]

[0020] The arrangement is obtained:

[0021]

[0022]

[0023] At the same time, the pixel coordinate system o'-uv is defined on the physical imaging plane, that is, the coordinate origin O' of the coordinate system O'-x'y' is translated to the upper left corner o' of the image, the distance is c x on the x-axis, the distance is c y on the y-axis, the u-axis is scaled by a times compared to the x' axis, and the v-axis is scaled by β times compared to the y' axis, and the coordinates of the imaging point P' in the pixel coordinate system are (u, v) T Then it is obtained:

[0024] u=αX'+c x

[0025] v=βY'+c y

[0026] The arrangement is obtained:

[0027]

[0028]

[0029] Use f x to represent αf, use f y to represent βf, and the formula is arranged into the form of homogeneous coordinates as follows:

[0030]

[0031] Where K is the camera intrinsic coefficient.

[0032] Further, the specific process of the step (2) is: converting the point cloud data into a pcd file and manually picking up the coordinates of the four corner points of the calibration board, at the same time, manually picking up the coordinates of the four corner points of the calibration board from the camera image data, and based on the sub-pixel corner detection function cornerSubPix() in OpenCV, further optimizing the picked corner point coordinates to obtain sub-pixel level precision corner point coordinates.

[0033] Further, the specific process of the step (3) is:

[0034] The laser radar and the camera acquire the coordinates of the point P on the calibration board at the same time, and the coordinates of the point P in the camera pixel coordinate system are P c , i.e. (u, v, 1) T , the coordinates of the point P in the laser radar coordinate system are P l , i.e. (X l , Y l , Z l ) T , the coordinates of the point P in the camera coordinate system are P', i.e. (X', Y', Z') T , the coordinates of the point P in the camera pixel coordinate system are P ltc , i.e. (u', v', 1) T , and the following equation is obtained:

[0035]

[0036] The above equation is combined with the intrinsic matrix in step (1) and written in the form of homogeneous coordinates, and the following equation is obtained:

[0037]

[0038] where K is the camera intrinsic coefficient, T l c is a homogeneous transformation matrix.

[0039] Further, the specific process of the step (6) is that the contour corresponds to a series of points, first, the Canny edge detector is used to detect the pixels of the picture contour boundary, and then the edge pixels are grouped into a contour based on the findContours() contour finding function in OpenCV.

[0040] Further, the specific process of the step (7) is that:

[0041] The K-D tree is used to establish data association for the pixel coordinates in the grayscale image, and the variances of the data in the u and v directions are calculated respectively The calculation process is as follows:

[0042]

[0043]

[0044] where respectively represent the average value of the data, n represents the number of points around the pixel coordinates, and (u i , v i) is the coordinate of the point, the median of the data with large variance is selected as the root node of the K-D tree to establish the tree, then the nearest neighbor pixel coordinate of each contour pixel in the reflectivity map is searched in the gray contour map, the method of searching the nearest neighbor uses the Euclidean distance; at the same time, the distance threshold is set to 7, the nearest neighbor point distance within the threshold is considered as a matching point, and the distance greater than the distance threshold is recorded as a non-matching point and does not participate in matching.

[0045] Further, the specific process of the step (8) is as follows:

[0046] According to the distance error of the matching point pair, an error equation is constructed as follows:

[0047]

[0048] Wherein e represents the distance between the matching point pair calculated according to the extrinsic parameter coefficient obtained in the iteration process, T l c is a homogeneous transformation matrix, n represents the number of points around the pixel coordinate, m represents the number of pixel coordinates, represents the coordinate in the camera pixel coordinate system, represents the coordinate converted from the laser point cloud to the camera pixel coordinate system according to the extrinsic parameter result. Then it is converted into a least square problem and solved based on the iterative optimization of the Ceres library, that is, the latest optimal extrinsic parameter coefficient can be obtained.

[0049] Compared with the prior art, the present application first records point cloud and image data by using ROS tool rosbag of solid-state laser radar and camera, converts the point cloud to pcd, and manually picks up the corner point coordinates of the calibration board, the picture taken by the camera is also processed, the corner point coordinates are optimized to sub-pixel level accuracy based on the cornerSubPix() function in OpenCV, and finally the 3D-2D data correlation is constructed, and the off-line external parameter is obtained by solving the least square problem iteratively. When the moving robot moves at a speed lower than the threshold, the online calibration is entered, the radar and the camera collect the current scene point cloud and image original data, the point cloud is projected onto the camera imaging plane through the off-line external parameter, the point cloud reflectivity map is generated by taking the point cloud reflectivity as the pixel, the original image of the camera is converted into a gray image, the contour pixels of the reflectivity map and the gray image are extracted, the nearest neighbor pixel coordinates of each contour pixel in the reflectivity map in the gray contour image are found, a distance threshold for participating in matching is set, an error equation is constructed according to the distance of the paired points and is converted into a least square problem, and finally the online external parameter is obtained by solving iteratively based on the Ceres library, and the off-line external parameter of the sensor is updated and calibrated in real time according to the online calibration result. Compared with the traditional off-line calibration, the present application realizes automatic updating without target, greatly reduces the error caused by manual participation in the traditional calibration process, realizes the online calibration of the two sensors in the scene without specific target, and therefore, based on the online calibration method of the present application, the mobile robot can more conveniently realize the data fusion between multiple cameras and multiple laser radars in the working process, ensures the stability and robustness of the data fusion of the multi-sensor system in the long-term working process, and provides a technical basis for the slam and perception technology of multiple sensors in the field of automatic driving, robots, AR / VR. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the overall flowchart of the online calibration of the camera and the solid-state laser radar in the present application;

[0051] Figure 2 is the matching result effect diagram in the present application without calibration;

[0052] Figure 3 is the offline calibration effect diagram in the present application;

[0053] Figure 4 is the point cloud reflectivity map and the gray image in the present application;

[0054] Figure 5 is the reflectivity contour map and the gray contour map in the present application;

[0055] Figure 6 is the online calibration effect diagram in the present application. DETAILED DESCRIPTION

[0056] The present application will be further described below.

[0057] As Figure 1 shown, the specific steps of the present application are:

[0058] (1) Use the camera factory factory calibration of the intrinsic coefficient, or by Zhang Zhengyou calibration method and Matlab toolbox calibration camera intrinsic coefficient, the specific process is:

[0059] Let O-xyz be the camera coordinate system, let the point P coordinate in the world coordinate system be (X, Y, Z) T , the point P' coordinate on the physical imaging plane is (X', Y') T , then according to similar triangles:

[0060]

[0061] Where f represents the focal length of the camera, because the imaging is inverted, a negative sign is generated, in order to eliminate the negative sign, the image formed by the camera is symmetrical to the front of the camera, then it can be written as:

[0062]

[0063] After sorting out:

[0064]

[0065]

[0066] At the same time, define the pixel coordinate system o'-uv on the physical imaging plane, that is, move the coordinate origin O' of the coordinate system O'-x'y' to the left corner of the image o', the distance on the x axis is c x , the distance on the y axis is c y , the u axis is scaled by α times compared with the x' axis, and the v axis is scaled by β times compared with the y' axis, let the coordinate of the imaging point P' in the pixel coordinate system be (u, v) T , then:

[0067] u = αX' + c x

[0068] v = βY' + c y

[0069] After sorting out:

[0070]

[0071]

[0072] Use f x to represent αf, and f y to represent βf, and the formula is sorted into the form of homogeneous coordinates as follows:

[0073]

[0074] Where K is the camera intrinsic parameter coefficient.

[0075] (2) During offline calibration, the solid-state lidar and camera collect environmental data respectively, and convert the point cloud collected by the solid-state lidar into PCD format. Extract the coordinates of the four corner points of the calibration board from the point cloud PCD and the camera image respectively and optimize its accuracy. The specific process is as follows: convert the point cloud data into a PCD file and manually pick the coordinates of the four corner points of the calibration board. At the same time, manually pick the coordinates of the four corner points of the calibration board from the camera image data. Based on the sub-pixel corner detection function cornerSubPix() in OpenCV, further optimize the corner coordinates to obtain corner coordinates with sub-pixel accuracy.

[0076] (3) Using the initial extrinsic parameters, the coordinates of the point cloud corner points in step (3) are transformed to the camera imaging plane using the initial extrinsic parameters and the intrinsic parameter coefficients in step (1), such as... Figure 2 As shown, the specific process is as follows:

[0077] The lidar and camera simultaneously acquire the coordinates of point P on the calibration board. Let the coordinates of point P in the camera pixel coordinate system be P_i. c That is, (u, v, 1) T In the lidar coordinate system, the coordinates are P l That is (X) l Y l Z l ) T The coordinates, when transformed to the camera coordinate system, are P′, i.e., (X′, Y′, Z′). T The coordinates P in the camera pixel coordinate system are obtained by converting the extrinsic and intrinsic parameters of the LiDAR and camera. ltc That is, (u′, v′, 1) T Then we have:

[0078]

[0079] Combining the above equation with the intrinsic parameter matrix in step (1) and rewriting it in homogeneous coordinate form, we have:

[0080]

[0081] Where T l c Let K be the homogeneous transformation matrix, and K be the camera intrinsic parameter coefficients.

[0082] (4) The distance error equation between paired points is constructed as follows:

[0083]

[0084] where e represents the distance between the matched points calculated from the extrinsic coefficients obtained in the iteration process, T l c is the homogeneous transformation matrix, n represents the number of pixel coordinates around the point, represents the coordinates in the camera pixel coordinate system, Z represents the point cloud depth value, and K represents the camera intrinsic coefficient, represents the extrinsic coefficient, represents the coordinates in the solid-state laser radar coordinate system, represents the coordinates of the point cloud in the radar coordinate system converted to the camera coordinate system by the extrinsic coefficient. Then it is converted into a least squares problem and solved based on the Ceres library iterative optimization, that is, the latest optimal extrinsic coefficient can be obtained, as shown in Figure 3 ;

[0085] (5) The mobile robot starts normal movement and driving, and enters the online calibration state when the speed of the mobile robot is less than the set speed threshold. The solid-state laser radar and the camera collect the original point cloud and image data of the current environment, respectively. The point cloud is projected onto the imaging plane of the camera through the offline initial extrinsic, and the reflectivity value of the point cloud is used as the pixel to generate a reflectivity map, which converts the original image into a grayscale image, as shown in Figure 4 ;

[0086] (6) The reflectivity map and the grayscale map obtained in step (5) are subjected to contour extraction, as shown in Figure 5 , and the specific process is as follows: the contour corresponds to a series of points. First, the Canny edge detector is used to detect the pixels of the picture contour boundary, and then the edge pixels are grouped into contours based on the findContours() contour finding function in OpenCV.

[0087] (7) The nearest neighbor pixel coordinates of each contour pixel in the reflectivity map are searched in the grayscale contour map. The pixel points with a Euclidean distance less than a set threshold are matched, and the specific process is as follows:

[0088] The K-D tree is used to establish data association for the pixel coordinates in the grayscale map, and the variances of the data in the u and v directions are calculated, and the calculation process is as follows:

[0089]

[0090]

[0091] where respectively represent the average value of the data, n represents the number of pixel coordinates around the point, and (u i , v i) is the coordinate of the point. The median of the data with large variance is selected as the root node of the K-D tree to build the tree, and then the nearest neighbor pixel coordinate in the gray contour map is searched for each contour pixel in the reflectivity map. The nearest neighbor is calculated using the Euclidean distance. At the same time, the distance threshold is set to 7, and the nearest neighbor point within the threshold is considered as a matching point, and the distance greater than the distance threshold is recorded as a non-matching point and does not participate in matching.

[0092] (8) After all the contours are matched, the distance error of the matching point pair can be used to construct an error equation and solve the optimal online extrinsic parameter coefficient. The specific process is as follows:

[0093] The error equation constructed according to the distance error of the matching point pair is as follows:

[0094]

[0095] Where e represents the distance between the matching points calculated according to the extrinsic parameter coefficient obtained in the iteration process, T l c is the homogeneous transformation matrix, n represents the number of points around the pixel coordinate, m represents the number of pixel coordinates, represents the coordinate in the camera pixel coordinate system, represents the coordinate converted from the laser point cloud to the camera pixel coordinate system according to the extrinsic parameter result. Then it is converted into a least squares problem and solved based on the Ceres library iterative optimization, that is, the latest optimal extrinsic parameter coefficient can be obtained.

[0096] (9) The offline extrinsic parameter coefficient obtained in step (4) is updated and calibrated in real time with the online extrinsic parameter coefficient obtained in step (8), as shown in Figure 6 .

[0097] The above method can be automatically implemented by using a calibration system, which includes an offline calibration part and an online calibration part. The offline calibration part includes a data acquisition module, a data labeling module and an offline optimization module. The data acquisition module is used to acquire point cloud image data of the environment. The data labeling module is used to manually pick up the corner point coordinates of the calibration board and optimize them to sub-pixel level accuracy. The offline optimization module is used to iteratively optimize the distance between the matching points to solve the extrinsic parameter. The online calibration part includes a data preprocessing module, a feature extraction module, an online optimization module and a calibration output module. The data preprocessing module is used to convert the original point cloud and image data into reflectivity map and gray map. The feature extraction module is used to extract the contour pixels of the reflectivity map and the gray map respectively. The online optimization module is used to iteratively optimize the online extrinsic parameter by matching the pixel coordinates. The calibration output module is used to update the offline calibration result in real time according to the online calibration result.

[0098] The above merely describes the preferred embodiments of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for online calibration of a camera and a solid-state lidar for a mobile robot, characterized in that, The specific steps are as follows: (1) Use the intrinsic parameter coefficients that are set by the manufacturer when the camera is manufactured, or calibrate the camera intrinsic parameter coefficients using Zhang Zhengyou calibration method and Matlab toolbox; (2) During offline calibration, the solid-state lidar and camera collect environmental data respectively, and convert the point cloud collected by the solid-state lidar into PCD format. Extract the coordinates of the four corner points of the calibration board from the point cloud PCD and camera images respectively and optimize their accuracy. (3) Transform the coordinates of the corner points of the point cloud in step (2) to the camera imaging plane using the initial extrinsic parameters and the intrinsic parameter coefficients in step (1); (4) Construct the distance error equation between the paired points and convert it into a least squares problem to solve for the optimal offline extrinsic coefficients of the camera and the lidar; (5) The mobile robot starts to move normally. When the speed of the mobile robot is less than the set speed threshold, it enters the online calibration state. The solid-state lidar and camera collect the original point cloud and image data of the current environment respectively. The original point cloud is projected onto the imaging plane of the camera through the optimal offline extrinsic coefficients. The reflectivity value of the original point cloud is used as the pixel to generate a reflectivity map and the original image is converted into a grayscale image. (6) Extract the contour from the reflectance map and grayscale map obtained in step (5); (7) For each contour pixel in the reflectance map, find the coordinates of the nearest neighbor pixel in the grayscale contour map, and match the pixel pairs whose Euclidean distance is less than the set threshold. (8) After all contours have been matched, an error equation is constructed based on the distance error of the matching point pairs and the optimal online extrinsic coefficients are solved. (9) Update and calibrate the offline extrinsic coefficients obtained in step (4) using the optimal online extrinsic coefficients obtained in step (8).

2. The camera and solid-state lidar online calibration method of the mobile robot according to claim 1, wherein, The specific process of step (1) is as follows: Let O-xyz be the camera coordinate system, and let the coordinates of point P in the world coordinate system be (X, Y, Z) T , the coordinates of point P on the physical imaging plane be (X ′ , Y ′ ) ′ T According to similar triangles, we have:​ Where f represents the camera's focal length, and because the image is inverted, a negative sign is generated. To eliminate the negative sign, the image formed by the camera is symmetrically positioned in front of the camera, which can then be written as: The results were: At the same time, define a pixel coordinate system o ′ -uv, i.e. coordinate system O ′ -x ′ y ′ Translate the coordinate origin O ′ to the upper left corner of the image o ′ , translate the distance c x on the x-axis, and translate the distance c y on the y-axis, scale the u-axis by α times compared to the x ′ -axis, and scale the v-axis by β times compared to the y ′ -axis, and let the imaging point P ′ have coordinates (u, v) in the pixel coordinate system T , then it is obtained that: u = aX ′ + c x v = βY ′ + c y The conclusion is as follows: with f x denotes αf, with f y denotes βf, the equation is arranged in homogeneous coordinates as follows: Where K is the camera intrinsic parameter coefficient.

3. The camera and solid-state lidar online calibration method of the mobile robot according to claim 1, wherein, The specific process of step (2) is as follows: convert the point cloud data into a pcd file and manually pick the coordinates of the four corner points of the calibration board. At the same time, manually pick the coordinates of the four corner points of the calibration board from the camera image data. Then, based on the sub-pixel corner detection function cornerSubPix() in OpenCV, further optimize the corner coordinates to obtain corner coordinates with sub-pixel accuracy.

4. The camera and solid-state lidar online calibration method of the mobile robot according to claim 2, wherein, The specific process of step (3) is as follows: The solid-state laser radar and the camera acquire the coordinates of point P on the calibration board at the same time, and the coordinates of point P in the camera pixel coordinate system are P c , that is, (u, v, 1) T , the coordinates of point P in the solid-state laser radar coordinate system are P l , that is, (X l , Y l , Z l ) T , the coordinates of point P converted to the camera coordinate system are P ′ , that is, (X ′ , Y ′ , Z ′ ) T , through the extrinsic parameters of the solid-state laser radar and the camera and the intrinsic parameter coefficients of the camera, the rotation is R, the translation is t, and the coordinates of point P converted to the camera pixel coordinate system are P ltc , that is, (u ′ , v ′ , 1) T , and the following equation is obtained: Combining the above equation with the intrinsic parameter matrix in step (1) and rewriting it in homogeneous coordinate form, we have: wherein is a homogeneous transformation matrix, K is the intrinsic camera parameter.

5. The camera and solid-state lidar online calibration method of the mobile robot according to claim 1, wherein, The specific process of step (6) is as follows: the contour corresponds to a series of points. First, the Canny edge detector is used to detect the pixels of the image contour boundary. Then, based on the findContours() contour lookup function in OpenCV, these edge pixels are combined into a contour.

6. The camera and solid-state lidar online calibration method of the mobile robot according to claim 1, wherein, The specific process of step (7) is as follows: The data correlation of pixel coordinates in the gray image is established using the K-D tree, and the variances of data in the u and v directions are calculated respectively The calculation process is as follows: wherein respectively represent the average value of the group of data, n represents the number of points around the pixel coordinate, (u i ,v i ) is the point coordinate, the median of the data with large variance is selected as the root node of the K-D tree to establish the tree, then the nearest neighbor pixel coordinate in the gray profile image is searched for each contour pixel in the reflectivity image, the method for finding the nearest neighbor uses the Euclidean distance; at the same time, the distance threshold is set to 7, the nearest neighbor point distance within the threshold is considered as a matching point, and greater than the distance threshold is recorded as a non-matching point and does not participate in matching.

7. The method of claim 1, wherein, The specific process of step (8) is as follows: The error equation is constructed based on the distance error of the matched point pairs as follows: where e represents the distance between the pair points calculated according to the extrinsic parameter coefficients obtained in the iteration process, is a homogeneous transformation matrix, n represents the number of pixel coordinate surrounding points, and m represents the number of pixel coordinates, represents the coordinates in the camera pixel coordinate system, represents the coordinates of the laser point cloud converted to the camera pixel coordinate system according to the extrinsic parameter results; then converted into a least square problem and solved based on the iterative optimization of the Ceres library, that is, the latest optimal extrinsic parameter coefficients are obtained.

Images