A method for joint calibration of infrared camera and laser radar

Abstract

The present invention relates to a method for joint calibration of an infrared camera and a laser radar. First, a corner detection algorithm is used to extract pixel coordinates on an infrared image, and then the least square method is used to solve the internal parameters to obtain the position of the calibration plate in the camera coordinate system. Then, the plane points of the calibration plate are extracted by single-point interaction to obtain the position of the calibration plate in the laser coordinate system. Finally, the constraint condition that the plane distance difference between the two is zero is used to perform iterative optimization and solve to obtain the external parameters between the two, and finally complete the joint calibration of the infrared camera and the laser radar. The present invention separates the internal parameter calibration of the infrared camera and the external parameter calibration of the laser radar. Although the steps become complicated, the accuracy is significantly improved, and the entire calibration process is simple and easy to operate, and has strong practicality.

Description

Technical Field

[0001] The invention belongs to the field of mobile measurement, and in particular relates to a method for joint calibration of an infrared camera and a laser radar. Background Art

[0002] With the development of technologies such as robots, people are gradually using robots for autonomous inspections in some high-risk and fixed scenarios, such as chemical plants and substations. They can efficiently and quickly detect and eliminate potential dangers, and can also reduce the possibility of accidents for workers to a certain extent, indirectly ensuring the safety of personnel and property. LiDAR is widely used in the field of robots, mainly for robot positioning and environmental perception. Although LiDAR can stably and quickly obtain the spatial position of surrounding objects, it lacks the ability to perceive other attributes of surrounding objects, such as color and temperature. Robots currently used for inspections in high-risk scenarios are mainly responsible for detecting high temperatures, toxic gas leaks, etc. It is impossible to complete such inspection tasks relying solely on a LiDAR, and other sensors must be carried to assist in completing the inspection tasks. In these high-risk scenarios, fire prevention is the top priority, and inspection robots must be equipped with instruments that sense temperature, such as infrared cameras, which can work at night and in harsh climates, safely and reliably. The combined use of LiDAR and infrared cameras for data fusion can not only obtain the temperature information of objects, but also quickly obtain the spatial position of objects, providing support for workers to quickly eliminate dangers. In order to improve the accuracy of locating temperature anomalies, it is necessary to jointly calibrate the lidar and infrared camera to obtain the intrinsic parameters of the infrared camera and the extrinsic parameters between the two. Therefore, studying the calibration method of infrared cameras and lidar is of great practical significance.

[0003] Due to the uniqueness of infrared camera imaging, the calibration method of traditional cameras cannot be used. The existing joint calibration of infrared cameras and lidars mostly uses manual registration methods, which are cumbersome to operate and have low accuracy. Since it is difficult for infrared cameras to obtain precise surface feature points similar to the calibration plate of visible light systems, in order to simplify the calibration process and improve the calibration accuracy, a specific infrared calibration plate is required. The calibration plate can heat some areas to form high radiation areas, and the rest are normal radiation areas. The areas are all squares of a certain length. The high radiation area and the normal radiation area are arranged alternately, and the intersection of the two will form a distinct intersection in the infrared image. By extracting the pixel coordinates of the intersection, the internal parameters of the infrared camera can be accurately calculated. The internal parameters include focal length, principal point and distortion coefficient. The joint calibration of infrared cameras and lidars is achieved by constructing constraints from the intersection of the infrared calibration plate to the laser plane, and using nonlinear optimization methods to perform multiple sets of iterative calculations of the external parameters between the two, and finally finding the optimal solution. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method for joint calibration of an infrared camera and a laser radar. First, the pixel coordinates on the infrared image are extracted using a corner detection algorithm, and then the intrinsic parameters are solved using the least squares method to obtain the position of the calibration plate in the camera coordinate system. Then, the plane points are extracted interactively by single points to obtain the position of the calibration plate in the laser coordinate system. Finally, the constraint condition that the plane distance difference between the two is zero is used to perform iterative optimization and solve to obtain the external parameters between the two, and finally complete the joint calibration of the infrared camera and the laser radar.

[0005] In order to achieve the above object, the technical solution provided by the present invention is a method for joint calibration of an infrared camera and a laser radar, comprising the following steps:

[0006] Step 1, collect calibration data;

[0007] Step 2, performing corner point detection on the infrared image;

[0008] Step 3: According to the corner points detected in step 2, the camera intrinsic parameters are calculated using the least squares algorithm to determine the position of the calibration plate in the camera coordinate system;

[0009] Step 4: Corner point backprojection to verify the accuracy of the intrinsic parameters;

[0010] Step 5, single-point interactive extraction of calibration plate plane points to determine the position of the calibration plate in the laser coordinate system;

[0011] Step 6: Use the intrinsic parameters calibrated in step 3 and the plane points of the calibration plate manually selected in step 5 to construct point-to-plane constraints, and iteratively optimize and solve to obtain the extrinsic parameters between the infrared camera and the lidar.

[0012] Moreover, the data acquisition in step 1 includes two groups, one group uses an infrared camera to shoot the calibration plate to obtain an infrared image of the calibration plate for camera intrinsic calibration; the other group simultaneously collects the infrared image and laser data of the calibration plate for external parameter calibration between the infrared camera and the lidar.

[0013] Moreover, the corner point detection in step 2 is mainly to extract the pixel coordinates of the intersection of the high radiation area and the normal radiation area in the calibration plate. The function findChessboardCornersSB in OpenCV4 can be used for corner point detection, which can directly obtain the corner points with sub-pixel accuracy.

[0014] Moreover, in step 3, the pixel coordinate system o-uv is constructed with the upper left corner of the image as the origin, and the two vertical sides connected to the origin as the u-axis and the v-axis; if the photography center is S, the intersection p of the photography direction and the image plane is the image principal point of the image, and the image coordinate system p-xy is constructed with the x-axis parallel to the u-axis of the pixel coordinate system and the y-axis parallel to the v-axis of the pixel coordinate system; the camera coordinate system S-XYZ is the origin of the photography center S, the main optical axis Sp is the z-axis, and the x and y axes are parallel to the x and y axes of the image coordinate system respectively; the world coordinate system W-XYZ is a right-handed coordinate system with the upper left corner point W of the calibration plate as the coordinate origin, the Z axis is perpendicular to the calibration plate plane, the horizontal direction is the Y axis, and the vertical direction is the X axis. Assume that the coordinates of a point M in space in W-XYZ are (X, Y, Z), and the straight line through points M and S intersects the image plane at point m. If the coordinates of S in W-XYZ are (X s ,Y s ,Z s ), let SP = 1, then the coordinates of m in S-XYZ are (x, y, 1), rotate W-XYZ around the Y axis, X axis and Z axis respectively by (β, α, γ) angles in the same direction as S-XYZ, then:

[0015]

[0016]

[0017] In the formula, (r 11 ,r 12 ,r 13 ,r 21 ,r 22 ,r 23 ,r 31 ,r 32 ,r 33 ) constitutes the rotation matrix R:

[0018]

[0019] The calculation formula for each element in the rotation matrix is:

[0020]

[0021] Optical lenses cannot completely simulate the pinhole imaging process, which will cause distortion. The main distortions are radial distortion and tangential distortion. The ideal imaging point m will shift to point m' along the radial and tangential directions between it and the principal point p. Its distortion can be modeled by five distortion coefficients (k1, k2, k3, p1, p2). The coordinates of m' are (x', y'), then:

[0022] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+p1(r2 +2x 2 )+2p2xy (5)

[0023] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+p2(r 2 +2y 2 )+2p1xy (6)

[0024] r 2 =x 2 +y 2 (7)

[0025] Where r is the radial length.

[0026] Finally, (x′, y′) is transformed to pixel coordinates (u, v) on the image plane through affine transformation:

[0027] u=f x ×x′+c x (8)

[0028] v=f y ×y′+c y (9)

[0029] In the formula, f x is the principal distance in the x direction, f y is the principal distance in the y direction, (c x ,c y ) is the pixel coordinate of the principal point.

[0030] Finally, the functional relationship of pixel coordinates (u, v) is obtained:

[0031] u=u(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2,X,Y,Z) (10)

[0032] v=v(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2,X,Y,Z) (11)

[0033] The coordinates of n spatial points (Xi ,Y i ,Z i ) and its corresponding image point coordinates (u i ,v i ), i = 1, 2, ..., n, and estimate the parameters (X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2), thus completing the infrared camera internal reference (f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2) calibration.

[0034] Moreover, the accuracy of the intrinsic parameter parameters verified by back-projection of the corner points in step 4 is to project the three-dimensional space points into two-dimensional plane pixel points using the parameters calculated in step 3, and then make a difference with the pixel coordinates extracted from the corner points. If the difference is within 0.5 pixels, it can be considered that the accuracy of the intrinsic parameter meets the requirement, otherwise the corner points are re-selected to calculate the internal parameter.

[0035] The process of projecting a three-dimensional space point into a two-dimensional plane pixel point is expressed as:

[0036]

[0037] In the formula, (X w ,Y w ,Z w ,1) represents a three-dimensional space point, (u,v) represents a two-dimensional plane pixel point, R and t can be obtained by internal calibration parameters.

[0038] Moreover, in step 5, you need to manually select the calibration plate plane in the laser data in the software, determine the approximate position of the calibration plate in the point cloud, and then click the center of the plane. The plane selection algorithm will select plane points according to a certain radius. Finally, the RANSAC algorithm is used to fit the plane points to determine the position of the calibration plate in the laser coordinate system.

[0039] In the laser coordinate system, the plane equation of the calibration plate is expressed as:

[0040] Ax+By+Cz+D=0 (13)

[0041] Where (x, y, z) is the coordinate of the laser point on the calibration plate plane.

[0042] The RANSAC algorithm can be used to solve the parameters A, B, C, and D, and obtain the calibration plate plane in the laser coordinate system.

[0043] Moreover, in step 6, the external parameters are set to (α′, β′, γ′, x, y, z), and the initial values ​​are all 0, where (α′, β′, γ′) are the Euler angles of rotation along the X-axis, Y-axis, and Z-axis respectively, and (x, y, z) are the translations along the X-axis, Y-axis, and Z-axis respectively. When the calibration plate plane in the laser coordinate system is transformed to the calibration plate plane in the camera coordinate system, the point product of the plane point and the normal vector of the calibration plate plane is zero. This constraint condition is used to construct the following constraint equation:

[0044]

[0045] Where P i is the plane position of the i-th chessboard grid in the camera coordinate system, n i is the plane normal vector position of the i-th chessboard in the camera coordinate system, Q ij is the position of the jth point of the ith chessboard in the laser coordinate system, R(α′,β′,γ′) represents the Euler angle converted into a rotation matrix, and T(x,y,z) represents the translation vector.

[0046] Using the least squares method to solve, after dozens of sets of data iterative optimization, the external parameters (α′, β′, γ′, x, y, z) will converge to an ideal value, thus completing the joint calibration of the infrared camera and lidar.

[0047] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0048] (1) Improved calibration accuracy

[0049] The present invention separates the infrared camera internal parameter calibration from the laser radar external parameter calibration. Although the steps become more complicated, the accuracy is significantly improved. The separate calibration is mainly based on the different calibration data requirements of the two: the internal parameter calibration requires the calibration plate to cover the entire frame as much as possible, so that the camera internal parameters can be calculated more accurately; the external parameter calibration requires the calibration plate to be placed in different positions and postures to provide sufficient constraints, so that the calculated external parameters are more accurate.

[0050] (2) Simple calibration process

[0051] Infrared camera calibration mainly includes two steps: corner point extraction and internal parameter calculation. Under the premise of ensuring the integrity of the calibration plate in the image, there is basically no need for manual selection and adjustment, and all operations are fool-proof. Joint calibration takes into account the existence of certain noise in laser radar data. In order to improve the calibration accuracy, manual plane selection is adopted. Although there is manual participation, there is also a plane fitting algorithm to help users extract the calibration plate plane quickly and accurately. After the selection is completed, the external parameters can be calculated. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a technical flow chart of an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of collecting calibration plate data by an infrared camera according to an embodiment of the present invention.

[0054] Figure 3 Schematic diagram of the mathematical model of camera imaging according to an embodiment of the present invention.

[0055] Figure 4 Schematic diagram of lens distortion according to an embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram of a manually selected calibration plate plan according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention provides a method for joint calibration of an infrared camera and a laser radar. Firstly, a corner detection algorithm is used to extract pixel coordinates on an infrared image. Then, the least square method is used to solve the intrinsic parameters to obtain the position of a calibration plate in a camera coordinate system. Then, plane points are extracted by single-point interaction to obtain the position of the calibration plate in a laser coordinate system. Finally, an iterative optimization solution is performed using the constraint condition that the plane distance difference between the two is zero to obtain the extrinsic parameters between the two, and finally the joint calibration of the infrared camera and the laser radar is completed.

[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0059] like Figure 1 As shown, the process of the embodiment of the present invention includes the following steps:

[0060] Step 1: Collect calibration data.

[0061] Data collection includes two groups. One group uses an infrared camera to shoot the calibration plate to obtain the infrared image of the calibration plate, such as Figure 2 As shown, it is used for camera intrinsic calibration; the other group simultaneously collects infrared images and laser data of the calibration plate, which is used for external parameter calibration between the infrared camera and the lidar.

[0062] Step 2: perform corner detection on the infrared image.

[0063] Corner detection mainly extracts the pixel coordinates of the intersection of the high radiation area and the normal radiation area in the calibration plate. The function findChessboardCornersSB in OpenCV4 can be used for corner detection. This function can directly obtain corners with sub-pixel accuracy.

[0064] Step 3: According to the corner points detected in step 2, the camera intrinsic parameters are calculated using the least squares algorithm to determine the position of the calibration plate in the camera coordinate system.

[0065] The imaging mathematical model of the camera is as follows: Figure 3 As shown: the pixel coordinate system o-uv is constructed with the upper left corner of the image as the origin and the two vertical sides connected to the origin as the u-axis and v-axis; if the photography center is S, the intersection point p of the photography direction and the image plane is the image principal point, and the image coordinate system p-xy is constructed with the x-axis parallel to the u-axis of the pixel coordinate system and the y-axis parallel to the v-axis of the pixel coordinate system; the camera coordinate system S-XYZ is the origin of the photography center S, the principal optical axis Sp is the z-axis, and the x and y axes are parallel to the x and y axes of the image coordinate system respectively; the world coordinate system W-XYZ is a right-handed coordinate system with the upper left corner point W of the calibration plate as the origin, the Z axis perpendicular to the calibration plate plane, the horizontal direction is the Y axis, and the vertical direction is the X axis. Assume that the coordinates of a point M in space in W-XYZ are (X, Y, Z), and the straight line through points M and S intersects the image plane at point m. If the coordinates of S in W-XYZ are (X s ,Y s ,Z s ), let SP = 1, then the coordinates of m in S-XYZ are (x, y, 1), rotate W-XYZ around the Y axis, X axis and Z axis respectively by (β, α, γ) angles in the same direction as S-XYZ, then:

[0066]

[0067]

[0068] In the formula, (r 11 ,r 12 ,r 13 ,r 21 ,r 22 ,r 23 ,r 31 ,r 32 ,r 33 ) constitutes the rotation matrix R:

[0069]

[0070] The calculation formula for each element in the rotation matrix is:

[0071]

[0072] Optical lenses cannot completely simulate the pinhole imaging process, which will cause distortion. The main distortions are radial distortion and tangential distortion, such as Figure 4 As shown in the figure, the ideal imaging point m will be offset to point m' along the radial and tangential directions between it and the principal point p. Its distortion can be modeled by five distortion coefficients (k1, k2, k3, p1, p2). The coordinates of m' are (x', y'), so:

[0073] x′=x(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2x 2 )+2p2xy (5)

[0074] y′=y(1+k1r 2 +k2r 4 +k3r 6 )+p2(r 2 +2y 2 )+2p1xy (6)

[0075] r 2 =x 2 +y 2 (7)

[0076] Where r is the radial length.

[0077] Finally, (x′, y′) is transformed to pixel coordinates (u, v) on the image plane through affine transformation:

[0078] u=f x ×x′+c x (8)

[0079] v=f y ×y′+c y (9)

[0080] In the formula, f x is the principal distance in the x direction, f y is the principal distance in the y direction, (c x ,c y ) is the pixel coordinate of the principal point.

[0081] Finally, the functional relationship of pixel coordinates (u, v) is obtained:

[0082] u=u(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y,k1,k2,k3,p1,p2,X,Y,Z) (10)

[0083] v=v(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2,X,Y,Z) (11)

[0084] The coordinates of n spatial points (X i ,Y i ,Z i ) and its corresponding image point coordinates (u i ,v i ), i = 1, 2, ..., n, and estimate the parameters (X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2), thus completing the infrared camera internal reference (f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2) calibration.

[0085] Step 4: Corner point backprojection verifies the accuracy of the intrinsic parameters.

[0086] First, use the parameters calculated in step 3 to project the 3D space point into a 2D plane pixel point, and then make a difference with the pixel coordinates extracted from the corner point. If the difference is within 0.5 pixels, it can be considered that the internal parameter accuracy meets the requirement. Otherwise, re-select the corner point to calculate the internal parameter.

[0087] The process of projecting a three-dimensional space point into a two-dimensional plane pixel point is expressed as:

[0088]

[0089] In the formula, (X w ,Y w ,Z w ,1) represents a three-dimensional space point, (u,v) represents a two-dimensional plane pixel point, R and t can be obtained by internal calibration parameters.

[0090] Step 5: Single-point interactive extraction of calibration plate plane points to determine the position of the calibration plate in the laser coordinate system.

[0091] In order to determine the position of the calibration plate in the laser coordinate system, it is necessary to extract the calibration plate plane points interactively, which can ensure the extraction accuracy and reduce the error. First, you need to manually select the calibration plate plane in the laser data in the software to determine the approximate position of the calibration plate in the point cloud, such as Figure 5 As shown, then click the center of the plane. The plane selection algorithm will select plane points according to a certain radius. Finally, the RANSAC algorithm is used to fit the plane points to determine the position of the calibration plate in the laser coordinate system.

[0092] In the laser coordinate system, the plane equation of the calibration plate is expressed as:

[0093] Ax+By+Cz+D=0 (13)

[0094] Where (x, y, z) is the coordinate of the laser point on the calibration plate plane.

[0095] The RANSAC algorithm can be used to solve the parameters A, B, C, and D, and obtain the calibration plate plane in the laser coordinate system.

[0096] Step 6: Use the intrinsic parameters calibrated in step 3 and the plane points of the calibration plate manually selected in step 5 to construct point-to-plane constraints, and iteratively optimize and solve to obtain the extrinsic parameters between the infrared camera and the lidar.

[0097] Assume that the external parameters are (α′, β′, γ′, x, y, z), and the initial values ​​are all 0, where (α′, β′, γ′) are the Euler angles of rotation along the X-axis, Y-axis, and Z-axis respectively, and (x, y, z) are the translations along the X-axis, Y-axis, and Z-axis respectively. When the calibration plate plane in the laser coordinate system is transformed to the calibration plate plane in the camera coordinate system, the point product of the plane point and the normal vector of the calibration plate plane is zero. The following constraint equation is constructed:

[0098]

[0099] Where P i is the plane position of the i-th chessboard grid in the camera coordinate system, n i is the plane normal vector position of the i-th chessboard in the camera coordinate system, Q ij is the position of the jth point of the ith chessboard in the laser coordinate system, R(α′,β′,γ′) represents the Euler angle converted into a rotation matrix, and T(x,y,z) represents the translation vector.

[0100] Using the least squares method to solve, after dozens of sets of data iterative optimization, the external parameters (α′, β′, γ′, x, y, z) will converge to an ideal value, thus completing the joint calibration of the infrared camera and lidar.

[0101] During specific implementation, the above process can be automatically operated using computer software technology.

[0102] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for joint calibration of an infrared camera and a laser radar, characterized in that: The steps include: Step 1, collect calibration data; Step 2, performing corner point detection on the infrared image; Step 3: According to the corner points detected in step 2, the camera intrinsic parameters are calculated using the least squares algorithm to determine the position of the calibration plate in the camera coordinate system; The pixel coordinate system o-uv is constructed with the upper left corner of the image as the origin and the two vertical sides connected to the origin as the u-axis and v-axis; if the photography center is S, the intersection p of the photography direction and the image plane is the image principal point of the image, and the image coordinate system p-xy is constructed with the x-axis parallel to the u-axis of the pixel coordinate system and the y-axis parallel to the v-axis of the pixel coordinate system; the camera coordinate system S-XYZ is the origin of the photography center S, the principal optical axis Sp is the z-axis, and the x and y axes are parallel to the x and y axes of the image coordinate system respectively; the world coordinate system W-XYZ is the right-hand coordinate system with the upper left corner point W of the calibration plate as the origin, the Z axis is perpendicular to the calibration plate plane, the horizontal direction is the Y axis, and the vertical direction is the X axis; assuming that the coordinates of a point M in space in W-XYZ are (X, Y, Z), and the straight line through points M and S intersects the image plane at point m, if the coordinates of S in W-XYZ are (X s ,Y s ,Z s ), let SP = 1, then the coordinates of m in S-XYZ are (x, y, 1), rotate W-XYZ around the Y axis, X axis and Z axis by angles β, α and γ respectively, in the same direction as S-XYZ, then: In the formula, r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 、r 33 Construct the rotation matrix R: The calculation formula for each element in the rotation matrix is: The ideal imaging point m will be offset to point m' along the radial and tangential directions with respect to the principal point p. Its distortion can be modeled by five distortion coefficients k1, k2, k3, p1, and p2. The coordinates of m' are (x', y'), and we have: x′=x(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2x 2 )+2p2xy (5) <h2 style=";text-align:left;direction:ltr">y′ = y(1+k1r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +k2r<h2 style=";text-align:left;direction:ltr"> 4 <h2 style=";text-align:left;direction:ltr"> +k3r<h2 style=";text-align:left;direction:ltr"> 6 <h2 style=";text-align:left;direction:ltr"> )+p2(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> )+2p1xy (6) r 2 =x 2 +y 2 (7) Where r is the radial length; Finally, (x′, y′) is transformed to pixel coordinates (u, v) on the image plane through affine transformation: u=f x ×x′+c x (8) v=f y ×y′+c y (9) In the formula, f x is the principal distance in the x direction, f y is the principal distance in the y direction, (c x ,c y ) is the pixel coordinate of the principal point; Finally, the functional relationship of pixel coordinates (u, v) is obtained: u=u(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2,X,Y,Z) (10) v=v(X S ,Y S ,Z S ,α,β,γ,f x ,f y ,c x ,c y ,k1,k2,k3,p1,p2,X,Y,Z) (11) The coordinates of n spatial points (X i ,Y i ,Z i ) and its corresponding image point coordinates (u i ,v i ), i = 1, 2, ..., n, and estimate the parameters X by iterative least squares method S , Y S , Z S , α, β, γ, f x 、f y 、c x 、c y , k1, k2, k3, p1, p2, thus completing the infrared camera internal parameter f x 、f y 、c x 、c y , calibration of k1, k2, k3, p1, p2; Step 4: Corner point backprojection to verify the accuracy of the intrinsic parameters; Step 5, single-point interactive extraction of calibration plate plane points to determine the position of the calibration plate in the laser coordinate system; Step 6: Use the intrinsic parameters calibrated in step 3 and the plane points of the calibration plate selected in step 5 to construct point-to-plane constraints, and iteratively optimize and solve to obtain the extrinsic parameters between the infrared camera and the lidar.

2. The method for joint calibration of an infrared camera and a laser radar as claimed in claim 1, characterized in that: The data collection in step 1 includes two groups. One group uses an infrared camera to shoot the calibration plate to obtain an infrared image of the calibration plate for camera intrinsic calibration; the other group simultaneously collects the infrared image and laser data of the calibration plate for external parameter calibration between the infrared camera and the lidar.

3. The method for joint calibration of an infrared camera and a laser radar as claimed in claim 1, characterized in that: Corner detection in step 2 is to extract the pixel coordinates of the intersection of the high radiation area and the normal radiation area in the calibration plate, and use the function findChessboardCornersSB in OpenCV4 to perform corner detection. This function can directly obtain corners with sub-pixel accuracy.

4. The method for joint calibration of an infrared camera and a laser radar as claimed in claim 1, characterized in that: In step 4, the accuracy of the intrinsic parameter is verified by back-projecting the corner points. The three-dimensional space points are projected into two-dimensional plane pixels using the parameters calculated in step 3, and then the difference is made with the pixel coordinates extracted from the corner points. If the difference is within λ pixels, the accuracy of the intrinsic parameter can be considered to meet the requirement. Otherwise, the corner points are re-selected to calculate the intrinsic parameter. The process of projecting a three-dimensional space point into a two-dimensional plane pixel point is expressed as: In the formula, (X w ,Y w ,Z w ,1) represents a three-dimensional space point, (u,v) represents a two-dimensional plane pixel point, R and t can be obtained by internal calibration parameters.

5. The method for joint calibration of an infrared camera and a laser radar as claimed in claim 1, characterized in that: In step 5, you need to manually select the calibration plate plane in the laser data in the software to determine the approximate position of the calibration plate in the point cloud, then click the center of the plane. The plane selection algorithm will select plane points according to a certain radius, and finally use the RANSAC algorithm to fit the plane points to determine the position of the calibration plate in the laser coordinate system. In the laser coordinate system, the plane equation of the calibration plate is expressed as: Ax+By+Cz+D=0 (13) Where (x, y, z) is the coordinate of the laser point on the calibration plate plane; The RANSAC algorithm can be used to solve the parameters A, B, C, and D, and obtain the calibration plate plane in the laser coordinate system.

6. The method for joint calibration of an infrared camera and a laser radar as claimed in claim 1, characterized in that: In step 6, the external parameters are set to α′, β′, γ′, x, y, and z, and the initial values ​​are all 0, where α′, β′, and γ′ are the Euler angles of rotation along the X-axis, Y-axis, and Z-axis respectively, and x, y, and z are the translations along the X-axis, Y-axis, and Z-axis respectively. When the calibration plate plane in the laser coordinate system is transformed to the calibration plate plane in the camera coordinate system, the point product of the plane point and the normal vector of the calibration plate plane is zero. This constraint condition is used to construct the following constraint equation: Where P i is the plane position of the i-th chessboard grid in the camera coordinate system, n i is the plane normal vector position of the i-th chessboard in the camera coordinate system, Q ij is the position of the jth point of the ith chessboard in the laser coordinate system, R(α′,β′,γ′) represents the Euler angle converted into a rotation matrix, and T(x,y,z) represents the translation vector; Using the least squares method to solve, after iterative optimization of multiple sets of data, the external parameters α′, β′, γ′, x, y, and z will converge to an ideal value, thus completing the joint calibration of the infrared camera and lidar.

Images