A Single-Line LiDAR and Camera Calibration Method Based on Point, Line, and Surface Matching

Abstract

This application belongs to the field of nuclear industry robot technology and aims to solve the problem of extrinsic parameter calibration of monocular cameras and lidar on large equipment. This application discloses a single-line lidar and camera calibration method based on point, line, and surface matching. Based on the multi-plane intersection features in the real environment, a visual target is pasted on the intersecting plane. Single-line laser point and line features, and line and surface features of the monocular camera based on the visual target are extracted. Then, the extracted laser point and line segment features are matched with spatial planes and lines to construct error equations for the distance from a spatial point to a line and the distance from a line segment to a spatial plane. Finally, the extrinsic parameters of the lidar and monocular camera are optimized and solved using a nonlinear optimization method. This application can achieve extrinsic parameter calibration of monocular cameras and lidar on large equipment, and can obtain higher accuracy and more stable calibration results.

Description

Technical Field

[0001] This application belongs to the field of nuclear industry robot technology, and in particular relates to a single-line lidar and camera calibration method based on point, line and surface matching. Background Technology

[0002] The final heat sink for nuclear power units during operation is the seawater cold source, responsible for removing core preheating and cooling of various nuclear safety equipment during shutdown. It plays a crucial role in nuclear power plant operation, and the condition of the intake directly affects the plant's safe operation and reliability. The heat generated during unit operation causes marine organisms to attach to the cold source tunnel. A robotic arm mounted on a mobile platform can autonomously clean the tunnel walls. To achieve autonomous cleaning, multi-sensor fusion for real-time positioning and sensing of the platform within the tunnel is necessary to ensure the safe operation of the mobile platform and robotic arm. Underground tunnel environments cannot receive satellite signals for positioning, and regularly shaped tunnels cannot support laser matching positioning. Most existing tunnel positioning technologies employ wireless communication modules deployed within the tunnel, using carrier ranging technology for positioning. However, pre-deploying targets is costly. This paper considers using a monocular camera and a single-line lidar to achieve positioning and sensing of equipment within the intake culvert. The first issue to consider is the calibration between the monocular camera and the single-line lidar.

[0003] Traditional methods for calibrating monocular cameras and single-line LiDAR typically involve moving a calibration board within the shared field of view of both the LiDAR and the camera. The planar information of the calibration board is obtained through visual observation, and line information is extracted by drawing the ranging information from the LiDAR on the board. The calibration of the LiDAR and monocular camera is then achieved through constraints imposed by the lines within the plane. However, on large-scale equipment, the distributed deployment of the camera and LiDAR results in a very small or nonexistent shared field of view, rendering traditional calibration methods ineffective. Summary of the Invention

[0004] The purpose of this application is to provide a single-line lidar and camera calibration method based on point, line, and surface matching, which solves the problem of extrinsic parameter calibration of monocular cameras and lidar on large equipment.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A single-line lidar and camera calibration method based on point, line, and surface matching includes:

[0007] Step 1: Extract point and line features from single-line lidar data, extract the target and its corner points in the camera's field of view, and then fit and generate plane and spatial straight lines;

[0008] Step 2: Associate the 2D laser point features with the camera space line features by distance, and select the 2D laser feature point that is closest to each spatial line and less than the threshold as the corresponding matching point; associate the 2D laser line segment features with the 3D camera space plane by calculating the distance from the line segment to the plane and the angle between the line segment direction vector and the plane normal.

[0009] Step 3: Obtain the matching sets of laser points and spatial lines, and the matching sets of laser line segments and spatial planes. Project the 2D laser points into the image space using initial extrinsic parameters, and construct error equations for the distances from spatial points to lines and from line segments to spatial planes. By minimizing these error equations, obtain the accurate extrinsic parameter transformation T between the single-line lidar and the camera. cl * .

[0010] In one embodiment, ARUCO visual targets are pasted on intersecting planes, and the ID of the visual targets, the pose of the targets in the camera coordinate system, and the coordinates of the target corners in the camera coordinate system are identified by visual methods.

[0011] In one embodiment, FALKO is used to extract point features from single-line lidar data.

[0012] In one embodiment, a laser line segment feature extraction method based on seed region growth is used to extract line features from single-line lidar data.

[0013] In one embodiment, step 1 includes:

[0014] Step 1.1: Extract ARUCO target and laser dot / line features;

[0015] Step 1.2: Generate a spatial straight line based on plane fitting using maker.

[0016] In one embodiment, the marker's pose T in the camera coordinate system cm for:

[0017]

[0018] In the formula, R cm Let t represent the rotation matrix. cm This represents the translation vector.

[0019] In one embodiment, the size of the marker is set to 2s, then the planar coordinates of its four corner points are... They are respectively:

[0020]

[0021] After obtaining the marker's posture T cmThen, the spatial coordinates of its four corner points are obtained according to the following formula:

[0022]

[0023] For a series of spatial points P c =[xyz] T The objective equation is constructed as follows for solving the space plane problem, assuming the parameters of the space plane are: π = [abcd] T Then we have:

[0024]

[0025] By solving the above equation, we obtain the spatial plane π;

[0026] Solve for the straight line obtained by the intersection of planes in 3D space. The Plück coordinates are represented as:

[0027]

[0028] In the formula, It is the direction vector of the line. It is composed of straight lines The normal vector of the plane π formed by the camera's optical center C;

[0029] spatial straight line It can be calculated from the intersection of two planes π1 and π2, as shown in the following formula:

[0030]

[0031] In the formula, It is a straight line The dualistic expression form, [·] × Represents the antisymmetric matrix of a vector.

[0032] In one embodiment, the initial extrinsic parameters of the single-line laser to the camera are:

[0033]

[0034] 2D laser feature points are:

[0035] p l =[xy 0] T

[0036] A straight line in 3D space is:

[0037]

[0038] Then the 2D point to the line in space The distance calculation steps include:

[0039] Calculate the line from the camera's optical center to space The nearest point Q is as follows:

[0040] Q = [n] × d / d T d

[0041] In the formula, [n] × It is a straight line The antisymmetric matrix of the direction vector;

[0042] After obtaining the coordinates of a point on a straight line in space, the initial extrinsic parameter matrix T between the lidar and the camera is used. cl 2D laser feature point p l Projecting onto the camera's spatial coordinate system yields a spatial point P. l ,as follows:

[0043] P l =R cl *p l +t cl

[0044] Then the 2D laser feature point to the spatial straight line distance d l for:

[0045]

[0046] In the formula, n is a straight line Direction vector.

[0047] In one embodiment, the 2D line segment l intersects the 3D spatial plane π = [n]. P d] T To perform matching, the distance d from the line segment to the plane is calculated. lp And associate it with the angle θ between the line segment direction vector and the plane normal;

[0048] For each spatial plane, select one with an angle greater than the threshold θ. th The distance is the smallest and less than the threshold. The line segments are used as matching pairs;

[0049] Calculate the angle θ between the 2D line segment l and the plane π, and denote p. s p e Representing the start and end points of line segment l respectively, we obtain the 3D coordinates P of the corresponding line segment in the image space coordinate system. s P e Then the direction vector n of the line L Represented as:

[0050]

[0051] Then the normal vector n between the line and the plane π is... P The included angle θ between them is:

[0052]

[0053] Setting points Let be the i-th point on a 2D laser line segment, then the distance d from this line segment to the plane π is... lp The calculation is as follows:

[0054]

[0055] In the formula, N represents the number of laser points contained in the 2D line segment.

[0056] In one embodiment, the extrinsic parameter transformation T cl * for:

[0057]

[0058] In the formula, p represents the i-th laser 2D feature point i to the corresponding spatial straight line The distance; Represents the j-th laser segment l j to the corresponding space plane π j Distance; Σ p With Σ l These represent the weighting factors for the laser point and line segment errors, respectively.

[0059] Compared with existing technologies, the single-line lidar and camera calibration method based on point, line, and surface matching provided in this application has the following advantages:

[0060] This application is based on the multi-plane intersection features in the real environment. Visual targets are pasted on the intersecting planes, and single-line laser point and line features and line and surface features based on visual targets in the monocular camera are extracted. Then, the extracted laser point and line segment features are matched with spatial planes and lines to construct error equations for the distance from spatial points to lines and the distance from line segments to spatial planes. Finally, the extrinsic parameters of the lidar and monocular camera are optimized and solved by nonlinear optimization method.

[0061] Furthermore, when the overlapping area between the camera and the lidar's field of view is small or non-overlapping, this application enables rapid calibration between the lidar and the monocular camera using this method.

[0062] Furthermore, this application achieves higher accuracy and more stable calibration results by matching 2D laser Falko feature points with 3D spatial lines and 2D laser line segment features with 3D spatial planes. Attached Figure Description

[0063] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the technical description will be briefly introduced below.

[0064] Figure 1 A schematic diagram of multi-plane intersection features and pasted visual targets in a real-world environment, provided for this application;

[0065] Figure 2 A flowchart of the single-line lidar and camera calibration method based on point, line, and surface matching provided in this application;

[0066] Figure 3 A schematic diagram illustrating the implementation of the single-line lidar and camera calibration method based on point, line, and surface matching provided in this application;

[0067] Figure 4 A schematic diagram of the intersecting planes provided in this application;

[0068] Figure 5 The point-line matching and line-surface matching effects provided in this application Figure 1 ;

[0069] Figure 6 The point-line matching and line-surface matching effects provided in this application Figure 2 . Detailed Implementation

[0070] The following detailed description provides further details on specific implementation methods.

[0071] like Figures 1 to 6 As shown, this application provides a single-line lidar and camera calibration method based on point, line, and surface matching, including:

[0072] Step 1: Feature extraction, including LiDAR point and line feature extraction and visual line and surface feature extraction.

[0073] ARUCO visual targets are pasted on intersecting planes. The ID of the visual targets, the pose of the targets in the camera coordinate system, and the coordinates of the target corner points in the camera coordinate system are identified by visual methods. Point features and line features in single-line lidar data are extracted by FALKO and the laser line segment feature extraction method based on seed region growth (existing technology), respectively.

[0074] Based on the coordinates of the target corner points in the camera coordinate system, a spatial plane is generated by fitting a spatial plane equation, and the Plück coordinate representation of the intersecting lines is obtained by intersecting the planes.

[0075] Furthermore, step 1 specifically includes:

[0076] Step 1.1: Extraction of ARUCO target and laser dot / line features.

[0077] ArUco code (hereinafter also referred to as marker) is a Hamming code generated using the ArUco library. Each marker has a black border for easy self-detection. The four corners of the border are the corner points of the marker, and the inside of the border is the binary code of the marker, where white is 1 and black is 0.

[0078] When the ArUco library detects a marker in an image, it can accurately identify its ID and provide the pixel coordinates of each corner point in the original order, as well as the marker's pose T in the camera coordinate system. cm ,as follows:

[0079]

[0080] In the formula, R cm Let t represent the rotation matrix. cm This represents the translation vector.

[0081] Point features and line features in single-line lidar data were extracted using FALKO and a seed region-grown laser line segment feature extraction method, respectively.

[0082] Step 1.2: Plane fitting and spatial line generation based on maker.

[0083] Assuming the marker's size is 2s, then the planar coordinates of its four corner points are... They are respectively:

[0084]

[0085] After obtaining the marker's posture T cm Then, the spatial coordinates of its four corner points can be obtained according to the following formula:

[0086]

[0087] For a series of spatial points P c =[xyz] T The objective equation is constructed as follows for solving the space plane problem, assuming the parameters of the space plane are: π = [abcd] T Then we have:

[0088]

[0089] By solving the above equation, we can obtain the spatial plane π, as shown in the following figure. Figure 4 As shown.

[0090] After obtaining the planes, we can solve for the lines formed by the intersection of the planes. In 3D space, the lines are... The Plück coordinates are represented as:

[0091]

[0092] In the formula, It is the direction vector of the line. It is composed of straight lines The normal vector of the plane π formed by the camera's optical center C.

[0093] spatial straight line It can be calculated from the intersection of two planes π1 and π2, as shown in the following formula:

[0094]

[0095] In the formula, It is a straight line The dualistic expression form, [·] × Represents the antisymmetric matrix of a vector.

[0096] Step 2: Match laser points and line segment features with spatial planes and lines, the effect of which is as follows. Figure 5 and Figure 6 As shown.

[0097] The matching of 2D laser feature points with camera spatial lines relies on distance. For each spatial line, the 2D laser feature point closest to it and less than a threshold is selected as the corresponding matching point. First, the closest point Q from the camera's optical center to the spatial line is calculated. Then, using the initial extrinsic parameter matrix between the LiDAR and the camera, the 2D laser feature points are projected onto the camera's spatial coordinate system to obtain spatial points. Finally, the distance from the 2D laser feature point to the spatial line is calculated.

[0098] The 2D laser line segment features are correlated with the 3D camera space plane by calculating the distance from the line segment to the plane and the angle between the line segment's direction vector and the plane's normal vector. First, the direction vector of the 2D line segment is calculated. Then, the angle between the line segment and the plane's normal vector is solved. Finally, based on i points on the 2D laser line segment, the distance from the line segment to the plane is calculated.

[0099] The matching of 2D laser feature points with spatial lines in the camera relies on distance correlation. For each spatial line, the closest line with a distance less than a threshold is selected. The 2D laser feature points are used as the corresponding matching points. Assume the initial extrinsic parameters from the single-line laser to the camera are... 2D laser feature point p l =[xy 0] T and 3D spatial straight lines Then the 2D point to the line in space The distance calculation steps are as follows:

[0100] First, calculate the straight line from the camera's optical center to space. The nearest point Q is as follows:

[0101] Q = [n] × d / d T d

[0102] In the formula, [n] × It is a straight line The antisymmetric matrix of the direction vector.

[0103] After obtaining the coordinates of a point on a straight line in space, the initial extrinsic parameter matrix T between the lidar and the camera is used. cl 2D laser feature point p l Projecting onto the camera space coordinate system yields a spatial point P. l ,as follows:

[0104] P l =R cl *p l +t cl

[0105] Finally, the 2D laser feature points to the spatial straight line distance d l It can be represented as:

[0106]

[0107] For a 2D line segment l and a 3D spatial plane π = [n] P d] T The matching is achieved by calculating the distance d from the line segment to the plane. lp And correlate it with the angle θ between the line segment direction vector and the plane normal. For each spatial plane, select those with an angle greater than the threshold θ. th The distance is the smallest and less than the threshold. The line segments are used as matching pairs.

[0108] First, calculate the angle θ between the 2D line segment l and the plane π, and denote p. s p e Representing the start and end points of line segment l respectively, we can obtain the 3D coordinates P of the corresponding line segment in the image space coordinate system. s P e Then the direction vector n of the line L It can be represented as:

[0109]

[0110] Obtain a straight line in space Direction vector n L Then, the normal vector n between the line and the plane πP The included angle θ between them is:

[0111]

[0112] Setting points Let be the i-th point on a 2D laser line segment, then the distance d from this line segment to the plane π is... lp The following can be calculated:

[0113]

[0114] In the formula, N represents the number of laser points contained in the 2D line segment.

[0115] Step 3: Optimize the solution.

[0116] Through the above steps, the matching set of laser points and spatial lines can be obtained:

[0117]

[0118] And, the set of matching laser line segments with the spatial plane:

[0119] Q l ={(l 1 ,π 1 )…(l n ,π n )}

[0120] By projecting 2D laser points into the image space using initial extrinsic parameters, error equations are constructed for the distance from a spatial point to a line and the distance from a line segment to a spatial plane.

[0121] After multiple observations, the error equation can be accumulated. By minimizing this error equation, the accurate extrinsic parameter transformation T between the single-line lidar and the camera can be obtained. cl * .

[0122]

[0123] In the formula, p represents the i-th laser 2D feature point i to the corresponding spatial straight line The distance; Represents the j-th laser segment l j to the corresponding space plane π j Distance; Σ p With Σ l These represent the weighting factors for the laser point and line segment errors, respectively.

[0124] Furthermore, to suppress outliers, the Huber kernel function is introduced as the weight function ω(·). The Huber kernel function is as follows:

[0125]

[0126] As shown in the formula, when the absolute value of the error exceeds a certain threshold δ, the function's growth changes from a squared form to a linear form, suppressing the rapid divergence of the overall error caused by mismatches. Furthermore, the Huber kernel function is smooth and differentiable at arbitrary values, facilitating its optimization. The LM method integrated into the Ceres optimization library is used for the solution.

[0127] The above description is only a specific embodiment of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A single-line lidar and camera calibration method based on point, line, and surface matching, characterized in that, include: Step 1: Extract point and line features from single-line lidar data, extract the target and its corner points in the camera's field of view, and then fit and generate plane and spatial straight lines, where the spatial straight lines are generated by the intersection of planes; Step 2: Associate the 2D laser point features with the camera space line features by distance, and select the 2D laser feature point that is closest to each spatial line and less than the threshold as the corresponding matching point; associate the 2D laser line segment features with the 3D camera space plane by calculating the distance from the line segment to the plane and the angle between the line segment direction vector and the plane normal. Step 3: Obtain the matching sets of laser points and camera spatial lines, and the matching sets of laser line segments and camera spatial planes. Project the 2D laser points into the image space using initial extrinsic parameters, and construct error equations for the distances from spatial points to lines and from line segments to spatial planes. By minimizing these error equations, obtain the extrinsic parameter transformation T between the single-line lidar and the camera. cl * .

2. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 1, characterized in that, In step 1, ARUCO visual targets are pasted on the intersecting planes, and the ID of the visual targets, the pose of the targets in the camera coordinate system, and the coordinates of the target corners in the camera coordinate system are identified by visual methods.

3. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 2, characterized in that, FALKO was used to extract point features from single-line lidar data.

4. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 2, characterized in that, A laser line segment feature extraction method based on seed region growth is used to extract line features from single-line lidar data.

5. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 1, characterized in that, Step 1 includes: Step 1.1: Extract ARUCO target and laser dot / line features; Step 1.2: Based on maker's plane fitting, generate spatial straight lines by plane intersection.

6. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 5, characterized in that, In step 1.1, the marker's pose T in the camera coordinate system cm for: In the formula, R cm Let t represent the rotation matrix. cm This represents the translation vector.

7. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 5, characterized in that, In step 1.2, the size of the marker is set to 2s, then the planar coordinates of its four corner points are... They are respectively: After obtaining the marker's posture T cm Then, the spatial coordinates of its four corner points are obtained according to the following formula: For a series of spatial points P c =[xyz] T The objective equation is constructed as follows for solving the space plane problem, assuming the parameters of the space plane are: π = [abcd] T Then we have: By solving the above equation, we obtain the spatial plane π; Solve for the straight line obtained by the intersection of planes in 3D space. The Plück coordinates are represented as: In the formula, It is the direction vector of the line. It is composed of straight lines The normal vector of the plane π formed by the camera's optical center C; spatial straight line It can be calculated from the intersection of two planes π1 and π2, as shown in the following formula: In the formula, It is a straight line The dualistic expression form, [·] × Represents the antisymmetric matrix of a vector.

8. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 1, characterized in that, In step 2, the initial extrinsic parameters of the single-line laser to the camera are: 2D laser feature points are: p l =[x y 0] T A straight line in 3D space is: Then the 2D laser feature point to the spatial straight line The distance calculation steps include: Calculate the line from the camera's optical center to space The nearest point Q is as follows: Q=[n] × d / d T d In the formula, [n] × It is a straight line The antisymmetric matrix of the direction vector; After obtaining the coordinates of a point on a straight line in space, the initial extrinsic parameter matrix T between the lidar and the camera is used. cl 2D laser feature point p l Projecting onto the camera's spatial coordinate system yields a spatial point P. l ,as follows: P l =R cl *p l +t cl Then the 2D laser feature point to the spatial straight line distance d l for: In the formula, n is a straight line Direction vector.

9. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 1, characterized in that, In step 2, the 2D line segment l intersects the 3D spatial plane π = [n] P d] T To perform matching, the distance d from the line segment to the plane is calculated. lp And associate it with the angle θ between the line segment direction vector and the plane normal; For each spatial plane, select one with an angle greater than the threshold θ. th The distance is the smallest and less than the threshold. The line segments are used as matching pairs; Calculate the angle θ between the 2D line segment l and the plane π, and denote p. s p e Representing the start and end points of line segment l respectively, we obtain the 3D coordinates P of the corresponding line segment in the image space coordinate system. s P e Then the direction vector n of the line L Represented as: Then the normal vector n between the line and the plane π is... P The included angle θ between them is: Setting points Let be the i-th point on a 2D laser line segment, then the distance d from this line segment to the plane π is... lp The calculation is as follows: In the formula, N represents the number of laser points contained in the 2D line segment.

10. The single-line lidar and camera calibration method based on point, line, and surface matching according to claim 1, characterized in that, In step 3, the extrinsic parameter transformation T cl * for: In the formula, p represents the i-th laser 2D feature point i to the corresponding spatial straight line The distance; Represents the j-th laser segment l j to the corresponding space plane π j Distance; Σ p With Σ l These represent the weighting factors for the laser point and line segment errors, respectively.

Images