A lidar-camera online self-calibration method based on semantic edge alignment

Abstract

The application discloses a kind of laser radar-camera online self-calibration method based on semantic edge alignment.First, the data collected by image and laser radar are carried out semantic segmentation, obtain image semantic segmentation and point cloud semantic segmentation result;Then, the edge of image semantic segmentation result is extracted, and the edge of point cloud semantic segmentation result is extracted after ring view transformation and filling.In the use of existing deviation initial external parameter, the point cloud semantic edge is projected to the image pixel coordinate system, and the external parameter between laser radar and camera is optimized and corrected by the way of constantly fine search maximum matching score from rough to fine.The application realizes online joint external parameter self-calibration without depending on calibration board, can adapt to the edge matching of arbitrary shape semantic category, does not depend on specific scene, and has high calibration accuracy and good reliability, improves the scene adaptability of laser radar and camera online self-calibration, and has high practical value for accurate multi-sensor fusion of automatic driving.

Description

Technical Field

[0001] This invention relates to a LiDAR-camera self-calibration method in the fields of autonomous driving technology and multi-sensor fusion, and in particular to an online LiDAR-camera self-calibration method based on semantic edge alignment. Background Technology

[0002] To ensure accurate and stable perception of their surroundings, autonomous vehicles are typically equipped with a range of different sensors. Cameras and LiDAR are two of the main sensor types used in autonomous vehicles. The complementary nature of these two sensors makes them the preferred combination for many perception tasks. LiDAR sensors can acquire spatial data over a wide area but have low resolution and lack color information, while camera sensors can acquire high-resolution RGB images but are sensitive to light and lack distance information. To compensate for each other's weaknesses, the combination of LiDAR and camera sensors has become a typical and essential setup for mobile robotics and autonomous vehicle applications.

[0003] Accurate extrinsic parameters, i.e., estimating the transformation matrix between the two sensor coordinate systems, are a crucial prerequisite for successful integration and are typically the first and most critical step. Artificial targets such as checkerboard patterns are commonly used for parameter calibration. However, due to sensor aging, bumps, and collisions during driving, the accuracy of these calibration parameters decreases over time. Particularly in vehicles, large errors in rotational parameters are more common than errors in translational parameters. Therefore, an accurate and reliable online calibration method is needed to effectively fuse geometric and optical information to calibrate the offset calibration parameters online during vehicle operation.

[0004] Existing self-calibration techniques typically require specific structural objects in the scene, such as pillars, poles, or large flat surfaces, or prior constraints such as parallel lane lines, resulting in poor scene adaptability for online self-calibration. This method utilizes the semantic edge information of arbitrary-shaped objects present in general road scenes, achieving self-calibration by aligning the semantic edges of the image with those in the laser beam, significantly improving the scene robustness of the calibration. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for online self-calibration of lidar and camera based on semantic edge alignment without calibration device, which solves the problem of extrinsic parameter deviation between sensors during the driving process of autonomous vehicles.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] I. An Online Self-Calibration Method for LiDAR-Camera Based on Semantic Edge Alignment

[0008] 1) Perform semantic segmentation processing on the image data acquired by the camera to obtain the image semantic segmentation results, and extract the image semantic edges based on the semantic segmentation results;

[0009] 2) Perform semantic segmentation processing on the point cloud data collected by the lidar to obtain the point cloud semantic segmentation result, and extract the point cloud semantic edge based on the point cloud semantic segmentation result;

[0010] 3) Project the semantic edges of the point cloud onto the camera image coordinate system using the initial extrinsic parameter matrix to obtain the projected 2D point cloud edges; perform iterative optimization of the extrinsic parameters based on the projected 2D point cloud edges and the image semantic edges to align the projected 2D point cloud edges with the image semantic edges, thereby obtaining the calibrated extrinsic parameter matrix.

[0011] In step 1), the image semantic edge is obtained by processing the image semantic segmentation result using an edge extraction method.

[0012] In step 2), the semantic edges of the point cloud are extracted based on the point cloud semantic segmentation results, specifically as follows:

[0013] First, the point cloud semantic segmentation result is transformed into a surround view to obtain a sparse surround view image. The sparse surround view image contains four channels, which represent the 3D coordinates and semantic prediction results of the point cloud, respectively. Then, the sparse surround view image is filled and completed into a dense image to obtain a dense surround view image. Finally, edge detection is used on the dense surround view image to extract edges, thereby obtaining the semantic edges of the point cloud.

[0014] Specifically, 3) refers to:

[0015] 3.1) Based on the current extrinsic matrix and the camera's intrinsic matrix, project the semantic edges of the point cloud onto the image to generate the projected 2D point cloud edges in the camera image coordinate system.

[0016] 3.2) Count the number of image semantic edge points of the same category near each 2D point cloud projection point in the projected 2D point cloud edge and record it as the matching number of each 2D point cloud projection point. If the matching number of each 2D point cloud projection point is greater than or equal to 1, then each 2D point cloud projection point is recorded as a matching edge point.

[0017] 3.3) If the ratio of the number of matched edge points to the number of points in the projected 2D point cloud edge is less than a preset threshold, a calibration anomaly alarm will be triggered; otherwise, proceed to 3.4).

[0018] 3.4) The matching confidence of each 2D point cloud projection point is obtained by weighted summation based on semantic category weight and matching number. The matching confidence of all 2D point cloud projection points in the edge of the projected 2D point cloud is used as the matching confidence of the current extrinsic matrix.

[0019] 3.5) Repeat steps 3.1)-3.4) to perform the extrinsic parameter search from coarse to fine, and use the extrinsic parameter matrix with the highest matching confidence as the calibrated extrinsic parameter matrix.

[0020] In step 1), a neural network is used to perform semantic segmentation processing on the image data acquired by the camera to obtain the image semantic segmentation result.

[0021] In step 2), a neural network is used to perform semantic segmentation processing on the point cloud data collected by the lidar to obtain the point cloud semantic segmentation result.

[0022] II. A computer device

[0023] The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method.

[0024] III. A computer-readable storage medium

[0025] The medium stores a computer program that, when executed by a processor, implements the steps of the method.

[0026] IV. A computer program product

[0027] The product includes a computer program / instructions that, when executed by a processor, implement the steps of the method.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. This invention fully utilizes the correspondence between the semantic edges of LiDAR point clouds and the semantic edges of camera images. Without requiring specific calibration objects, it can adapt to semantic category edge matching of arbitrary shapes, achieving online calibration of the extrinsic parameter matrices between the LiDAR and camera, thus ensuring the stability of parameters between sensors during autonomous vehicle operation. The calibration anomaly alarm design considers specific problems encountered in actual vehicle driving scenarios, reminding the owner that maintenance is required.

[0030] 2. This invention uses neural networks to obtain semantic segmentation results of images and point clouds. Compared with traditional algorithms, it has more stable features and better robustness. Moreover, as the amount of data collected during vehicle operation increases, the segmentation effect of the trained neural network will become more accurate.

[0031] 3. This invention does not rely on specific features in the scene, achieving high calibration accuracy and reliability. It overcomes the limitations of existing self-calibration methods that overly depend on specific artificial objects such as pillars and lane lines, improving the scene adaptability of online self-calibration of LiDAR and cameras. This has high practical value for accurate multi-sensor fusion in autonomous driving. Existing online calibration algorithms rely on features and assumptions in the scene, such as always being able to see lane lines and that the lane lines are parallel, or that the ground is level. This invention, however, does not require specific, clear targets in the scene. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, and to better demonstrate the inventiveness, practicality, and basic technical principles, the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart of the overall method of the present invention.

[0034] Figure 2 This is an image semantic edge map according to an embodiment of the present invention.

[0035] Figure 3 This is a point cloud semantic edge map according to an embodiment of the present invention.

[0036] Figure 4 These are the point cloud projection images of the lidar before and after calibration in an embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0038] The embodiments and implementation process of the complete method according to the invention are as follows:

[0039] like Figure 1 As shown, this invention provides an online self-calibration method for LiDAR-camera based on semantic edge alignment, comprising the following steps:

[0040] 1) Use a neural network to perform semantic segmentation on the image data acquired by the camera to obtain dense image semantic segmentation results. Based on the semantic segmentation results, use edge extraction methods to extract the semantic edges of the image.

[0041] 1) Specifically:

[0042] 1.1) The original image is processed using the OneFormer image semantic segmentation network to obtain semantic segmentation results of the original image size. Different semantic categories are represented by different RGB colors.

[0043] 1.2) For the semantic segmentation image obtained in 1.1), the Canny operator is used to fit and obtain semantic edges, such as... Figure 2 As shown, different colors represent different semantic categories.

[0044] 2) Use a neural network to perform semantic segmentation on the point cloud data collected by the lidar to obtain the point cloud semantic segmentation result, and extract the point cloud semantic edge based on the point cloud semantic segmentation result;

[0045] like Figure 3 As shown, step 2) specifically involves:

[0046] 2.1) Use the Cylinder3d point cloud semantic segmentation network to process the original point cloud and predict the semantic category of each point in the point cloud;

[0047] 2.2) For the point cloud semantic segmentation result obtained in 2.1), the point cloud is transformed into a range-view image using the LiDAR beam and its vertical and horizontal resolutions, generating a sparse range-view image corresponding to the point cloud semantic segmentation result. The sparse range-view image contains four channels, representing the 3D coordinates of the point cloud and the semantic prediction result, respectively. In the following steps, the semantic category single-channel image of the sparse range-view image is processed;

[0048] 2.3) The sparse ring view image obtained in step 2.2) is sparse. In order to fit the edges using the Canny operator, the inpaint function in the OpenCV library is first used to fill the pixels at the hole positions to obtain a dense ring view image.

[0049] 2.4) The semantic edges of the semantic category channels of the sparse ring view image are extracted using the Canny edge detection method. Since the channels of the sparse ring view image preserve the 3D position of the original point cloud, the semantic edges of the original point cloud can be obtained directly, thus obtaining the semantic edges of the point cloud.

[0050] 3) Project the semantic edges of the point cloud onto the camera image coordinate system using the initial calibration extrinsic parameter matrix to obtain the projected 2D point cloud edges; perform iterative optimization of the extrinsic parameters based on the projected 2D point cloud edges and the image semantic edges to align the projected 2D point cloud edges with the image semantic edges, thereby obtaining the calibration extrinsic parameter matrix.

[0051] Step 3) specifically refers to:

[0052] 3.1) Based on the current calibration extrinsic matrix and the camera's intrinsic matrix, project the semantic edges of the point cloud onto the image to generate the projected 2D point cloud edges in the camera image coordinate system. The mathematical expression for the projection is as follows:

[0053]

[0054] Where X, Y, Z are the 3D position coordinates of the point cloud in the lidar coordinate system, T is the extrinsic parameter matrix between the lidar and the camera, K is the intrinsic parameter matrix of the camera, Zc is the depth of the point in the camera coordinate system, and u and v are the two pixel coordinates of the projected point.

[0055] 3.2) Count the number of image semantic edge points of the same category as each 2D point cloud projection point within a 5x5 pixel window (centered on the 2D point cloud projection point) in the 2D point cloud edge after projection, and record it as the matching number of each 2D point cloud projection point. If the matching number of each 2D point cloud projection point is greater than or equal to 1, then each 2D point cloud projection point is recorded as a matching edge point.

[0056] 3.3) If the ratio of the number of matched edge points to the number of points in the projected 2D point cloud edge is less than 0.037, a calibration anomaly alarm will be triggered to remind the vehicle owner that maintenance is needed when the external parameter deviation is too large; otherwise, 3.4) will be executed.

[0057] 3.4) Add weights to different semantic categories, such as car category weight 3, pole category weight 5, traffic sign category weight 7, and other analogy weights 1; perform weighted summation on each 2D point cloud projection point according to the semantic category weight and the number of matches to obtain the matching confidence of each 2D point cloud projection point, and use the sum of the matching confidence of all 2D point cloud projection points in the edge of the projected 2D point cloud as the matching confidence of the current extrinsic matrix;

[0058] 3.5) Repeat steps 3.1)-3.4) to perform extrinsic parameter searches from coarse to fine at the three rotation angles: roll, pitch, and yaw. The extrinsic parameter matrix with the highest matching confidence is used as the calibrated extrinsic parameter matrix. Specifically, first, a local optimum search is performed at 1.5° intervals between -5° and 5.5°. Then, a more refined local optimum search is performed at 0.5° intervals within the angle range of -1.5° to 1.5° of the local optimum solution. Finally, a search is performed at 0.2° intervals within the angle range of -0.5° to 0.5° of the refined local optimum angle.

[0059] Figure 4This paper demonstrates the effects of this invention before and after calibration on the KITTI Odometry dataset. The first row shows the results of projecting all points in the point cloud onto the image, and the second row shows the alignment results of the projected 2D point cloud edges with the semantic edges of the image. The first column shows the results after projection according to the ground truth of the LiDAR-camera extrinsic parameter calibration. The second column shows the projection results after adding 5 degrees of random noise at 3 rotation angles. The third column shows the projection results after the invention corrects the extrinsic parameters through self-calibration after adding random noise. Before calibration, there is a significant deviation between the semantic edges of the point cloud and the semantic edges of the image from the perspective of semantic edge projection. After calibration, the large rotation error is corrected by the algorithm, and the edge alignment results almost perfectly match the ground truth projection results.

[0060] Table 1 lists the calibration results of this invention on the KITTI Odometry dataset. Four frames of data were randomly selected from the dataset, and 5-degree and 3-degree angular noise were randomly added to each. After calibrating the lidar-camera extrinsic parameters using this invention, the error between the calibration results and the true values ​​was calculated. The table shows that the calibration accuracy of this invention for rotation angles is less than 0.5 degrees. Furthermore, setting the angle anomaly value to 4° and adding 5-degree noise demonstrates the effectiveness of the anomaly alarm function, and the alarm latency is statistically analyzed. As shown in the last column of Table 1, the alarm latency is less than 0.1 seconds.

[0061] Table 1 shows the self-calibration results of this invention on the KITTI Odometry dataset.

[0062]

[0063] To further demonstrate the effectiveness of this invention, Table 2 lists the calibration results of this invention on the KITTI Odometry dataset 00 sequence. Random noise was uniformly added to all data in the 00 sequence, and the average error between the calibrated value and the true value after calibration by this invention was calculated. As can be seen from the table, this invention also achieved an accuracy of less than 0.5 degrees across the entire sequence.

[0064] Table 2 shows the self-calibration results of this invention under the KITTI Odometry00 sequence.

[0065] 00 sequence roll(°) pitch (°) yaw(°) Average calibration error (°) 5 degrees of noise 0.0980 0.1534 0.2277 0.1597 3 degrees of noise 0.1310 0.1351 0.2266 0.1643

[0066] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for online self-calibration of a lidar-camera system based on semantic edge alignment, characterized in that, Includes the following steps: 1) Perform semantic segmentation processing on the image data acquired by the camera to obtain the image semantic segmentation result, and extract the image semantic edges based on the semantic segmentation result; 2) Perform semantic segmentation processing on the point cloud data collected by the lidar to obtain the point cloud semantic segmentation result, and extract the point cloud semantic edge based on the point cloud semantic segmentation result; In step 2), the semantic edges of the point cloud are extracted based on the point cloud semantic segmentation results, specifically as follows: First, the point cloud semantic segmentation result is transformed into a surround view to obtain a sparse surround view image. The sparse surround view image contains four channels, which represent the 3D coordinates and semantic prediction results of the point cloud, respectively. Then, the sparse surround view image is filled and completed into a dense image to obtain a dense surround view image. Next, edge detection is used on the dense surround view image to extract edges to obtain the semantic edges of the point cloud. 3) Project the semantic edges of the point cloud onto the camera image coordinate system using the initial extrinsic parameter matrix to obtain the projected 2D point cloud edges; perform iterative optimization of the extrinsic parameters based on the projected 2D point cloud edges and the image semantic edges to align the projected 2D point cloud edges with the image semantic edges, thereby obtaining the calibrated extrinsic parameter matrix. Specifically, 3) refers to: 3.1) Based on the current extrinsic matrix and the camera's intrinsic matrix, project the semantic edges of the point cloud onto the image to generate the projected 2D point cloud edges in the camera image coordinate system; 3.2) Count the number of image semantic edge points of the same category near each 2D point cloud projection point in the projected 2D point cloud edge and record it as the matching number of each 2D point cloud projection point. If the matching number of each 2D point cloud projection point is greater than or equal to 1, then each 2D point cloud projection point is recorded as a matching edge point. 3.3) If the ratio of the number of matched edge points to the number of points in the projected 2D point cloud edge is less than a preset threshold, a calibration anomaly alarm will be triggered; otherwise, proceed to step 3.4). 3.4) The matching confidence of each 2D point cloud projection point is obtained by weighted summation based on semantic category weight and matching number. The matching confidence of all 2D point cloud projection points in the edge of the projected 2D point cloud is used as the matching confidence of the current extrinsic matrix. 3.5) Repeat steps 3.1)-3.4) to perform extrinsic parameter search from coarse to fine, and use the extrinsic parameter matrix with the highest matching confidence as the calibrated extrinsic parameter matrix.

2. The online self-calibration method for LiDAR-camera based on semantic edge alignment according to claim 1, characterized in that, In step 1), the image semantic edge is obtained by processing the image semantic segmentation result using an edge extraction method.

3. The online self-calibration method for LiDAR-camera based on semantic edge alignment according to claim 1, characterized in that, In step 1), a neural network is used to perform semantic segmentation processing on the image data acquired by the camera to obtain the image semantic segmentation result.

4. The online self-calibration method for LiDAR-camera based on semantic edge alignment according to claim 1, characterized in that, In step 2), a neural network is used to perform semantic segmentation processing on the point cloud data collected by the lidar to obtain the point cloud semantic segmentation result.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 4.

Images