Laser radar-camera external parameter online calibration method and system based on natural scene

Abstract

The invention provides a laser radar-camera external parameter online calibration method and system based on a natural scene, and relates to the technical field of sensor calibration, and the method comprises the steps: carrying out the coarse alignment of an obtained laser radar point cloud and image, and obtaining a coarse alignment external parameter matrix; adding disturbance to the coarse alignment extrinsic parameter matrix, and projecting the laser radar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic parameter matrix and the disturbed coarse alignment extrinsic parameter matrix; respectively processing the laser radar point cloud and the image by using a semantic segmentation algorithm to obtain an image semantic segmentation image and a point cloud semantic segmentation image; sending the original image, the two sparse depth images and the two semantic segmentation images into the trained external parameter calibration convolutional neural network, and performing feature extraction, category-by-category feature matching and feature aggregation to obtain a six-degree-of-freedom external parameter matrix; according to the method, the calculation precision and generalization of the algorithm are ensured, and the effectiveness of the algorithm in a scene with non-rich geometric features can be ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of sensor calibration technology, and in particular to a method and system for online calibration of lidar-camera extrinsic parameters based on natural scenes. Background Art

[0002] With the advancement of sensor technology, the 3D reconstruction algorithms that have emerged in recent years mostly use multiple sensor types to generate high-integrity, high-precision 3D point clouds. This requires 3D reconstruction algorithms to be able to fuse and reconstruct multimodal data. This first requires rigorous spatiotemporal alignment of visible light cameras, thermal infrared cameras, and lidar. Extrinsic parameter calibration algorithms accurately estimate extrinsic parameters such as rigid body transformations between different sensors and are a crucial step in the spatiotemporal alignment of multimodal sensors.

[0003] Early sensor calibration methods relied on specific calibration targets, such as a checkerboard or homemade objects, to obtain the matching relationship between the two-dimensional features of the image and the three-dimensional features of the lidar point cloud through manual or automatic annotation, and then used this matching relationship to calculate the external parameters. For fields such as unmanned driving and geographic information mapping, perception equipment needs to operate for a long time in complex scenes. Due to changes in the operating environment (such as wind, rain, thunder, etc.), the external parameters will produce uncontrollable deviations. In order to be able to adaptively calculate the external parameters of multiple sensors, these parameters need to be calibrated online. The main difficulty is the lack of significant artificial calibration targets in natural scenes. It is difficult to form an effective correspondence based on features extracted from cross-modal data, and it is necessary to consider different imaging principles, perspective differences, resolution differences, and other issues at the same time. Summary of the Invention

[0004] In order to solve the above problems, this paper proposes a lidar-camera extrinsic parameter online calibration method and system based on natural scenes, which ensures the computational accuracy and generalization of the algorithm, and can ensure the effectiveness of the algorithm in scenes with poor geometric features.

[0005] According to some embodiments, the present disclosure adopts the following technical solutions:

[0006] The online calibration method of lidar-camera extrinsic parameters based on natural scenes includes:

[0007] Obtain the lidar point cloud and image collected by the camera and lidar in the natural scene respectively, and perform coarse alignment on the lidar point cloud and image to obtain the coarse alignment extrinsic parameter matrix;

[0008] Add perturbations to the coarse alignment extrinsic parameter matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic parameter matrix and the perturbed coarse alignment extrinsic parameter matrix;

[0009] Use semantic segmentation algorithms to process the lidar point cloud and image separately to obtain image semantic segmentation images and point cloud semantic segmentation images;

[0010] The original image, two sparse depth images, and two semantic segmentation images are fed into the trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and the camera.

[0011] Furthermore, the lidar point cloud and image are obtained by collecting data with overlapping viewing angles using a camera and lidar that are pre-time synchronized.

[0012] Furthermore, the rough alignment extrinsic parameter matrix is obtained as follows:

[0013] Based on the monocular image depth estimation method, the relative depth image corresponding to the image is calculated;

[0014] According to the image intrinsic parameter matrix, the relative depth image is back-projected into three-dimensional space to generate a three-dimensional point cloud of the image;

[0015] The three-dimensional point cloud of the image is coarsely aligned with the lidar point cloud to obtain a coarse alignment extrinsic parameter matrix.

[0016] Furthermore, the semantic segmentation algorithm uses a two-dimensional image semantic segmentation algorithm and a three-dimensional point cloud semantic segmentation algorithm for images and lidar point clouds respectively.

[0017] Furthermore, the feature extraction is to extract image features and depth features from the original image and the sparse depth image respectively using a pre-trained ResNet-18 network that does not include a fully connected layer;

[0018] The category-by-category feature matching is to calculate the matching error between image features and depth features for each semantic category and construct a three-dimensional loss cube;

[0019] The feature aggregation is to directly regress the six-degree-of-freedom rigid body transformation between the camera and the lidar from the three-dimensional loss cube, and use the six-degree-of-freedom rigid body transformation to correct the calibration extrinsic parameters to obtain the final six-degree-of-freedom extrinsic parameter matrix.

[0020] Furthermore, the loss function of the extrinsic parameter calibration convolutional neural network includes regression loss, loss based on geometric features and consistency loss;

[0021] The regression loss is the difference between the external parameters calculated by the convolutional neural network and the true value;

[0022] The geometric feature-based loss consists of point-to-point distance error and point-to-plane distance error;

[0023] The consistency loss includes perspective consistency loss and semantic consistency loss.

[0024] According to some embodiments, the present disclosure adopts the following technical solutions:

[0025] The LiDAR-camera extrinsic parameter online calibration system based on natural scenes includes:

[0026] The coarse alignment module is configured to: obtain the lidar point cloud and image collected by the camera and lidar in the natural scene to be calibrated, respectively, and coarsely align the lidar point cloud and image to obtain a coarse alignment extrinsic parameter matrix;

[0027] a depth projection module configured to: add perturbations to the coarse alignment extrinsic matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic matrix and the perturbed coarse alignment extrinsic matrix;

[0028] The semantic segmentation module is configured to: use a semantic segmentation algorithm to process the lidar point cloud and the image respectively to obtain an image semantic segmentation image and a point cloud semantic segmentation image;

[0029] The extrinsic parameter calibration module is configured to feed the original image, two sparse depth images, and two semantic segmentation images into a trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and camera.

[0030] Furthermore, the lidar point cloud and image are obtained by collecting data with overlapping viewing angles using a camera and lidar that are pre-time synchronized.

[0031] Furthermore, the rough alignment extrinsic parameter matrix is obtained as follows:

[0032] Based on the monocular image depth estimation method, the relative depth image corresponding to the image is calculated;

[0033] According to the image intrinsic parameter matrix, the relative depth image is back-projected into three-dimensional space to generate a three-dimensional point cloud of the image;

[0034] The three-dimensional point cloud of the image is coarsely aligned with the lidar point cloud to obtain a coarse alignment extrinsic parameter matrix.

[0035] Furthermore, the semantic segmentation algorithm uses a two-dimensional image semantic segmentation algorithm and a three-dimensional point cloud semantic segmentation algorithm for images and lidar point clouds respectively.

[0036] Furthermore, the feature extraction is to extract image features and depth features from the original image and the sparse depth image respectively using a pre-trained ResNet-18 network that does not include a fully connected layer;

[0037] The category-by-category feature matching is to calculate the matching error between image features and depth features for each semantic category and construct a three-dimensional loss cube;

[0038] The feature aggregation is to directly regress the six-degree-of-freedom rigid body transformation between the camera and the lidar from the three-dimensional loss cube, and use the six-degree-of-freedom rigid body transformation to correct the calibration extrinsic parameters to obtain the final six-degree-of-freedom extrinsic parameter matrix.

[0039] Furthermore, the loss function of the extrinsic parameter calibration convolutional neural network includes regression loss, loss based on geometric features and consistency loss;

[0040] The regression loss is the difference between the external parameters calculated by the convolutional neural network and the true value;

[0041] The geometric feature-based loss consists of point-to-point distance error and point-to-plane distance error;

[0042] The consistency loss includes perspective consistency loss and semantic consistency loss.

[0043] According to some embodiments, the present disclosure adopts the following technical solutions:

[0044] A computer program product includes a computer program, which, when executed by a processor, implements the natural scene-based lidar-camera extrinsic parameter online calibration method.

[0045] According to some embodiments, the present disclosure adopts the following technical solutions:

[0046] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method for online calibration of lidar-camera extrinsic parameters based on natural scenes is implemented.

[0047] According to some embodiments, the present disclosure adopts the following technical solutions:

[0048] An electronic device includes: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the natural scene-based lidar-camera extrinsic parameter online calibration method.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention proposes a lidar-camera extrinsic parameter online calibration method, which can be based on images (visible light images or thermal infrared images) and lidar point clouds collected in natural scenes, adopt a semantically assisted end-to-end lidar-camera extrinsic parameter calibration convolutional neural network, and design regression loss, geometric feature-based loss and consistency loss to optimize the neural network. The extrinsic parameters of the camera (visible light camera or thermal infrared camera) and lidar are calibrated online to obtain a six-degree-of-freedom extrinsic parameter matrix between the lidar and the camera. Through the above method, online calibration of the lidar and camera is realized, and automatic correction of extrinsic parameters between sensors in operation is supported. The technical solution has good robustness, scalability and computational efficiency, and can be used in unmanned driving perception, geographic information surveying and mapping and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0052] Figure 1 This is a flow chart of the method of Example 1.

[0053] Figure 2 This is the structure diagram of the convolutional neural network calibrated with external parameters in Example 1. DETAILED DESCRIPTION

[0054] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0057] Example 1

[0058] In one embodiment of the present disclosure, a method for online calibration of lidar-camera extrinsic parameters based on natural scenes is provided to achieve online calibration of lidar and camera, support automatic correction of extrinsic parameters between sensors in operation, and have good robustness, scalability, and computational efficiency. It can be used in fields such as unmanned driving perception, geographic information surveying and mapping, etc. Figure 1 As shown, the specific steps are:

[0059] Step S1: Obtain the lidar point cloud and image collected by the camera and lidar in the natural scene to be calibrated respectively, and perform coarse alignment on the lidar point cloud and image to obtain a coarse alignment extrinsic parameter matrix.

[0060] Use pre-time-synchronized cameras and lidars to collect images and lidar point clouds in natural scenes, respectively. The images and lidar point clouds must have overlapping viewing angles. The camera here can be a visible light camera or a thermal infrared camera, and the corresponding visible light images or thermal infrared images are collected. The natural scene refers to a collection scene that does not contain artificial targets such as checkerboards used for calibration, which is commonly seen in application fields such as unmanned driving or geographic information surveying and mapping.

[0061] Since the subsequent process requires the projection of the laser point cloud onto the image coordinate system, it is necessary to calculate the coarse alignment extrinsic parameter matrix of the camera and lidar. The alignment of the image and point cloud usually requires establishing a correlation between the data, and the most suitable correlation is the 3D point correlation. Therefore, this embodiment uses 3D point correlation to achieve coarse alignment. The specific steps are as follows:

[0062] Step S1-1: Calculate the relative depth image corresponding to the image based on a monocular image depth estimation method.

[0063] Use a monocular depth estimation convolutional neural network (such as MiDaS) to calculate the relative depth image corresponding to the image; considering that a fixed module camera is used here, it is necessary to calibrate the monocular depth estimation convolutional neural network before use, and establish a transformation relationship between the relative depth calculated by the convolutional neural network and the absolute depth. This transformation relationship is jointly represented by the scale ratio and the offset, so as to transform the relative depth image into the absolute depth image. At the same time, considering that the prediction accuracy of monocular depth estimation decreases when the target distance is far, pixels with absolute depth greater than a certain threshold are discarded.

[0064] Step S1-2: Based on the image intrinsic parameter matrix, the absolute depth image is back-projected into three-dimensional space to generate a three-dimensional point cloud of the image.

[0065] According to the image intrinsic parameter matrix K, the absolute depth image is back-projected into the three-dimensional space to generate the image three-dimensional point cloud. The back-projection operation is expressed as:

[0066]

[0067] Among them, (u i ,v i ) represents the coordinate of the i-th pixel in the absolute depth image in the image coordinate system, d i is the laser point cloud depth value of pixel i, K -1is the inverse of the camera intrinsic parameter matrix. The above operation converts the two-dimensional depth image result into a three-dimensional point cloud.

[0068] Step S1-3: Roughly align the three-dimensional point cloud of the image with the lidar point cloud to obtain a rough alignment extrinsic parameter matrix.

[0069] The three-dimensional point cloud and lidar point cloud corresponding to the image after the above processing belong to different coordinate systems. Considering that the subsequent algorithm requires that the perspective difference between the image and the lidar should not be too large, the two types of point clouds need to be roughly aligned.

[0070] The rough alignment method of the three-dimensional point cloud is used to obtain the rough alignment external parameter matrix T0 of the image and the lidar. In this embodiment, the rough alignment method of the three-dimensional point cloud is performed by the commonly used point cloud registration method based on FPFH features.

[0071] Since monocular depth estimation itself is a degenerate problem, the depth map calculated by the corresponding monocular depth estimation convolutional neural network contains large noise. Therefore, the coarse alignment extrinsic parameter matrix T0 obtained in this step is not accurate and needs to be optimized in subsequent steps.

[0072] Step S2: Add perturbation to the coarse alignment extrinsic parameter matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic parameter matrix and the perturbed coarse alignment extrinsic parameter matrix.

[0073] Based on the coarse alignment extrinsic parameter matrix T0 obtained in the previous step, the lidar point cloud is projected in the image coordinate system to obtain a sparse depth image D0.

[0074] To provide more constraints for extrinsic calibration of convolutional neural networks, the extrinsic matrix T0 is slightly perturbed to obtain T1, i.e., T1 = ΔT·T0. The rotation matrix and translation vector of ΔT are uniquely determined by the quaternion perturbation Δq and the translation vector perturbation Δt. It is necessary to ensure that the perturbed viewpoint overlaps with the original image viewpoint. To increase information gain, T1 is perturbed in both rotation and translation. The lidar point cloud is projected onto the T1 pose matrix to obtain a redundant sparse depth image D1.

[0075] Step S3: Use the semantic segmentation algorithm to process the lidar point cloud and image respectively to obtain an image semantic segmentation image and a point cloud semantic segmentation image.

[0076] The image and laser point cloud are processed simultaneously using a 2D image semantic segmentation algorithm and a 3D point cloud semantic segmentation algorithm, respectively, to generate image semantics and point cloud semantics. The 2D image semantic segmentation algorithm uses DeepLab v3+ or another 2D image semantic segmentation convolutional neural network, while the 3D point cloud semantic segmentation algorithm uses the PointNet series or another 3D point cloud semantic segmentation convolutional neural network. The 3D point cloud semantic segmentation results are projected onto T0 and T1, respectively, to produce two projected semantic images, S0 and S1. The 2D point cloud semantic segmentation results do not require projection, as projection simply transforms the 3D point cloud segmentation into a 2D segmentation. These steps prepare the input data for the LiDAR-Camera extrinsic parameter online calibration convolutional neural network.

[0077] Step S4: The original image, two sparse depth images, and two semantic segmentation images are fed into the trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and the camera.

[0078] This embodiment uses a semantically assisted end-to-end lidar-camera extrinsic parameter calibration method, which feeds the original image, two sparse depth images, and two semantic images into an extrinsic parameter calibration convolutional neural network to obtain a six-degree-of-freedom extrinsic parameter matrix between the lidar and the camera. Since the extrinsic parameter matrix can be uniquely determined by the rotation quaternion and the translation vector, considering the computational complexity, this embodiment calculates the rotation quaternion and the translation vector in the extrinsic parameter calibration method to obtain the six-degree-of-freedom extrinsic parameter matrix; the semantic image includes two types: the semantic image corresponding to the original image and the semantic image of the semantic projection of the three-dimensional point cloud.

[0079] The input of the image-laser extrinsic online calibration network is two projected semantic images, two sparse depth images and an image. The following is an example of a visible light image. Figure 2 As shown in the figure, the network framework consists of three parts: feature extraction module, feature matching module and feature aggregation module. Since the calculation processes of the above three parts are differentiable, the designed network can be trained end-to-end.

[0080] The feature extraction module extracts depth features from two sparse depth images and a visible light image generated by projecting the lidar three-dimensional point cloud onto the image plane.

[0081] Considering that the online calibration of external parameters as the first step of 3D perception plays a decisive role in the accuracy of subsequent tasks, a relatively more complex network structure is used in this network. This project intends to use a pre-trained ResNet-18 network without a fully connected layer to extract visible light image features and depth image features. The activation functions are Rectified Linear Unit (ReLU) and Leaky Rectified Linear Unit (LReLU), respectively, to extract visible light image features f from the two input images. vi and the corresponding depth image feature f li .

[0082] In extracting visible light image features f vi and the corresponding depth image feature f li Afterwards, the matching error between the two is calculated using the same structure of the category-by-category feature matching module. In this embodiment, the matching error is calculated using the correlation layer and stored in a three-dimensional loss cube (Cost Volume). The calculation is as follows:

[0083]

[0084] Among them, c(f) represents the flattened vector of the feature map f, where f can be the visible light image feature f vi Or the corresponding depth image feature f li , N represents the length of the feature column vector, and T represents the transpose operation. For the feature map f, the loss of the pixels within a certain threshold d is calculated to form a local loss cube; the dimension of the entire three-dimensional loss cube is d 2 ×H×W, where H and W represent the height and width of the image feature map.

[0085] To better utilize semantic information, this embodiment uses a per-category feature matching module. Specifically, a rotation quaternion and translation vector are calculated for each category in the semantic image. Specifically, a three-dimensional loss cube is constructed for each semantic category, and a subsequent feature aggregation module is cascaded to output the rotation quaternion and translation vector.

[0086] The feature aggregation module directly regresses the six-degree-of-freedom extrinsic parameter matrix between the visible light camera and the lidar from the three-dimensional loss cube; this module consists of a fully connected layer with 512 neurons and two branches with stacked fully connected layers, which represent rotation and translation respectively; the network output is a 1×3 translation vector and a 1×4 rotation quaternion.

[0087] When designing the loss function, we use the regression loss commonly used in deep learning and the distance loss based on geometric features. Given a visible light image I and an initial depth image D0 projected based on the coarse alignment external parameter matrix T0, the total loss is calculated as follows:

[0088]

[0089] Among them, α and β are loss weights.

[0090] (1) The regression loss is calculated by the difference between the external parameters (i.e. translation vector and rotation quaternion) calculated by the online calibration network and the true value, which is calculated by the translation vector loss and rotation quaternion loss Together they constitute:

[0091]

[0092] in, and and are the translation vector and rotation quaternion predicted by the neural network at T0 and T1 poses, respectively. t0 and q0, t1 and q1 are the corresponding true values. ‖·‖2 represents the L2 norm and γ is the loss weight.

[0093] (2) is the loss based on geometric features, consisting of point-to-point distance error Point-to-plane distance error The above errors are calculated as follows:

[0094]

[0095] Among them, T LC is the true value of the extrinsic parameter matrix between visible light and laser image, T pred is the relative transformation calculated for each iteration of the network, T0 is the initial pose, P i is a 3D point in the lidar point cloud, n is P i The normal of the plane, M is the number of three-dimensional points, and ζ is the loss weight.

[0096] For the three-dimensional laser radar point, the following method is used to extract the plane where the three-dimensional point in the laser radar point cloud is located. i , use the K nearest neighbor algorithm to calculate the nearest three-dimensional point set S. Assuming that all three-dimensional points in the three-dimensional point set S are located on the spatial plane, by solving the following equation:

[0097]

[0098] The plane normal n=m / |m| can be obtained if P i All three-dimensional points in the neighborhood set S of satisfy the conditions:

[0099]

[0100] Then determine P i is a characteristic three-dimensional point on the plane, where p th Indicates a threshold with a very small value. In addition, it can also analyze the straight line features in the scene and establish the straight line to straight line distance loss.

[0101] (3) Consistency loss, including perspective consistency loss and semantic consistency loss Right now:

[0102]

[0103] in, and is the predicted relative translation vector and rotation quaternion, and is the corresponding true value, ‖·‖2 represents the L2 norm; and is the relative translation vector and rotation quaternion corresponding to the predicted semantic category n, and is the corresponding true value.

[0104] The prediction transformation T is obtained by calibrating the network online through external parameters. pred , the coarse alignment extrinsic matrix T0 is corrected using the following formula to obtain the final six-degree-of-freedom extrinsic matrix

[0105]

[0106] The external parameter online calibration network is trained by changing different translation and rotation error ranges, and gradually training from a larger error range to a smaller error range.

[0107] Example 2

[0108] In one embodiment of the present disclosure, a natural scene-based online calibration system for lidar and camera extrinsic parameters is provided, comprising:

[0109] The coarse alignment module is configured to: obtain the lidar point cloud and image collected by the camera and lidar in the natural scene to be calibrated, respectively, and coarsely align the lidar point cloud and image to obtain a coarse alignment extrinsic parameter matrix;

[0110] a depth projection module configured to: add perturbations to the coarse alignment extrinsic matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic matrix and the perturbed coarse alignment extrinsic matrix;

[0111] The semantic segmentation module is configured to: use a semantic segmentation algorithm to process the lidar point cloud and the image respectively to obtain an image semantic segmentation image and a point cloud semantic segmentation image;

[0112] The extrinsic parameter calibration module is configured to feed the original image, two sparse depth images, and two semantic segmentation images into a trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and camera.

[0113] Furthermore, the lidar point cloud and image are obtained by collecting data with overlapping viewing angles using a camera and lidar that are pre-time synchronized.

[0114] Furthermore, the rough alignment extrinsic parameter matrix is obtained as follows:

[0115] Based on the monocular image depth estimation method, the relative depth image corresponding to the image is calculated;

[0116] According to the image intrinsic parameter matrix, the relative depth image is back-projected into three-dimensional space to generate a three-dimensional point cloud of the image;

[0117] The three-dimensional point cloud of the image is coarsely aligned with the lidar point cloud to obtain a coarse alignment extrinsic parameter matrix.

[0118] Furthermore, the semantic segmentation algorithm uses a two-dimensional image semantic segmentation algorithm and a three-dimensional point cloud semantic segmentation algorithm for images and lidar point clouds respectively.

[0119] Furthermore, the feature extraction is to extract image features and depth features from the original image and the sparse depth image respectively using a pre-trained ResNet-18 network that does not include a fully connected layer;

[0120] The category-by-category feature matching is to calculate the matching error between image features and depth features for each semantic category and construct a three-dimensional loss cube;

[0121] The feature aggregation is to directly regress the six-degree-of-freedom rigid body transformation between the camera and the lidar from the three-dimensional loss cube, and use the six-degree-of-freedom rigid body transformation to correct the calibration extrinsic parameters to obtain the final six-degree-of-freedom extrinsic parameter matrix.

[0122] Furthermore, the loss function of the extrinsic parameter calibration convolutional neural network includes regression loss, loss based on geometric features and consistency loss;

[0123] The regression loss is the difference between the external parameters calculated by the convolutional neural network and the true value;

[0124] The geometric feature-based loss consists of point-to-point distance error and point-to-plane distance error;

[0125] The consistency loss includes perspective consistency loss and semantic consistency loss.

[0126] Example 3

[0127] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the natural scene-based lidar-camera extrinsic parameter online calibration method.

[0128] Example 4

[0129] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, which is used to store computer instructions. When the computer instructions are executed by a processor, the method for online calibration of lidar-camera extrinsic parameters based on natural scenes is implemented.

[0130] Example 5

[0131] In one embodiment of the present disclosure, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the natural scene-based lidar-camera extrinsic parameter online calibration method.

[0132] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0134] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A lidar-camera extrinsic parameter online calibration method based on natural scenes, characterized by: include: Obtain the lidar point cloud and image collected by the camera and lidar in the natural scene respectively, and perform coarse alignment on the lidar point cloud and image to obtain the coarse alignment extrinsic parameter matrix; Add perturbations to the coarse alignment extrinsic parameter matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic parameter matrix and the perturbed coarse alignment extrinsic parameter matrix; Use semantic segmentation algorithms to process the lidar point cloud and image separately to obtain image semantic segmentation images and point cloud semantic segmentation images; The original image, two sparse depth images, and two semantic segmentation images are fed into the trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and the camera.

2. The method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to claim 1, wherein: The laser radar point cloud and image are obtained by collecting data with overlapping viewing areas using a camera and a laser radar that are pre-time synchronized.

3. The method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to claim 1, wherein: The rough alignment extrinsic parameter matrix is obtained as follows: Based on the monocular image depth estimation method, the relative depth image corresponding to the image is calculated; According to the image intrinsic parameter matrix, the relative depth image is back-projected into three-dimensional space to generate a three-dimensional point cloud of the image; The three-dimensional point cloud of the image is coarsely aligned with the lidar point cloud to obtain a coarse alignment extrinsic parameter matrix.

4. The method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to claim 1, wherein: The semantic segmentation algorithm uses a two-dimensional image semantic segmentation algorithm and a three-dimensional point cloud semantic segmentation algorithm for images and lidar point clouds respectively.

5. The method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to claim 1, wherein: The feature extraction is to extract image features and depth features from the original image and the sparse depth image respectively using a pre-trained ResNet-18 network that does not include a fully connected layer; The category-by-category feature matching is to calculate the matching error between image features and depth features for each semantic category and construct a three-dimensional loss cube; The feature aggregation is to directly regress the six-degree-of-freedom rigid body transformation between the camera and the lidar from the three-dimensional loss cube, and use the six-degree-of-freedom rigid body transformation to correct the calibration extrinsic parameters to obtain the final six-degree-of-freedom extrinsic parameter matrix.

6. The method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to claim 1, wherein: The loss function of the extrinsic parameter calibration convolutional neural network includes regression loss, loss based on geometric features and consistency loss; The regression loss is the difference between the external parameters calculated by the convolutional neural network and the true value; The geometric feature-based loss consists of point-to-point distance error and point-to-plane distance error; The consistency loss includes perspective consistency loss and semantic consistency loss.

7. The laser radar-camera extrinsic parameter online calibration system based on natural scenes is characterized by: include: The coarse alignment module is configured to: obtain the lidar point cloud and image collected by the camera and lidar in the natural scene to be calibrated, respectively, and coarsely align the lidar point cloud and image to obtain a coarse alignment extrinsic parameter matrix; a depth projection module configured to: add perturbations to the coarse alignment extrinsic matrix, and project the lidar point cloud into a sparse depth image and a redundant sparse depth image based on the coarse alignment extrinsic matrix and the perturbed coarse alignment extrinsic matrix; The semantic segmentation module is configured to: use a semantic segmentation algorithm to process the lidar point cloud and the image respectively to obtain an image semantic segmentation image and a point cloud semantic segmentation image; The extrinsic parameter calibration module is configured to feed the original image, two sparse depth images, and two semantic segmentation images into a trained extrinsic parameter calibration convolutional neural network for feature extraction, category-by-category feature matching, and feature aggregation, ultimately obtaining a six-degree-of-freedom extrinsic parameter matrix between the lidar and camera.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for online calibration of lidar-camera extrinsic parameters based on natural scenes according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the online calibration method of the lidar-camera extrinsic parameters based on natural scenes as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the natural scene-based lidar-camera extrinsic parameter online calibration method as described in any one of claims 1 to 6.

Images