A laser radar and camera external parameter online calibration method and device and storage medium

Abstract

The application relates to a laser radar and camera external parameter online calibration method and device based on a deep neural network and a storage medium, wherein the method comprises the following steps: building a laser radar and camera joint calibration platform; calibrating camera internal parameters; synchronously time stamping; acquiring a gray image and a point cloud and completing pretreatment; initializing external parameters; converting the point cloud to a camera coordinate system; based on a random mask mode, respectively covering the point cloud and the gray image according to a preconfigured covering ratio; projecting the covered point cloud to an image plane to obtain depth information projection and reflectivity information projection; constructing an external parameter online calibration network, which is composed of an input branch, a prediction branch, a fusion branch and a result output module, and realizing external parameter calibration by fusing the features of the depth information projection, a depth map, the reflectivity information projection and a covered gray image; training the external parameter online calibration network; and completing external parameter online calibration. Compared with the prior art, the application has the advantages of good calibration effect, wide application range and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of surveying and mapping, in particular to a laser radar and camera extrinsic parameter online calibration method, device and storage medium based on a deep neural network. BACKGROUND

[0002] The existing laser radar-camera extrinsic parameter online calibration algorithm usually uses the PCA algorithm to segment the rod-shaped target in the point cloud data, and uses reflectivity and ground segmentation to obtain the lane lines on the road surface. As for the line features in the image data, the existing method usually uses the semantic segmentation method to obtain sparse line features. However, there are more types of line features in real scenes, which can be extracted from point cloud and image data at the same time, but a more robust algorithm is needed to ensure the accuracy of feature matching. This cross-modal corresponding line matching algorithm first projects the point cloud into the image, and then uses the ICP (Iterative Closest Point) algorithm to find the nearest neighbor relationship between the point cloud projected into the image and the feature line in the image. This algorithm is very low in efficiency and can only deal with the case where the laser radar and camera extrinsic parameters are slightly inaccurate. The optimization problem of the backend of the past calibration system is usually a least squares problem, and the classic LM (Levenberg-Marquardt) algorithm is used to solve it, which requires that the feature matching of the front end of the calibration system has good robustness. Unlike the corresponding point association between cameras, the data association between cross-modal laser radars and cameras is more difficult, and often requires clear and prominent line features in the scene. The end-to-end laser radar-camera calibration system can learn the hidden space features that are beneficial to cross-modal data association from the original laser data and image data, so it can complete accurate laser radar-camera extrinsic parameter online calibration in a wider range of scenes.

[0003] The past end-to-end laser radar-camera extrinsic parameter calibration only considers the depth information of the laser points projected into the image plane. The depth information represents the three-dimensional spatial structure of the scene, and the reflectivity information represents the texture of the scene to some extent. However, it should be noted that in addition to the reflectivity difference caused by the texture of the scene, the depth of the scene also affects the reflectivity of the laser points. Therefore, how to fuse depth information and reflectivity information and surpass the performance of methods that only use depth information has become the key to the problem. SUMMARY

[0004] The purpose of the present application is to provide a laser radar and camera extrinsic parameter online calibration method, device and storage medium based on a deep neural network, which fuses depth information and reflectivity information to realize extrinsic parameter calibration and improve calibration accuracy.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] A kind of laser radar and camera external parameter online calibration method based on deep neural network, comprising the following steps:

[0007] Step 1) build laser radar and camera joint calibration platform;

[0008] Step 2) determine distortion model based on the lens curvature of camera, and calibrate camera internal parameter using calibration instrument, wherein the camera internal parameter includes distortion coefficient and projection matrix;

[0009] Step 3) synchronize the time stamp of laser radar and camera based on synchronization signal;

[0010] Step 4) during the movement of joint calibration platform, according to the rotation direction and angle of laser inside laser radar, the original point cloud obtained by laser radar is endowed with microsecond level time stamp, and motion distortion is removed, to obtain preprocessed point cloud;

[0011] Step 5) obtain the image taken by camera during the movement of joint calibration platform and convert it into gray image;

[0012] Step 6) initialize the external parameter of laser radar and camera, wherein the external parameter represents the relative pose relationship of laser radar and camera;

[0013] Step 7) convert the preprocessed point cloud to camera coordinate system based on the initialized external parameter to obtain converted point cloud;

[0014] Step 8) based on random mask method, cover converted point cloud and gray image respectively according to preconfigured covering ratio to obtain covered point cloud and covered gray image;

[0015] Step 9) according to distortion coefficient and projection matrix, project covered point cloud to image plane to obtain depth information projection and reflectivity information projection;

[0016] Step 10) construct end-to-end laser radar and camera external parameter online calibration network, the external parameter online calibration network is composed of input branch, prediction branch, fusion branch and result output module,

[0017] The input branch includes depth information branch, depth estimation branch, reflectivity information branch and image feature branch, wherein the input of the depth information branch is the depth information projection, and the output is the feature extracted from the depth information projection, the input of the depth estimation branch is the depth map obtained by depth estimation network based on depth estimation on covered gray image, and the output is the feature extracted from the depth map, the input of the reflectivity information branch is the reflectivity information projection, and the output is the cascaded result of the feature extracted from the reflectivity information projection and the output of the depth information branch with channel attention, the input of the image feature branch is covered gray image, and the output is the feature extracted from covered gray image;

[0018] The outputs of the depth information branch and the depth estimation branch jointly constitute a first prediction branch, and an output of the first prediction branch is a six-degree-of-freedom external parameter,

[0019] The outputs of the reflectivity information branch and the image feature branch jointly constitute a second prediction branch, and an output of the second prediction branch is a six-degree-of-freedom external parameter,

[0020] The outputs of the depth information branch, the depth estimation branch, the reflectivity information branch and the image feature branch jointly constitute a fusion prediction branch, and an output of the fusion prediction branch is a six-degree-of-freedom external parameter,

[0021] The result output module supervises the external parameter estimation of the three branches based on a loss weight, and takes the external parameter estimation result of the fusion prediction branch as a final external parameter calibration result.

[0022] Step 11) training the external parameter online calibration network on a cloud computing resource;

[0023] Step 12) completing online calibration of the external parameters of the laser radar and the camera based on the trained external parameter online calibration network.

[0024] The position of the laser radar and the camera in the joint calibration platform is determined based on an overlapping area of the sensing ranges of the two.

[0025] The distortion model includes a Pinhole model, a Kannala-Brandt model, an MEI model and a Scaramuzza model.

[0026] The synchronization signal includes a synchronization line and a GPS.

[0027] The depth estimation network is a monocular depth estimation network or a binocular depth estimation network.

[0028] The six-degree-of-freedom external parameter includes three translational parameters and three rotational parameters.

[0029] The depth information branch and the reflectivity information branch do not share weights.

[0030] The loss weight of the fusion prediction branch is greater than the loss weight of the depth prediction branch and the loss weight of the reflectivity prediction branch.

[0031] A laser radar and camera external parameter online calibration device based on a deep neural network, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the method as described above when executing the program.

[0032] A storage medium having a program stored thereon, wherein the program is executed to implement the method as described above.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] (1) The present application constructs an online calibration network of the extrinsic parameters of the end-to-end laser radar and camera, fuses the depth information and reflectivity information based on a feature-level information fusion method, realizes the extrinsic parameter calibration, is suitable for more complex calibration scenes, and improves the calibration performance.

[0035] (2) The present application randomly covers the point cloud and the image in a random mask manner, realizes data enhancement, greatly enhances the data diversity without increasing the acquisition cost, and effectively overcomes the problem of data overfitting. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The method flowchart of the present application is shown in the figure.

[0037] Figure 2 The structure diagram of the online calibration network of the extrinsic parameters of the end-to-end laser radar and camera of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The present application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the premise of the technical scheme of the present application, and gives a detailed implementation manner and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0039] A laser radar and camera extrinsic parameter online calibration method based on a deep neural network, as shown in the figure, comprises the following steps: Figure 1

[0040] Step 1) Build a joint calibration platform for laser radar and camera.

[0041] The position setting of the laser radar and the camera in the joint calibration platform is determined based on the overlapping area of the sensing ranges of the two, and it should be ensured that the overlapping area is large.

[0042] Step 2) Determine a distortion model based on the lens curvature of the camera, and calibrate the camera intrinsic parameters (such as high-precision checkerboard target) using a high-precision calibration instrument, wherein the camera intrinsic parameters include distortion coefficients and a projection matrix.

[0043] The distortion model includes Pinhole model, Kannala-Brandt model, MEI model, and Scaramuzza model.

[0044] Step 3) Synchronize the timestamps of the laser radar and the camera based on a synchronous signal such as a synchronous line or GPS.

[0045] ​Step 4) During the motion of the joint calibration platform, the original point cloud obtained by the laser radar is given a microsecond-level time stamp according to the rotation direction and angle of the laser inside the laser radar, and motion distortion is removed to obtain a preprocessed point cloud.

[0046] Step 5) An image captured by the camera during the motion of the joint calibration platform is obtained and converted into a gray image.

[0047] Step 6) The extrinsic parameters of the laser radar and the camera are initialized, wherein the extrinsic parameters represent the relative pose relationship between the laser radar and the camera.

[0048] Step 7) The preprocessed point cloud is converted to the camera coordinate system based on the initialized extrinsic parameters to obtain a converted point cloud.

[0049] Step 8) Based on the random mask method, the converted point cloud and the gray image are respectively covered according to a preconfigured covering ratio to obtain a covered point cloud and a covered gray image.

[0050] The random mask can achieve the purpose of data augmentation, thereby improving the robustness of the model and effectively expanding the application scenarios of the algorithm.

[0051] Step 9) The covered point cloud is projected to the image plane according to the distortion coefficient and the projection matrix to obtain depth information projection and reflectivity information projection.

[0052] Step 10) An end-to-end extrinsic parameter online calibration network for the laser radar and the camera is constructed, as shown in Figure 2 .

[0053] The extrinsic parameter online calibration network is composed of an input branch, a prediction branch, a fusion branch, and a result output module,

[0054] The input branch includes a depth information branch, a depth estimation branch, a reflectivity information branch, and an image feature branch. Since the depth and reflectivity representations are different, the depth information branch and the reflectivity information branch do not share weights.

[0055] The input of the depth information branch is the depth information projection, and the output is the extracted features of the depth information projection.

[0056] The input of the depth estimation branch is a depth map obtained by performing depth estimation on the covered gray image based on a depth estimation network, and the depth estimation network is a monocular depth estimation network or a binocular depth estimation network, which is specifically determined according to the applicable scene and actual demand. The output is the extracted features of the depth map. This depth estimation process specifically belongs to the conventional settings in the field, and in order to avoid obscuring the purpose of the present application, it will not be described here.

[0057] The input of the reflectivity information branch is a reflectivity information projection, and the output is a concatenated result of the extracted features of the reflectivity information projection and the output of the depth information branch to which channel attention is given. The purpose of the reflectivity information branch concatenating the depth information branch is to enable the reflectivity information branch to learn the depth information, reduce the influence of the depth information on the reflectivity attenuation, and enable the reflectivity branch to retain as much texture information as possible.

[0058] The input of the image feature branch is the covered gray image, and the output is the extracted features of the covered gray image.

[0059] The outputs of the depth information branch and the depth estimation branch jointly constitute a first prediction branch, and the output of the first prediction branch is a six-degree-of-freedom external parameter.

[0060] The outputs of the reflectivity information branch and the image feature branch jointly constitute a second prediction branch, and the output of the second prediction branch is a six-degree-of-freedom external parameter.

[0061] The outputs of the depth information branch, the depth estimation branch, the reflectivity information branch, and the image feature branch jointly constitute a fusion prediction branch, and the output of the fusion prediction branch is a six-degree-of-freedom external parameter.

[0062] The six-degree-of-freedom external parameter includes three translational parameters and three rotational parameters.

[0063] The result output module supervises the external parameter estimation of the three branches based on a loss weight, and takes the external parameter estimation result of the fusion prediction branch as the final external parameter calibration result. The loss weight of the fusion prediction branch is greater than the loss weight of the depth prediction branch and the loss weight of the reflectivity prediction branch.

[0064] Step 11) training the external parameter online calibration network on a cloud computing resource;

[0065] Step 12) completing online calibration of the external parameters of the laser radar and the camera based on the trained external parameter online calibration network.

[0066] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0067] In this embodiment, the above-mentioned online calibration method of external parameters is verified by using a 64-line laser radar and a Pinhole camera imaging model camera. The rotation parameters of the initial external parameters are all set to 0, the translation parameters are respectively set to tx=-40mm, ty=-750mm, and tz=2700mm, the resolution of the input image is uniformly adjusted to 640x480 pixels, and the Adam optimizer is used for training. The random mask covering ratio is 30%, which can increase the data to 30 times of the original data. The number of iterations for training is set to 100, the initial learning rate is 0.001, and the learning rate decay coefficient is set to 0.9. The augmented data is divided into a ratio of 4:3:3 for training, verification, and testing. In this embodiment, the performance of the above-mentioned method is evaluated by using the reprojection error, and the evaluation result is that the proposed method can obtain an average reprojection error within 2 pixels, which meets the needs of most three-dimensional reconstruction application scenarios and has good calibration effect.

[0068] The preferred embodiments of the present application are described in detail above. It should be understood that those skilled in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning, or limited experiments based on the prior art according to the concept of the present application should be within the scope of protection determined by the claims.

Claims

1. A method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks, characterized in that, Includes the following steps: Step 1) Build a joint calibration platform for lidar and camera; Step 2) Determine the distortion model based on the lens curvature of the camera and calibrate the camera intrinsic parameters using a calibration instrument. The camera intrinsic parameters include distortion coefficients and projection matrix. Step 3) Synchronize the timestamps of the lidar and camera based on the synchronization signal; Step 4) During the joint calibration platform movement, according to the rotation direction and angle of the laser inside the lidar, the original point cloud acquired by the lidar is given a microsecond-level timestamp and motion distortion is removed to obtain a preprocessed point cloud. Step 5) Acquire images captured by the camera during the joint calibration platform's movement and convert them into grayscale images; Step 6) Initialize the extrinsic parameters of the lidar and camera, wherein the extrinsic parameters represent the relative pose relationship between the lidar and the camera; Step 7) Based on the initialized extrinsic parameters, transform the preprocessed point cloud to the camera coordinate system to obtain the transformed point cloud; Step 8) Based on the random masking method, mask the converted point cloud and grayscale image according to the pre-configured masking ratio to obtain the masked point cloud and masked grayscale image. Step 9) Based on the distortion coefficients and projection matrix, project the occluded point cloud onto the image plane to obtain the depth information projection and reflectance information projection. Step 10) Construct an end-to-end online extrinsic parameter calibration network for the LiDAR and camera. This online extrinsic parameter calibration network consists of an input branch, a prediction branch, a fusion branch, and a result output module. The input branches include a depth information branch, a depth estimation branch, a reflectance information branch, and an image feature branch. The input of the depth information branch is a depth information projection, and the output is the extracted features of the depth information projection. The input of the depth estimation branch is a depth map obtained by depth estimation of the occluded grayscale image based on the depth estimation network, and the output is the extracted features of the depth map. The input of the reflectance information branch is a reflectance information projection, and the output is the concatenation result of the extracted features of the reflectance information projection and the output of the depth information branch with channel attention. The input of the image feature branch is the occluded grayscale image, and the output is the extracted features of the occluded grayscale image. The outputs of the depth information branch and the depth estimation branch together constitute the first prediction branch, and the output of the first prediction branch is a six-degree-of-freedom extrinsic parameter. The outputs of the reflectance information branch and the image feature branch together constitute the second prediction branch, and the output of the second prediction branch is a six-degree-of-freedom extrinsic parameter. The outputs of the depth information branch, depth estimation branch, reflectivity information branch, and image feature branch together constitute the fusion prediction branch. The output of the fusion prediction branch is a six-degree-of-freedom extrinsic parameter. The result output module estimates the extrinsic parameters of the three branches based on the loss weight supervision, and uses the extrinsic parameter estimation result of the fusion prediction branch as the final extrinsic parameter calibration result. Step 11) Train the extrinsic parameter online calibration network on cloud computing resources; Step 12) Perform online calibration of the extrinsic parameters of the LiDAR and camera based on the trained online extrinsic parameter calibration network.

2. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The positions of the lidar and camera in the joint calibration platform are determined based on the area of ​​the overlapping region of their sensing ranges.

3. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The distortion models include the Pinhole model, the Kannala-Brandt model, the MEI model, and the Scaramuzza model.

4. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The synchronization signal includes a synchronization line and GPS.

5. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The depth estimation network is either a monocular depth estimation network or a binocular depth estimation network.

6. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The six-degree-of-freedom extrinsic parameters include three translation parameters and three rotation parameters.

7. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The depth information branch and the reflectivity information branch do not share weights.

8. The method for online calibration of extrinsic parameters of lidar and camera based on deep neural networks according to claim 1, characterized in that, The loss weight of the fusion prediction branch is greater than the loss weight of the depth prediction branch and the loss weight of the reflectance prediction branch.

9. An online calibration device for extrinsic parameters of a lidar and camera based on a deep neural network, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images