A method for calibrating extrinsic parameters of laser radar and camera

Abstract

The present invention relates to the field of image processing technology, and in particular to a method for calibrating laser radar and camera extrinsic parameters, comprising the following steps: determining whether the density of a laser point cloud meets the requirements; generating a laser intensity image, a laser depth image, and a laser index image, and performing image enhancement processing; processing a visual image into a visual intensity image and generating a visual depth image; estimating depth corresponding point pairs and intensity corresponding point pairs; pruning corresponding point pairs and optimizing extrinsic parameters; deleting laser point clouds that are not visible under the camera's viewing angle, generating a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image; optimizing and solving extrinsic parameters from coarse to fine to obtain the final optimized extrinsic parameters. The method provided by the present invention can obtain high-quality laser intensity images and laser depth images by combining data accumulation evaluation criteria, image enhancement schemes, and coarse to fine laser radar image correction strategies.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for calibrating extrinsic parameters of a laser radar and a camera. Background Art

[0002] LiDAR and vision cameras are widely used environment perception sensors in smart vehicles and various robotic systems. LiDAR can measure distance but cannot perceive texture and color, while vision cameras can capture rich texture but lack depth. Data fusion of LiDAR and vision cameras promotes and enhances various robotic applications, such as simultaneous localization and mapping, 3D object detection and tracking, semantic segmentation, bird's-eye view based perception, etc. Extrinsic parameter calibration is a crucial prerequisite for effective data fusion, which aims to accurately estimate the six-degree-of-freedom rigid body transformation between the LiDAR and camera coordinate systems.

[0003] So far, the calibration methods between LiDAR and visual cameras have been widely studied and continuously improved in terms of accuracy and convenience. However, most of the existing robust and well-known methods are target-based, requiring the customization of high-precision and carefully designed calibration boards, targets or rooms, which is very time-consuming and costly. In contrast, target-free methods are very convenient, efficient and low-cost, but face challenges in robustness, accuracy and applicability to various scenarios due to the lack of strong constraints provided by boards, targets or rooms. In addition, with the rapid development of deep learning technology, end-to-end calibration methods have shown amazing results on specific datasets. However, the generalization performance of the above methods is poor, and it is difficult to directly apply them to different sensor combinations and scenarios that the model has not yet learned.

[0004] In general, target-free methods remain the most practical and attractive research topic due to their convenience and low cost. Such methods are mainly divided into motion-based and feature-based methods. Motion-based methods are robust to various scenes, but require sufficient motion excitation, and errors and time offsets in sensor self-motion estimation limit the calibration accuracy. Feature-based methods rely on cross-modal estimation of geometric features such as 3D-2D corresponding lines, edges, surfaces or points, or similarity metrics based on statistical features, such as mutual information, normalized information distance. Edge-, line-, and surface-based methods have poor scene adaptability due to their reliance on significant and stable geometric features. Since cross-modal edge matching is challenging, the extrinsic parameters can only converge when the optimized initial values ​​are close to the ground truth. Mutual information-based methods are easily affected by lighting and object material, and rely on good initial guesses to converge due to the locally convex cost function. In contrast, point-based methods are more robust to scenes due to the richness, universality, and ease of detection of point features. In addition, advanced image point feature extraction and matching networks have attracted widespread attention due to their strong generalization capabilities in various complex scenes, which has promoted the research on 3D-2D corresponding point estimation in image space, but there is a lack of a complete high-precision laser camera calibration framework based on 3D-3D corresponding point pair estimation.

[0005] In summary, most of the existing LiDAR and visual camera calibration methods rely on customized artificial targets, while the calibration methods without artificial targets are not accurate and robust enough. How to accurately and robustly complete the extrinsic parameter calibration between LiDAR and visual cameras anytime and anywhere without artificial targets is a research topic with practical value. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for calibrating the extrinsic parameters of a laser radar and a camera, which only requires a single data collection, and ensures that a sufficiently dense laser point cloud is obtained through data accumulation and evaluation, and generates high-quality laser intensity images, laser depth images, visual intensity images and visual depth images, estimates corresponding point pairs, and finally optimizes the extrinsic parameters from coarse to fine by gradually pruning corresponding point pairs and minimizing reprojection errors. After each iteration, the laser intensity image and laser depth image are corrected by removing the laser point cloud that is invisible to the camera viewpoint, thereby obtaining a laser radar image that adapts to various fields of view.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for calibrating laser radar and camera extrinsic parameters comprises the following steps:

[0009] S1: Use the data accumulation evaluation criteria to judge the density of the laser point cloud collected by the laser radar in a single time until the density of the laser point cloud meets the requirements;

[0010] S2: Projecting the laser point cloud into the image to generate a laser intensity image, a laser depth image and a laser index image, and performing image enhancement processing on the laser intensity image, the laser depth image and the laser index image to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image;

[0011] S3: Process the visual image captured by the camera into a visual intensity image, and generate a visual depth image through a monocular depth estimation network;

[0012] S4: An image matching network is used to cross-modally estimate the 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and then the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image;

[0013] S5: gradually pruning the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimizing the reprojection error to optimize the extrinsic parameters;

[0014] S6: using the optimized extrinsic parameters to delete the laser point cloud that is invisible in the camera view in the original laser point cloud, projecting the retained laser point cloud, and generating a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image;

[0015] S7: Set the number of iterations, iteratively execute step S2 to perform image enhancement processing on the laser intensity image, laser depth image and laser index image, as well as steps S4, S5 and S6, optimize and solve the external parameters from coarse to fine, and obtain the final optimized external parameters.

[0016] Furthermore, the data accumulation evaluation criterion in step S1 is formula (1):

[0017] (1);

[0018] in: is the horizontal field of view of the laser radar, is the number of point clouds accumulated in a sphere with near-field radius, To evaluate the threshold, is the near field radius.

[0019] Furthermore, in step S1, for a non-repeated scanning laser radar, a dense laser point cloud is obtained by keeping still and accumulating each frame of laser point cloud. For a repetitive scanning laser radar, a dense laser point cloud is obtained by the following method:

[0020] According to formula (2), the laser radar translation distance between two adjacent frames of laser point cloud and the laser radar rotation angle between two adjacent frames of laser point cloud are calculated, and the translation distance and rotation angle are compared with the corresponding thresholds respectively. If the translation distance or rotation angle is greater than the corresponding threshold, the frame of laser point cloud is accumulated into the coordinate system of the first scan, otherwise the frame of laser point cloud is discarded:

[0021] (2);

[0022] in: Indicates the translation distance of the laser radar between two adjacent frames of laser point cloud. Indicates the rotation angle of the laser radar between two adjacent frames of laser point cloud. represents the translation increment, represents the rotation increment, represents the trace of the matrix, represents the two-norm.

[0023] Furthermore, in step S2, the method of projecting the laser point cloud into the image to generate the laser intensity image, the laser depth image and the laser index image is as follows:

[0024] S21: Projecting the laser point cloud into the image through the virtual camera. The posture of the virtual camera during the first projection Formula (3):

[0025] (3);

[0026] S22: updating the posture of the virtual camera to the rotation matrix of the current extrinsic parameters in each iteration optimization of the extrinsic parameters;

[0027] S23: Assign the reflection intensity of the laser point cloud to the projected pixel value to generate a laser intensity image, assign the depth information of the laser point cloud to the projected pixel value to generate a laser depth image, assign the index corresponding to the laser point cloud to the projected pixel value to obtain a laser index image.

[0028] Further, in step S2, the laser intensity image, the laser depth image and the laser index image are enhanced by the following method to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image:

[0029] S24: multi-level expansion kernel filling: first extract the invalid pixel area mask of the laser intensity image or the laser depth image, then fill the laser intensity image and the laser depth image with a rectangular expansion kernel, copy the pixels corresponding to the mask positions of the invalid pixel area to the laser intensity image or the laser depth image before expansion, and keep the pixel values ​​of the valid pixel area unchanged;

[0030] S25: Noise elimination and contrast enhancement: The global histogram equalization method is used to enhance the contrast of the laser intensity image and highlight the image details, and the median filtering method is used to remove noise while retaining edge details to obtain an enhanced laser intensity image. The Gaussian filtering method is used to smooth the laser depth image, and the gamma correction method is used to enhance the average brightness of the laser depth image to obtain an enhanced laser depth image.

[0031] S26: Index image enhancement: find invalid pixels in the laser index image, and search from near to far in a neighborhood with a radius of set pixels with the invalid pixel as the center. If a valid pixel is found, the index of the invalid pixel is filled with the index of the valid pixel. If no valid pixel is found, the invalid pixel remains an invalid pixel, thereby obtaining an enhanced laser index image.

[0032] Furthermore, in step S3, the visual image captured by the camera is processed into a visual intensity image by using the following method: the visual image is first grayscaled, and then the grayscaled visual image is histogram equalized to obtain a visual intensity image.

[0033] Further, the process of obtaining the 3D-to-2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D-to-2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image in step S4 is as follows:

[0034] S41: firstly, a semi-dense matching network is used for matching, and 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are estimated iteratively across modalities, and then 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image, and the corresponding 3D to 2D corresponding point pairs are respectively saved to the corresponding depth corresponding point pair set of the enhanced laser depth image and the visual depth image and the intensity corresponding point pair set of the enhanced laser intensity image and the visual intensity image;

[0035] S42: In the last iteration, matching is first performed using a semi-dense matching network, and then using a two-stage matching network, and the outputs of the two matches are accumulated into a set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image and a set of intensity corresponding point pairs for matching the enhanced laser intensity image with the visual intensity image.

[0036] Furthermore, in step S5, the following method is used to gradually prune the 3D-to-2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D-to-2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimize the reprojection error to optimize the extrinsic parameters:

[0037] S51: using a RANSAC-based algorithm to estimate initial extrinsic parameters from a set of depth corresponding point pairs matched between the enhanced laser depth image and the visual depth image and a set of intensity corresponding point pairs matched between the enhanced laser intensity image and the visual intensity image;

[0038] S52: Optimize the external parameters by minimizing the reprojection error (4):

[0039] (4);

[0040] in: represents the set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image, represents the set of intensity corresponding point pairs that match the enhanced laser intensity image with the visual intensity image, Represents a set of depth corresponding point pairs The weights of the depth corresponding point pairs in , represents the Cauchy robust kernel function, represents the optimized external rotation parameters, Represents a set of depth corresponding point pairs The Laser point cloud, , Represents a set of depth corresponding point pairs The total number of laser point clouds in Represents the set of intensity corresponding point pairs The Laser point cloud, , Represents the set of intensity corresponding point pairs The total number of laser point clouds, represents the optimized external translation parameters, Represents a set of depth corresponding point pairs The Visual pixels, , Represents the set of intensity corresponding point pairs The intensities in correspond to the weights of the point pairs, Represents the set of intensity corresponding point pairs Middle Visual pixels, , represents the square of the two-norm;

[0041] S53: Traverse the depth corresponding point pair set The set of all depth corresponding point pairs and intensity corresponding point pairs in All the intensity corresponding point pairs in , the depth corresponding point pairs and the intensity corresponding point pairs are pruned according to formula (5), and the depth corresponding point pairs and the intensity corresponding points that satisfy formula (5) are deleted;

[0042] (5);

[0043] in: Indicates The external rotation parameters after the first progressive optimization are: Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The laser point cloud in Indicates The external translation parameters after the first progressive optimization, Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The visual pixels in Represents the reprojection error threshold for each pruning calculation, represents the set threshold of the maximum reprojection error, Indicates the number of progressive optimizations;

[0044] S54: optimizing the external parameters by minimizing the reprojection error equation (4) using the pruned depth corresponding point pairs and intensity corresponding point pairs;

[0045] S55: Execute steps S53 and S54 in a loop until the set number of progressive optimization times is reached.

[0046] Further, in step S6, the laser point cloud that is invisible from the camera perspective in the original laser point cloud is deleted by using the optimized extrinsic parameters, and the retained laser point cloud is projected to generate the corrected laser intensity image, the corrected laser depth image, and the corrected laser index image as follows:

[0047] S61: Use the currently optimized extrinsic parameters to transform all laser point clouds into the camera coordinate system according to equation (6):

[0048] (6);

[0049] in: For the Laser point cloud Representation in camera coordinate system;

[0050] S62: Delete the laser point cloud that meets one of the following conditions: delete the laser point cloud outside the field of view; delete the laser point cloud whose pixel coordinates exceed the camera visual image range after projection; delete the far-field laser point cloud and the laser point cloud that does not meet the depth test, project the retained laser point cloud, and generate a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image.

[0051] Furthermore, according to formula (7), the laser point cloud outside the field of view is deleted, and according to formula (8), the laser point cloud whose pixel coordinates after projection are beyond the range of the camera visual image is deleted:

[0052] (7);

[0053] (8);

[0054] in: Indicates Laser point cloud of coordinate, Indicates the maximum angle between the line connecting the laser point cloud in the camera field of view and the camera optical center and the camera optical axis. represents the two-norm, Indicates Pixels after laser point cloud projection, represents the height of the camera visual image, Indicates the width of the camera's visual image.

[0055] Beneficial effects of the invention:

[0056] 1. The present invention provides a laser radar and camera extrinsic parameter calibration method, which uses the designed data accumulation evaluation criteria to determine whether the density of the laser point cloud collected by the laser radar in a single time meets the requirements, so that the laser radar and camera extrinsic parameter calibration can be performed with only a single data collection.

[0057] 2. Ensure a sufficiently dense laser point cloud through data accumulation and evaluation, and strive to estimate the extrinsic parameters between the lidar and the visual camera from a dense point cloud and a frame of visual image. The dense point cloud is projected and image enhanced to obtain a high-quality laser point cloud image. The visual image is grayed into an intensity image and the corresponding visual depth image is generated through a monocular depth estimation network, so as to estimate the corresponding point pairs of the 3D laser point cloud and the 2D visual image in the 2D visual image space. Finally, the extrinsic parameters are optimized from coarse to fine by gradually pruning the point pairs and minimizing the reprojection error. After each iteration, the laser intensity image and laser depth image are corrected by deleting the laser point cloud that is invisible from the camera viewpoint, so as to obtain a lidar image that can adapt to various fields of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0059] A method for calibrating the external parameters of a laser radar and a camera, the flow chart of which is as follows: Figure 1 As shown, the specific steps include:

[0060] S1: Use the data accumulation evaluation criteria to determine whether the density of the laser point cloud collected by the laser radar in a single time meets the requirements. If it does, proceed to the next step. If not, continue to collect until the density of the laser point cloud collected by the laser radar in a single time meets the requirements.

[0061] The data accumulation evaluation criterion is used to evaluate whether the collected laser point cloud is dense enough. For feature matching-based calibration, the external translation is difficult to estimate without many near-field objects. This is because as the distance increases, the projection of the laser point cloud is gradually unaffected by the translation component of the projection matrix. Therefore, the data accumulation evaluation needs to consider the density of the near-field laser point cloud. In addition, in order to accommodate lidars with various visual fields, the data accumulation evaluation should include the field of view of the lidar.

[0062] Specifically, the designed data accumulation evaluation criterion is formula (1):

[0063] (1);

[0064] in: is the horizontal field of view of the laser radar, is the number of point clouds accumulated in a sphere with near-field radius, To evaluate the threshold, is the near field radius.

[0065] The designed data accumulation evaluation criteria are used to judge whether the density of the laser point cloud collected by the lidar in a single time meets the requirements, which can ensure that a dense point cloud is obtained when collecting data in a single time.

[0066] Specifically, for a non-repetitive scanning LiDAR, a dense laser point cloud can be obtained by keeping still and accumulating the point cloud of each frame.

[0067] For repeated scanning LiDAR, multiple frames of laser point clouds can be dynamically accumulated to obtain dense laser point clouds. In order to avoid the accumulation of invalid data and overlap and artifacts caused by the repetitiveness of continuous scanning, the odometer attitude increment between two frames of laser point clouds can be used. , to determine whether the laser point cloud for this frame should be accumulated.

[0068] According to formula (2), the laser radar translation distance between two adjacent frames of laser point cloud and the laser radar rotation angle between two adjacent frames of laser point cloud are calculated, and the translation distance and rotation angle are compared with the corresponding thresholds respectively. If the translation distance or rotation angle is greater than the corresponding threshold, the frame of laser point cloud is accumulated into the coordinate system of the first scan, otherwise the frame of laser point cloud is discarded:

[0069] (2);

[0070] in: Indicates the translation distance of the laser radar between two adjacent frames of laser point cloud. Indicates the rotation angle of the laser radar between two adjacent frames of laser point cloud. Represents the rigid body transformation group in three-dimensional space, including the rotation group and translation group, represents the translation increment, , represents the translation group in three-dimensional space, represents the rotation increment, , represents the rotation group in three-dimensional space, represents the trace of the matrix, represents the two-norm, It indicates that the odometer attitude increment consists of two parts: rotation increment and translation increment.

[0071] S2: Projecting the laser point cloud into the image to generate a laser intensity image, a laser depth image and a laser index image, and performing image enhancement processing on the laser intensity image, the laser depth image and the laser index image to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image;

[0072] Specifically, the method of projecting the laser point cloud into the image to generate the laser intensity image, the laser depth image and the laser index image is as follows:

[0073] S21: Projecting the laser point cloud into the image through the virtual camera. The posture of the virtual camera during the first projection Formula (3):

[0074] (3);

[0075] S22: As the initial value of the iteration, the posture of the virtual camera is updated to the rotation matrix of the current extrinsic parameters in each iterative optimization of the extrinsic parameters, which can make the posture of the virtual camera closer to the real visual camera.

[0076] S23: Assign the reflection intensity of the laser point cloud to the projected pixel value to generate a laser intensity image, assign the depth information of the laser point cloud to the projected pixel value to generate a laser depth image, assign the index corresponding to the laser point cloud to the projected pixel value to obtain a laser index image.

[0077] When projecting a laser point cloud, an index map is created to store the position of the 3D laser radar point corresponding to the pixel in the original point cloud. Although a dense laser point cloud is collected, the density of the laser point cloud is not uniform, and there are some holes in the projected laser radar image, which will affect the image clarity and subsequent feature extraction. Therefore, it is necessary to use an image enhancement solution to improve the quality of the laser intensity image and the laser depth image, so that a high-quality projected image can be obtained using only a small amount of raw data.

[0078] Specifically, the following method can be used to perform image enhancement processing on the laser intensity image, the laser depth image and the laser index image to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image:

[0079] S24: multi-level expansion kernel filling: first extract the invalid pixel area mask of the laser intensity image or the laser depth image, then fill the laser intensity image and the laser depth image with a rectangular expansion kernel, copy the pixels corresponding to the mask positions of the invalid pixel area to the laser intensity image or the laser depth image before expansion, and keep the pixel values ​​of the valid pixel area unchanged;

[0080] The same dilation kernel is used for both laser intensity and depth images. Rectangular dilation kernels are very suitable for filling holes and connecting the edges of objects.

[0081] It is worth noting that each dilation first extracts the invalid pixel area mask, and after dilation, the pixels corresponding to the mask position are copied to the laser intensity image or laser depth image before dilation. In this process, the pixel values ​​of the valid pixel area remain unchanged, thereby retaining the original details such as edges and corners.

[0082] S25: Noise elimination and contrast enhancement: The global histogram equalization method is used to enhance the contrast of the laser intensity image and highlight the image details, and the median filtering method is used to remove noise while retaining edge details to obtain an enhanced laser intensity image. The Gaussian filtering method is used to smooth the laser depth image, and the gamma correction method is used to enhance the average brightness of the laser depth image to obtain an enhanced laser depth image.

[0083] Since the laser intensity image contains rich texture details, global histogram equalization can enhance the contrast of the image and highlight the details of the image. Therefore, the laser intensity image is processed by global histogram equalization, and then the median filter is applied to remove noise while retaining the edge details.

[0084] For laser depth images, Gaussian filtering is used to smooth the images due to the lack of rich texture. During the data collection and calibration process, most points are near-field points, while some far-field points may reduce the overall brightness after depth value mapping, so gamma correction is applied to enhance the average brightness of laser depth images.

[0085] S26: Index image enhancement: find invalid pixels in the laser index image, and search from near to far in a neighborhood with a radius of set pixels with the invalid pixel as the center. If a valid pixel is found, the index of the invalid pixel is filled with the index of the valid pixel. If no valid pixel is found, the invalid pixel remains an invalid pixel, thereby obtaining an enhanced laser index image.

[0086] Since the previous two enhancement methods are only image-level enhancements to facilitate feature matching, the filled areas still have no corresponding lidar points. Therefore, a simple strategy is needed to enhance the index map. For invalid pixels in the index map, search from near to far in a neighborhood with a radius of a set pixel, with the invalid pixel as the center. If a valid pixel can be found, it is directly filled with the index of the valid pixel. If not found, it is still an invalid value. In this way, most indexes can be filled without introducing large errors.

[0087] Using the above-mentioned image enhancement scheme to process the LiDAR image can further improve the quality of the LiDAR image, which is beneficial to the subsequent cross-modal processing.

[0088] S3: Process the visual image captured by the camera into a visual intensity image, and generate a visual depth image through a monocular depth estimation network;

[0089] Specifically, the following method can be used to process the visual image captured by the camera into a visual intensity image: first, the visual image captured by the camera is grayed, and then the grayed visual image is histogram equalized to obtain a visual intensity image. The generated visual intensity image is used to match the laser intensity image, and the visual depth image is used to match the laser depth image.

[0090] S4: An image matching network is used to cross-modally estimate the 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and then the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image;

[0091] Specifically, the process of obtaining the 3D-to-2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D-to-2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image is as follows:

[0092] S41: First, a semi-dense matching network is used for matching, and the 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are estimated iteratively across modalities. Then, the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image, and the corresponding 3D to 2D corresponding point pairs are respectively saved to the corresponding depth corresponding point pair sets matched between the enhanced laser depth image and the visual depth image. And enhance the intensity corresponding point pair set of laser intensity image and visual intensity image matching ;

[0093] in: , Represents a set of depth corresponding point pairs The Laser point cloud, , Represents a set of depth corresponding point pairs The total number of laser point clouds, Represents a set of depth corresponding point pairs The Visual pixels, , , Represents the set of intensity corresponding point pairs The Laser point cloud, , Represents the set of intensity corresponding point pairs The total number of laser point clouds, Represents the set of intensity corresponding point pairs Middle Visual pixels, , represents the translation group in three-dimensional space, Represents two-dimensional real number space.

[0094] An image matching network is used to cross-modally estimate the corresponding point pairs between the enhanced laser depth image and the enhanced visual depth image, as well as the enhanced laser intensity image and the enhanced visual intensity image; the cross-modal estimation of 3D to 2D correspondence becomes a matching problem of 2D image feature points. Once the 2D to 2D correspondence is established, the 3D to 2D corresponding point pairs can be directly derived from the enhanced laser index image.

[0095] S42: In the last iteration, matching is first performed using a semi-dense matching network, and then using a two-stage matching network, and the outputs of the two matches are accumulated into a set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image and a set of intensity corresponding point pairs for matching the enhanced laser intensity image with the visual intensity image.

[0096] In order to improve the generalization performance of the network for scenes, a semi-dense matching network is first used for matching, and then a two-stage matching network is used for matching. The outputs of the two matching methods are accumulated into a set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image and a set of intensity corresponding point pairs for matching the enhanced laser intensity image with the visual intensity image. Specifically, in the first few iterations, since the semi-dense matching network performs well in matching weak texture areas, the semi-dense matching network is used to extract the correspondence between the laser intensity image and the visual intensity image, and between the laser depth image and the visual depth image, and form corresponding sets. Then in the last iteration, the two-stage matching network and the semi-dense matching network are used simultaneously for matching, and their outputs are accumulated into and In this way, more corresponding relationships can be mined, and the fluctuation of calibration results caused by network randomness can be weakened.

[0097] S5: gradually pruning the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimizing the reprojection error to optimize the extrinsic parameters;

[0098] Specifically, the following method can be used to gradually prune the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimize the reprojection error to optimize the extrinsic parameters:

[0099] S51: using a RANSAC-based algorithm to estimate initial extrinsic parameters from a set of depth corresponding point pairs matched between the enhanced laser depth image and the visual depth image and a set of intensity corresponding point pairs matched between the enhanced laser intensity image and the visual intensity image;

[0100] The RANSAC algorithm here is the abbreviation of Random Sample Consensus. It is an algorithm that calculates the mathematical model parameters of the data based on a set of sample data sets containing abnormal data to obtain valid sample data. It is an existing technology. The RANSAC-based algorithm is used to estimate the initial extrinsic parameters from the set of depth corresponding point pairs matched between the enhanced laser depth image and the visual depth image and the set of intensity corresponding point pairs matched between the enhanced laser intensity image and the visual intensity image. It can be used to estimate the initial extrinsic parameters from a set with many abnormal values. and Estimate the initial extrinsic rotation parameters in .

[0101] S52: Optimize the external parameters by minimizing the reprojection error (4):

[0102] (4);

[0103] in: Represents a set of depth corresponding point pairs The weights of the depth corresponding point pairs in , represents the Cauchy robust kernel function, represents the optimized external rotation parameters, represents the optimized external translation parameters, Represents the set of intensity corresponding point pairs The intensities in correspond to the weights of the point pairs, represents the square of the two-norm;

[0104] S53: Since the data of a single shot can be obtained There are even more corresponding point pairs, among which there are some mismatched or low-precision points. Therefore, it is necessary to gradually prune the point pairs through the following strategy to achieve progressive optimization from coarse to fine. The point pairs are not pruned in the first optimization because there is only an initial guess of the rotation, and its error is relatively large.

[0105] When pruning, traverse the set of depth corresponding point pairs The set of all depth corresponding point pairs and intensity corresponding point pairs in All the intensity corresponding point pairs in , the depth corresponding point pairs and the intensity corresponding point pairs are pruned according to formula (5), and the depth corresponding point pairs and the intensity corresponding points that satisfy formula (5) are deleted;

[0106] (5);

[0107] in: Indicates The external rotation parameters after the first progressive optimization are: Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The laser point cloud in Indicates The external translation parameters after the first progressive optimization, Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The visual pixels in Represents the reprojection error threshold for each pruning calculation, represents the set threshold of the maximum reprojection error, Indicates the number of progressive optimizations;

[0108] S54: optimizing the external parameters by minimizing the reprojection error equation (4) using the pruned depth corresponding point pairs and intensity corresponding point pairs;

[0109] S55: Execute steps S53 and S54 in a loop until the set number of progressive optimization times is reached.

[0110] S6: using the optimized extrinsic parameters to delete the laser point cloud that is invisible in the camera view in the original laser point cloud, projecting the retained laser point cloud, and generating a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image;

[0111] Specifically, the following method can be used to use optimized extrinsic parameters to delete the laser point cloud that is invisible in the camera's view in the original laser point cloud, project the retained laser point cloud, and generate a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image:

[0112] S61: Use the currently optimized extrinsic parameters to transform all laser point clouds into the camera coordinate system according to equation (6):

[0113] (6);

[0114] in: For the Laser point cloud The representation in the camera coordinate system is: Indicates the serial number of the laser point cloud;

[0115] S62: Delete the laser point cloud that meets one of the following conditions: delete the laser point cloud outside the field of view; delete the laser point cloud whose pixel coordinates exceed the camera visual image range after projection; delete the far-field laser point cloud and the laser point cloud that does not meet the depth test, project the retained laser point cloud, and generate a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image.

[0116] Since far-field points are sparse and do not contribute much to the extrinsic translation calibration, we can first remove points greater than half of the farthest lidar detection distance, and then, by building a depth buffer, only keep the lidar point with the minimum depth at each pixel.

[0117] Optimized, according to formula (7), delete the laser point cloud outside the field of view, and according to formula (8), delete the laser point cloud whose pixel coordinates exceed the camera visual image range after projection:

[0118] (7);

[0119] (8);

[0120] in: Indicates Laser point cloud of coordinate, Indicates the maximum angle between the line connecting the laser point cloud in the camera field of view and the camera optical center and the camera optical axis. represents the 2-norm, Indicates Pixels after laser point cloud projection, represents the height of the camera visual image, Indicates the width of the camera's visual image.

[0121] S7: Set the number of iterations, iteratively execute step S2 to perform image enhancement processing on the laser intensity image, laser depth image and laser index image, as well as steps S4, S5 and S6, optimize and solve the external parameters from coarse to fine, and obtain the final optimized external parameters.

[0122] In this iterative process, the laser intensity image and the laser depth image are continuously corrected, the virtual camera posture is constantly close to the visual camera, and the information expressed by the laser intensity image and the laser depth image is constantly close to the visual image. The extrinsic parameters are optimized from coarse to fine in this process. After the iteration is completed, the final optimized extrinsic parameters can be obtained.

[0123] In summary, the present invention provides a method for calibrating the extrinsic parameters of a laser radar and a camera, which estimates the corresponding point pairs between 3D and 2D in a 2D image space, and is applicable to laser radars of any scanning type and field of view. Calibration can be completed by collecting data only once, and a large number of 3D and 2D corresponding point pairs can be mined from a pair of dense point clouds and a visual image, achieving pixel-level calibration accuracy. By combining data accumulation evaluation criteria, image enhancement schemes, and coarse-to-fine laser radar image correction strategies, high-quality laser intensity images and laser depth images can be obtained.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A laser radar and camera extrinsic parameter calibration method, characterized in that: The steps include: S1: Use the data accumulation evaluation criteria to judge the density of the laser point cloud collected by the laser radar in a single time until the density of the laser point cloud meets the requirements; S2: Projecting the laser point cloud into the image to generate a laser intensity image, a laser depth image and a laser index image, and performing image enhancement processing on the laser intensity image, the laser depth image and the laser index image to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image; S3: Processing the visual image captured by the camera into a visual intensity image, and generating a visual depth image through a monocular depth estimation network; S4: An image matching network is used to cross-modally estimate the 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and then the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image, and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image; S5: gradually pruning the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimizing the reprojection error to optimize the extrinsic parameters; S6: using the optimized extrinsic parameters to delete the laser point cloud that is invisible in the camera view in the original laser point cloud, projecting the retained laser point cloud, and generating a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image; S7: Set the number of iterations, iteratively execute step S2 to perform image enhancement processing on the laser intensity image, laser depth image and laser index image, as well as steps S4, S5 and S6, optimize and solve the external parameters from coarse to fine, and obtain the final optimized external parameters.

2. The method for calibrating laser radar and camera extrinsic parameters according to claim 1, characterized in that: The data accumulation evaluation criterion in step S1 is formula (1): （1）； in: is the horizontal field of view of the laser radar, is the number of point clouds accumulated in a sphere with near-field radius, To evaluate the threshold, is the near field radius.

3. The method for calibrating laser radar and camera extrinsic parameters according to claim 1, characterized in that: In step S1, for a non-repeated scanning laser radar, a dense laser point cloud is obtained by keeping still and accumulating each frame of laser point cloud. For a repetitive scanning laser radar, a dense laser point cloud is obtained by the following method: According to formula (2), the laser radar translation distance between two adjacent frames of laser point cloud and the laser radar rotation angle between two adjacent frames of laser point cloud are calculated, and the translation distance and rotation angle are compared with the corresponding thresholds respectively. If the translation distance or rotation angle is greater than the corresponding threshold, the frame of laser point cloud is accumulated into the coordinate system of the first scan, otherwise the frame of laser point cloud is discarded: （2）； in: Indicates the translation distance of the laser radar between two adjacent frames of laser point cloud. Indicates the rotation angle of the laser radar between two adjacent frames of laser point cloud. represents the translation increment, represents the rotation increment, represents the trace of the matrix, represents the two-norm.

4. The method for calibrating laser radar and camera extrinsic parameters according to claim 1, characterized in that: In step S2, the laser point cloud is projected into the image to generate the laser intensity image, the laser depth image and the laser index image as follows: S21: Projecting the laser point cloud into the image through the virtual camera. The posture of the virtual camera during the first projection Formula (3): （3）； S22: updating the posture of the virtual camera to the rotation matrix of the current extrinsic parameters in each iteration optimization of the extrinsic parameters; S23: Assign the reflection intensity of the laser point cloud to the projected pixel value to generate a laser intensity image, assign the depth information of the laser point cloud to the projected pixel value to generate a laser depth image, assign the index corresponding to the laser point cloud to the projected pixel value to obtain a laser index image.

5. The method for calibrating laser radar and camera extrinsic parameters according to claim 4, characterized in that: In step S2, the laser intensity image, the laser depth image and the laser index image are enhanced by the following method to obtain an enhanced laser intensity image, an enhanced laser depth image and an enhanced laser index image: S24: multi-level expansion kernel filling: first extract the invalid pixel area mask of the laser intensity image or the laser depth image, then fill the laser intensity image and the laser depth image with a rectangular expansion kernel, copy the pixels corresponding to the mask positions of the invalid pixel area to the laser intensity image or the laser depth image before expansion, and keep the pixel values ​​of the valid pixel area unchanged; S25: Noise elimination and contrast enhancement: The global histogram equalization method is used to enhance the contrast of the laser intensity image and highlight the image details, and the median filtering method is used to remove noise while retaining edge details to obtain an enhanced laser intensity image. The Gaussian filtering method is used to smooth the laser depth image, and the gamma correction method is used to enhance the average brightness of the laser depth image to obtain an enhanced laser depth image. S26: Index image enhancement: find invalid pixels in the laser index image, and search from near to far in a neighborhood with a radius of set pixels with the invalid pixel as the center. If a valid pixel is found, the index of the invalid pixel is filled with the index of the valid pixel. If no valid pixel is found, the invalid pixel remains an invalid pixel, thereby obtaining an enhanced laser index image.

6. The method for calibrating laser radar and camera extrinsic parameters according to claim 1, characterized in that: In step S3, the visual image captured by the camera is processed into a visual intensity image by using the following method: firstly, the visual image is grayed, and then the grayed visual image is histogram equalized to obtain a visual intensity image.

7. The method for calibrating laser radar and camera extrinsic parameters according to claim 5, characterized in that: The process of obtaining the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image in step S4 is as follows: S41: firstly, a semi-dense matching network is used for matching, and 2D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and 2D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are estimated iteratively across modalities, and then 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image are obtained by enhancing the laser index image, and the corresponding 3D to 2D corresponding point pairs are respectively saved to the corresponding depth corresponding point pair set of the enhanced laser depth image and the visual depth image and the intensity corresponding point pair set of the enhanced laser intensity image and the visual intensity image; S42: In the last iteration, matching is first performed using a semi-dense matching network, and then using a two-stage matching network, and the outputs of the two matches are accumulated into a set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image and a set of intensity corresponding point pairs for matching the enhanced laser intensity image with the visual intensity image.

8. The method for calibrating laser radar and camera extrinsic parameters according to claim 7, characterized in that: In step S5, the following method is used to gradually prune the 3D to 2D corresponding point pairs between the enhanced laser depth image and the visual depth image and the 3D to 2D corresponding point pairs between the enhanced laser intensity image and the visual intensity image, and minimize the reprojection error to optimize the extrinsic parameters: S51: using a RANSAC-based algorithm to estimate initial extrinsic rotation parameters from a set of depth corresponding point pairs matched between the enhanced laser depth image and the visual depth image and a set of intensity corresponding point pairs matched between the enhanced laser intensity image and the visual intensity image; S52: Optimize the external parameters by minimizing the reprojection error (4): (4)； in: represents the set of depth corresponding point pairs for matching the enhanced laser depth image with the visual depth image, represents the set of intensity corresponding point pairs that match the enhanced laser intensity image with the visual intensity image, Represents a set of depth corresponding point pairs The weights of the depth corresponding point pairs in , represents the Cauchy robust kernel function, represents the optimized external rotation parameters, Represents a set of depth corresponding point pairs The Laser point cloud, , Represents a set of depth corresponding point pairs The total number of laser point clouds in Represents the set of intensity corresponding point pairs The Laser point cloud, , Represents the set of intensity corresponding point pairs The total number of laser point clouds in represents the optimized external translation parameters, Represents a set of depth corresponding point pairs The Visual pixels, , Represents the set of intensity corresponding point pairs The intensities in correspond to the weights of the point pairs, Represents the set of intensity corresponding point pairs Middle Visual pixels, , represents the square of the two-norm; S53: Traverse the depth corresponding point pair set The set of all depth corresponding point pairs and intensity corresponding point pairs in All the intensity corresponding point pairs in , the depth corresponding point pairs and the intensity corresponding point pairs are pruned according to formula (5), and the depth corresponding point pairs and the intensity corresponding points that satisfy formula (5) are deleted; (5)； in: Indicates The external rotation parameters after the first progressive optimization are: Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The laser point cloud in Indicates The external translation parameters after the first progressive optimization, Represents a set of depth corresponding point pairs or the set of intensity corresponding point pairs The visual pixels in Represents the reprojection error threshold for each pruning calculation, represents the set threshold of the maximum reprojection error, Indicates the number of progressive optimizations; S54: optimizing the external parameters by minimizing the reprojection error equation (4) using the pruned depth corresponding point pairs and intensity corresponding point pairs; S55: Execute steps S53 and S54 in a loop until the set number of progressive optimization times is reached.

9. The method for calibrating laser radar and camera extrinsic parameters according to claim 8, characterized in that: In step S6, the method of deleting the laser point cloud that is invisible in the camera view from the original laser point cloud by using the optimized extrinsic parameters, projecting the retained laser point cloud, and generating the corrected laser intensity image, the corrected laser depth image, and the corrected laser index image is as follows: S61: Use the currently optimized extrinsic parameters to transform all laser point clouds into the camera coordinate system according to equation (6): （6）； in: For the Laser point cloud Representation in camera coordinate system; S62: Delete the laser point cloud that meets one of the following conditions: delete the laser point cloud outside the field of view; delete the laser point cloud whose pixel coordinates exceed the camera visual image range after projection; delete the far-field laser point cloud and the laser point cloud that does not meet the depth test, project the retained laser point cloud, and generate a corrected laser intensity image, a corrected laser depth image, and a corrected laser index image.

10. The method for calibrating laser radar and camera extrinsic parameters according to claim 9, characterized in that: According to formula (7), the laser point cloud outside the field of view is deleted, and according to formula (8), the laser point cloud whose pixel coordinates after projection exceed the range of the camera visual image is deleted: （7）； （8）； in: Indicates Laser point cloud of coordinate, Indicates the maximum angle between the line connecting the laser point cloud in the camera field of view and the camera optical center and the camera optical axis. represents the two-norm, Indicates The pixels after the laser point cloud is projected, represents the height of the camera visual image, Indicates the width of the camera's visual image.

Images