A laser radar and camera external parameter calibration method, device and electronic equipment

Abstract

The application provides a laser radar and camera external parameter calibration method and device and electronic equipment, and relates to the technical field of multi-sensor fusion and computer vision. The method comprises the following steps: determining a high-confidence plane point set satisfying a preset geometric shape feature from laser radar point cloud data containing information of a checkerboard calibration board in a calibration scene according to a planarity index and a scatter index of a corresponding near-neighbor point set of each point in each candidate neighborhood scale; the calibration scene comprises a laser radar, a camera and a checkerboard calibration board; determining initial external parameter parameters between the laser radar and the camera according to the high-confidence plane point set; optimizing the initial external parameter parameters by applying a consistency optimization objective function to obtain an external parameter transformation matrix between the laser radar and the camera, and realizing laser radar and camera external parameter calibration.

Description

Technical Field

[0001] This application relates to the fields of multi-sensor fusion and computer vision technology, and more specifically, to a method, apparatus, and electronic device for calibrating the extrinsic parameters of a lidar and camera. Background Technology

[0002] LiDAR and cameras are two indispensable core sensing sensors in mobile robots, autonomous driving, and intelligent inspection systems. LiDAR, by emitting laser beams and measuring time-of-flight, can directly acquire high-precision three-dimensional spatial coordinate information of targets, unaffected by lighting conditions, but the generated point cloud data lacks texture and color information. Cameras, on the other hand, can capture rich texture, color, and shape information of targets, but cannot directly provide depth information and are susceptible to interference from environmental factors such as lighting and weather. To achieve complementary advantages and accurate fusion of data from these two types of sensors, it is essential to first obtain the accurate spatial extrinsic parameter transformation relationship between them, namely the rotation matrix and translation vector from the LiDAR coordinate system to the camera coordinate system. Therefore, high-precision and robust extrinsic parameter calibration technology is a key prerequisite technology for multi-sensor fusion sensing systems.

[0003] With the widespread application of non-repetitive scanning solid-state lidar, its non-uniform scanning characteristics have brought new challenges to existing calibration methods: the non-repetitive scanning mode results in a sparse and uneven spatial distribution of point clouds, and traditional geometric feature extraction methods with fixed neighborhood scales are difficult to take into account the feature stability of regions with different densities, which is prone to feature point omissions or false detections; traditional back-end optimization methods use equal weight strategies to process all valid points, which cannot reflect the reliability differences between different points and have poor convergence stability. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, and electronic device for calibrating the extrinsic parameters of a lidar and camera.

[0005] One aspect of this application provides a method for extrinsic parameter calibration of a lidar and a camera, comprising: determining a high-confidence planar point set satisfying preset geometric shape characteristics from the lidar point cloud data containing checkerboard calibration board information at each candidate neighborhood scale based on the planarity index and scatter index of the nearest neighbor point set corresponding to each point in the lidar point cloud data of the calibration scene; the calibration scene includes the lidar, the camera, and the checkerboard calibration board; determining initial extrinsic parameter parameters between the lidar and the camera based on the high-confidence planar point set; and optimizing the initial extrinsic parameter parameters by applying a consistency optimization objective function to obtain the extrinsic parameter transformation matrix between the lidar and the camera, thereby realizing the extrinsic parameter calibration of the lidar and the camera.

[0006] According to an embodiment of this application, based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scene at each candidate neighborhood scale, a high-confidence planar point set satisfying a preset geometric shape feature is determined from the aforementioned point cloud data. This includes: determining the optimal neighborhood scale of each point in the aforementioned point cloud data based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information at each candidate neighborhood scale; and determining, from the aforementioned point cloud data, points whose planarity and dispersion indices of the nearest neighbor set corresponding to the optimal neighborhood scale satisfy the preset geometric shape feature, thereby obtaining a high-confidence planar point set.

[0007] According to an embodiment of this application, the process of determining the planarity index and dispersion index of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the above calibration scenario includes: determining the covariance matrix of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the above calibration scenario at each candidate neighborhood scale; performing eigenvalue decomposition on the covariance matrix to obtain multiple non-negative eigenvalues; and determining the planarity index and dispersion index of the nearest neighbor set corresponding to each point in the point cloud data at each candidate neighborhood scale based on the non-negative eigenvalues.

[0008] According to an embodiment of this application, the optimal neighborhood scale of each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scenario is determined based on the planarity index and the dispersion index of the nearest neighbor set corresponding to each point at each candidate neighborhood scale. This includes: determining the function value of the information entropy evaluation function based on the planarity index and the dispersion index of the nearest neighbor set corresponding to each point at each candidate neighborhood scale; and determining the optimal neighborhood scale of each point in the point cloud data based on the function values ​​of multiple information entropy evaluation functions.

[0009] According to an embodiment of this application, the initial extrinsic parameters between the lidar and the camera are determined based on the correspondence between the calibration features of the target plane point cloud region corresponding to the checkerboard calibration board in the high-confidence plane point set and the calibration features of the camera image containing the checkerboard calibration board information in the calibration scenario.

[0010] According to an embodiment of this application, the above-mentioned consistency optimization objective function is: ; ; ; Where L is the consistency optimization objective function; N is the number of points in the point cloud data that participate in the optimization of the initial extrinsic parameters; For the validity constraint of the i-th point; Point-level reliability weights; The geometric consistency residual at the i-th point; For point-level feature vectors Point-level reliability weights The function; Let be the projected coordinates of the i-th point on the chessboard calibration board; Let be the laser reflection intensity corresponding to the i-th point; Let be the radial distance from the i-th point to the center of the checkerboard calibration board.

[0011] According to embodiments of this application, the function The training process of the multilayer perceptron network is as follows: acquiring point cloud data samples; determining high-reliability points in the point cloud data samples based on a first comparison result between the geometric consistency residuals of each point in the point cloud data samples and a preset high-reliability residual threshold; determining low-reliability points in the point cloud data samples based on a second comparison result between the geometric consistency residuals of each point in the point cloud data samples and a preset low-reliability residual threshold; determining the point-level feature vectors of the target sample points using the high-reliability points and the low-reliability points as target sample points; and training the multilayer perceptron network using the point-level feature vectors of the target sample points as input and the corresponding point-level reliability weight samples as output.

[0012] According to an embodiment of this application, the process of determining the target planar point cloud region corresponding to the chessboard calibration board in the high-confidence planar point set includes: using the point with the smallest curvature in the high-confidence planar point set as the initial seed point; determining the current growth region and seed point queue based on the initial seed point; and marking the initial seed point; determining the spatial nearest neighbor points of the initial seed point from the high-confidence planar point set, centered on the initial seed point, to obtain multiple initial neighbor points; determining unmarked initial nearest neighbor points from the multiple initial neighbor points to obtain at least one first target nearest neighbor point; using the first target nearest neighbor point whose angle between the normal vector of the first target nearest neighbor point and the normal vector of the initial seed point is less than or equal to a preset normal angle threshold as the second target nearest neighbor point; and obtaining at least one third target nearest neighbor point based on the second target nearest neighbor point whose local curvature at the optimal neighborhood scale is less than or equal to a preset curvature threshold; wherein, the normal vector of the first target nearest neighbor point is the eigenvector corresponding to the smallest non-negative eigenvalue of the first target nearest neighbor point; and adding the third target nearest neighbor point to the above... The current growth region and the aforementioned seed point queue are updated, and the aforementioned third target nearest neighbor points are marked. Using the aforementioned third target nearest neighbor points as initial seed points, the initial neighbor point determination operation, the first target nearest neighbor point determination operation, the third target nearest neighbor point determination operation, the current growth region and seed point queue update operation, and the third target nearest neighbor point marking operation are repeatedly performed until a preset first termination condition is met, resulting in a candidate planar region. If there are unmarked points in the aforementioned high-confidence planar point set, the unmarked points are used as the high-confidence planar point set, and the operations of determining the current growth region and seed point queue and marking the initial seed point, determining the initial neighbor points, determining the first target nearest neighbor points, determining the third target nearest neighbor points, updating the current growth region and seed point queue, marking the third target nearest neighbor points, and determining the candidate planar region are repeatedly performed until a preset second termination condition is met, resulting in multiple candidate planar regions. According to preset rules, the aforementioned target planar point cloud region is determined from the multiple aforementioned candidate planar regions.

[0013] Another aspect of this application provides a lidar and camera extrinsic parameter calibration device, comprising: a target point set determination module, configured to determine a high-confidence planar point set satisfying preset geometric shape characteristics from the lidar point cloud data containing checkerboard calibration board information, based on the planarity index and scattering index of the nearest neighbor point set corresponding to each point at each candidate neighborhood scale; the calibration scene includes a lidar, a camera, and a checkerboard calibration board; an initial extrinsic parameter determination module, configured to determine the initial extrinsic parameter between the lidar and the camera based on the high-confidence planar point set; and a transformation matrix determination module, configured to apply a consistency optimization objective function to optimize the initial extrinsic parameter to obtain the extrinsic parameter transformation matrix between the lidar and the camera, thereby realizing the extrinsic parameter calibration of the lidar and the camera.

[0014] Another aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the methods described above.

[0015] According to embodiments of this application, by analyzing the planarity and scattering indices of each point in the lidar point cloud at multiple candidate neighborhood scales, a high-confidence planar point set belonging to the checkerboard calibration board can be accurately screened, effectively eliminating edge, noise, and non-planar interference points, thereby obtaining stable and reliable initial extrinsic parameters. Furthermore, employing a consistency optimization objective function to refine and iterate the initial extrinsic parameters can significantly reduce reprojection errors and geometric residuals, improve the calibration accuracy and robustness of the extrinsic parameter transformation matrix, and effectively enhance the stability of point cloud target plane extraction. Attached Figure Description

[0016] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart illustrating a lidar and camera extrinsic parameter calibration method according to an embodiment of this application is shown.

[0018] Figure 2 A flowchart illustrating a high-confidence planar point screening process based on spatial feature analysis according to an embodiment of this application is shown.

[0019] Figure 3 The illustration shows a schematic diagram of the process for extracting a target plane by growing a confined region according to an embodiment of this application.

[0020] Figure 4 The illustration shows a schematic diagram of the initial extrinsic parameter optimization solution method according to an embodiment of this application.

[0021] Figure 5 A block diagram of a lidar and camera extrinsic parameter calibration device according to an embodiment of this application is shown schematically.

[0022] Figure 6 A block diagram of an electronic device suitable for implementing a lidar and camera extrinsic parameter calibration method according to an embodiment of this application is shown schematically. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0028] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0029] Figure 1 A flowchart illustrating a lidar and camera extrinsic parameter calibration method according to an embodiment of this application is shown.

[0030] like Figure 1 As shown, the lidar and camera extrinsic parameter calibration method provided in the embodiments of this application includes operations S101 to S103.

[0031] In operation S101, based on the planarity index and scattering index of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing the checkerboard calibration board information in the calibration scene, a high-confidence planar point set that meets the preset geometric shape characteristics is determined from the point cloud data.

[0032] In the embodiments of this application, the calibration scenario includes a LiDAR, a camera, and a checkerboard calibration board. Point cloud data from the LiDAR, including the checkerboard calibration board, and camera image data are collected, and camera intrinsic parameter calibration and point cloud data preprocessing are performed.

[0033] Spatial feature analysis is performed on the preprocessed point cloud data. Local geometric features of each point in the preprocessed point cloud data are calculated at multiple candidate neighborhood scales, and high-confidence planar point sets are selected based on the geometric features.

[0034] In operation S102, the initial extrinsic parameters between the lidar and the camera are determined based on the high-confidence plane point set.

[0035] In the embodiments of this application, restricted region growing is performed on a high-confidence planar point set to extract the target planar point cloud region corresponding to the checkerboard calibration board. A local planar coordinate system for the checkerboard calibration board is constructed based on the target planar point cloud region, and point cloud-side calibration features are extracted. The checkerboard calibration board in the camera image is detected, image-side calibration features are extracted, and a cross-modal correspondence between the point cloud-side calibration features and the image-side calibration features is established. The initial extrinsic parameters between the LiDAR and the camera are solved based on the cross-modal correspondence.

[0036] In operation S103, the consistency optimization objective function is applied to optimize the initial extrinsic parameters, thereby obtaining the extrinsic parameter transformation matrix between the lidar and the camera, and realizing the extrinsic parameter calibration of the lidar and the camera.

[0037] According to embodiments of this application, by analyzing the planarity and scattering indices of points in the lidar point cloud at multiple candidate neighborhood scales, a high-confidence planar point set belonging to the checkerboard calibration board can be accurately selected, effectively eliminating edge, noise, and non-planar interference points, thereby obtaining stable and reliable initial extrinsic parameters. Further refinement and iteration of the initial extrinsic parameters using a consistency optimization objective function can significantly reduce reprojection errors and geometric residuals, improving the calibration accuracy and robustness of the extrinsic parameter transformation matrix. This method fully utilizes multi-scale local geometric features, reduces dependence on manually set neighborhood sizes, and achieves high-precision, high-consistency automated extrinsic parameter calibration between the lidar and the camera.

[0038] In the embodiments of this application, the lidar point cloud data containing checkerboard calibration board information in the calibration scene is preprocessed point cloud data. The checkerboard calibration board is a black and white alternating planar checkerboard calibration board; the camera intrinsic parameter calibration adopts the checkerboard corner point calibration method, which obtains the camera intrinsic parameter matrix and distortion parameters by acquiring multiple sets of checkerboard images under different poses.

[0039] A joint calibration system consisting of a lidar and a camera was adopted, using a black and white checkerboard calibration board as the target board. The checkerboard calibration board was placed within the shared field of view of both the lidar and the camera, and lidar point cloud data and camera image data were acquired simultaneously to ensure that the calibration board appeared completely in the field of view of both sensors. Multiple sets of checkerboard images at different poses were acquired using the camera, and the camera's intrinsic parameter matrix K and distortion parameters were obtained using Zhang Zhengyou's checkerboard corner point calibration method. Preprocessing operations were performed on the acquired raw point cloud data.

[0040] The preprocessing process of lidar point cloud data includes: spatially cropping the original point cloud data using a pass-through filter to remove far-field background and invalid outlier regions; performing statistical outlier removal on the cropped point cloud data by analyzing the distribution characteristics of the distance from each point to its nearest neighbor and removing noise points whose mean distance exceeds a preset standard deviation based on the radius of the drive wheel; and downsampling the noise-removed point cloud data using a voxel grid filter to homogenize the point cloud data density while maintaining the macroscopic geometric structure of the point cloud data, thereby obtaining the preprocessed point cloud data.

[0041] According to an embodiment of this application, based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scene at each candidate neighborhood scale, a high-confidence planar point set satisfying a preset geometric shape feature is determined from the point cloud data. This includes: determining the optimal neighborhood scale of each point in the point cloud data based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information at each candidate neighborhood scale; and determining points in the point cloud data whose planarity and dispersion indices of the nearest neighbor set corresponding to the optimal neighborhood scale satisfy the preset geometric shape feature, thereby obtaining a high-confidence planar point set.

[0042] Furthermore, based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing the checkerboard calibration board information in the calibration scenario at each candidate neighborhood scale, the optimal neighborhood scale of each point in the point cloud data is determined. This includes: determining the function value of the information entropy evaluation function based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the point cloud data at each candidate neighborhood scale; and determining the optimal neighborhood scale of each point in the point cloud data based on the function values ​​of multiple information entropy evaluation functions.

[0043] In the embodiments of this application, the process of determining the planarity index and dispersion index of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scene at each candidate neighborhood scale includes: determining the covariance matrix of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scene at each candidate neighborhood scale; performing eigenvalue decomposition on the covariance matrix to obtain multiple non-negative eigenvalues; and determining the planarity index and dispersion index of the nearest neighbor set corresponding to each point in the point cloud data at each candidate neighborhood scale based on the non-negative eigenvalues.

[0044] The following is for reference. Figure 2 and Figure 3 In conjunction with specific embodiments, Figure 1 The method shown will be further explained.

[0045] Figure 2 A flowchart illustrating a high-confidence planar point screening process based on spatial feature analysis according to an embodiment of this application is shown.

[0046] like Figure 2 As shown, the high-confidence planar point screening method based on spatial feature analysis includes operations S210 to S250.

[0047] In operation S210, the preprocessed point cloud set... any point in In a candidate neighborhood scale set containing multiple scale values ​​arranged in ascending order Traverse the candidate neighborhood scale k. For each candidate neighborhood scale k, obtain the point... k-nearest neighbor set .

[0048] In operation S220: based on the k nearest neighbor set Construct a local covariance matrix that reflects the degree of local geometric discretization. The calculation method is as follows:

[0049] ;

[0050] ;

[0051] in, Point The neighborhood mean at a candidate neighborhood scale k, i.e., the geometric center of the k nearest neighbors. Representing neighborhood points The three-dimensional coordinate vector. The calculation formula is used to calculate the outer product of the deviation vectors of each neighboring point and the centroid, and then average them to obtain a 3×3 real symmetric matrix. The diagonal elements of this matrix represent the variance of the neighboring points along the three coordinate axes, and the off-diagonal elements represent the covariance between the coordinate axes, thus fully characterizing the point. Local spatial distribution characteristics at scale k.

[0052] In operation S230, the local covariance matrix... Eigenvalue decomposition yields three non-negative eigenvalues ​​arranged in descending order. ,satisfy and the corresponding three mutually orthogonal eigenvectors. Among them, the largest eigenvalue corresponding feature vector The direction of the maximum variance of the spatial distribution of the neighborhood point set is represented by the second largest eigenvalue. Corresponding feature vector Indicates and Orthogonal second largest variance directions, minimum eigenvalue The variance of the normal direction is represented by its corresponding eigenvector. That is, a point Normal vector estimation at this neighborhood scale. Based on the relative magnitude relationships between the aforementioned eigenvalues, a planarity index is constructed. and scattered indicators They are defined as follows:

[0053] ;

[0054] ;

[0055] Among them, the planeness index Quantify the degree to which a local point cloud exhibits a planar structure, when Much larger And close When this occurs, the indicator approaches 1; scatter index The quantification measures the degree to which a local point cloud is randomly distributed. When the three feature values ​​are close in magnitude, the index approaches 1.

[0056] To address the fixed-scale feature extraction bias caused by uneven point cloud density distribution in non-repetitive scanning solid-state lidar, an information theory-based adaptive scale selection mechanism is introduced when operating the S240. An information entropy evaluation function is constructed based on planarity and dispersion indices. And by minimizing this function, the optimal neighborhood scale is selected for each point. :

[0057] ;

[0058] ;

[0059] in, Point The information entropy evaluation function at candidate neighborhood scale k. Let represent the u-th local geometric feature component, and m be the number of geometric feature components involved in the information entropy calculation. A lower information entropy value indicates that a certain geometric feature component is significantly dominant at that scale, and the geometric feature description is more deterministic and distinguishable. This is determined by the optimal neighborhood scale. The determination formula searches across all candidate scales to make The minimum value is taken as the scale, and the point is determined. optimal neighborhood scale This achieves an adaptive effect, using a larger scale in sparse regions to ensure statistical stability and a smaller scale in dense regions to maintain local detail resolution.

[0060] In the S250 operation, after determining the optimal neighborhood scale, a high-confidence planar point set is obtained by screening based on the planarity index and the scatter index. :

[0061] ;

[0062] in, Indicates the planeness threshold. This represents the scatter threshold. Only when the point... Planarity index at the optimal scale Not less than Furthermore, scattered indicators Not higher than Only then is the point included in the high-confidence planar point set. Planarity threshold. Ensure that the selected points have a significant dominance of planar structure, excluding points on linear structures; scattering threshold. Ensure that the selected points are highly consistent in the normal direction, thus excluding points located at plane edges, corners, or scattered due to measurement noise. Preferably, It can be set to 0.25. It can be set to 0.15.

[0063] The high-confidence planar point screening method in this application achieves effective separation of planar points and non-planar points from the source through the above-mentioned dual threshold screening, providing a highly consistent and reliable candidate point set input for subsequent restricted region growth.

[0064] Figure 3 The illustration shows a schematic diagram of the process for extracting a target plane by growing a confined region according to an embodiment of this application.

[0065] like Figure 3 As shown, the method for extracting the target plane by growing a restricted region includes operations S310-S380.

[0066] In operation S310, the point with the smallest curvature in the high-confidence plane point set is used as the initial seed point. The current growth region and seed point queue are determined based on the initial seed point, and the initial seed point is marked.

[0067] In embodiments of this application, this operation involves seed point initialization. From a high-confidence plane point set... The point with the minimum curvature is selected as the initial seed point. Add the seed point to the seed point queue and the current growth region. The smaller the curvature, the flatter the local region where the point is located, and the closer it is to the ideal planar core region. Using this point as the growth starting point ensures that the region expansion starts from the most stable and reliable position inside the plane and moves outward.

[0068] In operation S320, the spatial nearest neighbor of the initial seed point is determined from the set of high-confidence plane points, with the initial seed point as the center, and multiple initial neighboring points are obtained.

[0069] In the embodiments of this application, this operation involves seed point dequeueing and neighborhood search. The currently pending seed point is retrieved from the head of the seed point queue. Centered on the seed point, in the point set Search for its spatial nearest neighbors to obtain multiple initial neighborhood points.

[0070] In operation S330, unlabeled initial nearest neighbors are determined from multiple initial neighboring points to obtain at least one first target nearest neighbor.

[0071] In the embodiments of this application, for each neighboring point found... If the point has been visited or belongs to a growth region, it is skipped; if the point has not been visited, it is the first target nearest neighbor and the next step of normal consistency and curvature consistency check is performed.

[0072] In operation S340, the first target nearest neighbor point whose angle between the normal vector of the first target nearest neighbor point and the normal vector of the initial seed point is less than or equal to a preset normal angle threshold is used as the second target nearest neighbor point. Based on the second target nearest neighbor point whose local curvature at the optimal neighborhood scale is less than or equal to a preset curvature threshold point, at least one third target nearest neighbor point is obtained. The normal vector of the first target nearest neighbor point is the eigenvector corresponding to the smallest non-negative eigenvalue of the first target nearest neighbor point.

[0073] In the embodiments of this application, for the normal consistency check, the neighborhood points are calculated. normal vector With the current seed point normal vector The angle between normal vector This can be derived from the minimum eigenvalue after eigenvalue decomposition of the local covariance matrix in operation S230. The corresponding eigenvector. Normal vector. Alternatively, the neighborhood can be determined as follows: based on the optimal neighborhood scale obtained from operation S240, the eigenvector corresponding to the smallest eigenvalue after eigenvalue decomposition of the local covariance matrix of the nearest neighbor set at the optimal neighborhood scale is taken as the neighborhood point. normal vector .

[0074] included angle The angle is calculated using the inverse cosine of the dot product of the two normal vectors, reflecting the difference in orientation of the local plane containing the two points. Less than or equal to the preset normal angle threshold This indicates that the local plane containing the neighboring point and the seed point are coplanar. By passing the normal consistency test, the second target nearest neighbor point is obtained.

[0075] For curvature consistency testing, neighborhood points are obtained. Local curvature at the optimal neighborhood scale curvature It is possible or The curvature is measured using indicators such as curvature to reflect the degree of curvature of the local surface at that point. Less than or equal to the preset curvature threshold This indicates that the neighboring point is located in a flat region rather than at a high curvature edge or corner. The third target nearest neighbor is obtained through the curvature consistency test.

[0076] In operation S350, the third target's nearest neighbor is added to the current growth region and seed point queue, the current growth region and seed point queue are updated, and the third target's nearest neighbor is marked.

[0077] In the embodiments of this application, for neighborhood points that simultaneously pass the normal consistency test and the curvature consistency test... Perform the following operations: add the neighboring point to the current growth region; mark the neighboring point as visited; add the neighboring point to the seed point queue as a candidate seed point for subsequent region expansion, thereby achieving connectivity growth of the region along a smooth plane.

[0078] In operation S360, the third target nearest neighbor point is used as the initial seed point. The initial neighbor point determination operation, the first target nearest neighbor point determination operation, the third target nearest neighbor point determination operation, the current growth region and seed point queue update operation, and the third target nearest neighbor point marking operation are repeatedly executed until the preset first termination condition is met, and the candidate plane region is obtained.

[0079] In the embodiments of this application, operations S320 to S350 are repeated. The growth process of the current region is stopped when any of the following termination conditions are met: Condition 1, the seed point queue is empty, that is, there are no more unvisited neighboring points satisfying the normal and curvature constraints at the boundary of the current region; Condition 2, the point cloud size of the current growth region has reached the preset maximum region threshold. This prevents overgrowth caused by an excessively loose threshold setting or the presence of large planar structures in the scene. After stopping growth in the current region, candidate planar regions are obtained.

[0080] In operation S370, if there are unmarked points in the high-confidence plane point set, the unmarked points are used as the high-confidence plane point set. The operations of determining the current growth region and seed point queue, marking the initial seed point, determining the initial neighbor point, determining the first target nearest neighbor point, determining the third target nearest neighbor point, updating the current growth region and seed point queue, marking the third target nearest neighbor point, and determining the candidate plane region are repeatedly executed until the preset second termination condition is met, resulting in multiple candidate plane regions.

[0081] In the embodiments of this application, if the current region growth terminates, the high-confidence plane point set... If there are still unvisited points, then select the point with the smallest curvature from the remaining unvisited points as the new seed point, and repeat operations S301 to S360 until... If all points are visited or the number of remaining unvisited points is less than the preset minimum cluster size threshold, multiple candidate planar regions are obtained.

[0082] When operating the S380, the target planar point cloud region is determined from multiple candidate planar regions according to preset rules.

[0083] In the embodiments of this application, from all the extracted candidate planar regions, the target planar point cloud region corresponding to the checkerboard calibration board is selected according to criteria such as region size, planar fitting residual and shape features.

[0084] The embodiments of this application perform restricted region growth on a high-confidence planar point set, shrinking the growth space from a global point set to a point set with significant local planar structure. This effectively reduces the problems of over-splitting and over-adhesion of regions and improves the extraction accuracy of the checkerboard target plane.

[0085] According to an embodiment of this application, the initial extrinsic parameters between the lidar and the camera are determined based on the correspondence between the calibration features of the target plane point cloud region corresponding to the checkerboard calibration board in the high-confidence plane point set and the calibration features of the camera image containing checkerboard calibration board information in the calibration scene.

[0086] In the embodiments of this application, starting from the extracted target planar point cloud region, a local planar coordinate system in which the chessboard is located is constructed; the point cloud is projected onto the planar coordinate system, and the calibration features of the point cloud side are extracted, including the three-dimensional spatial coordinates of the chessboard corner points, the planar normal vector of the calibration board, and the geometric information such as the boundary contour.

[0087] The method involves detecting a checkerboard calibration board in camera images and extracting image side-calibration features. This includes: extracting image side-calibration features from acquired camera images containing the checkerboard calibration board using a checkerboard corner detection algorithm to achieve sub-pixel accuracy corner localization and outputting the two-dimensional coordinates (u, v) of each checkerboard corner point in the image pixel coordinate system. Simultaneously, based on the physical dimensions of the checkerboard and the known number of rows and columns, the three-dimensional position index in the world coordinate system corresponding to each detected corner point is determined.

[0088] Establishing cross-modal correspondences between point cloud-side calibration features and image-side calibration features includes:

[0089] Based on the pinhole camera imaging model and the calibrated camera intrinsic parameter matrix K and distortion parameters, the coordinates of the extracted point cloud-side 3D corner points are transformed to the camera coordinate system using a rotation matrix R and a translation vector t. Then, they are projected onto the image pixel plane using the camera intrinsic parameter matrix K to obtain the reprojected 2D coordinates of each 3D corner point on the image. Following the row and column index order of the checkerboard corner points, the reprojected corner points are matched one-to-one with the 2D image corner points detected in the image side calibration feature extraction method, establishing a 2D-3D geometric correspondence between each pair of matched points.

[0090] The quality of the above cross-modal correspondence is checked by a two-way consistency check, that is, after the corner point of the point cloud is reprojected onto the image, the corner point of the image is then verified in reverse to see if it falls within the preset neighborhood of the reprojected point. Incorrect matches caused by corner point misdetection, occlusion or incomplete point cloud feature extraction are eliminated, and high-quality correspondences are retained for the initial extrinsic parameter solution.

[0091] The initial extrinsic parameters between the lidar and the camera are solved based on the cross-modal correspondence, including: solving for the initial extrinsic parameters between the lidar and the camera using PnP or nonlinear optimization methods based on the established high-quality 2D-3D geometric correspondence, and obtaining the initial rotation matrix. Translation vector .

[0092] In the embodiments of this application, a consistency optimization objective function containing point-level reliability weights is constructed, the point-level reliability weights are predicted by a lightweight multilayer perceptron, the initial extrinsic parameters are optimized and solved, and the extrinsic parameter transformation matrix between the lidar and the camera is output to complete the extrinsic parameter calibration.

[0093] The traditional basic consistency loss function is:

[0094] ;

[0095] in, This represents the geometric consistency residual of the i-th point participating in the optimization. This represents the validity constraint at the i-th point. The function assigns equal weighted contributions to all points that satisfy the validity constraint.

[0096] Based on this, point-level reliability weights are introduced. A weighted consistency loss function is constructed and used as the objective function for consistency optimization, specifically as follows:

[0097] ;

[0098] The contribution at each effective point is generalized from a fixed constant 1 to a continuously adjustable function. This allows the residuals of high-reliability points to receive greater weight and dominate the gradient descent direction during the optimization process, while the contribution of low-reliability points is suppressed, thereby reducing the interference of noise points, unstable edge points, and outliers.

[0099] Point-level reliability weights A neural function with parameter θ The prediction is as follows:

[0100] ;

[0101] The input to the function is a point-level feature vector. Defined as:

[0102] ;

[0103] ;

[0104] Where L is the consistency optimization objective function; N is the number of points in the point cloud data that participate in the optimization of the initial extrinsic parameters; For the validity constraint of the i-th point; Point-level reliability weights; The geometric consistency residual at the i-th point; For point-level feature vectors Point-level reliability weights The function; Let be the projected coordinates of the i-th point in the coordinate system of the chessboard calibration board. Let be the laser reflection intensity corresponding to the i-th point; Let be the radial distance from the i-th point to the center of the checkerboard calibration board.

[0105] Point-level feature vectors It simultaneously characterizes the geometric position distribution characteristics and physical reflection characteristics of points, enabling neural functions to perceive the reliability distribution patterns in different positional regions and reflectivity modes.

[0106] neural functions Implemented using a lightweight multilayer perceptron, the network structure is 4→32→32→1, meaning the input layer contains 4 neurons. The structure has four dimensions: the first hidden layer contains 32 neurons, the second hidden layer contains 32 neurons, and the output layer contains 1 neuron. Both the first and second hidden layers use the ReLU activation function, with the following functional form: It is computationally simple and has low computational overhead, which is beneficial for achieving fast inference in resource-constrained environments such as embedded platforms. The output layer uses the Sigmoid activation function, whose functional form is: This naturally maps the output value to the (0,1) interval, satisfying the point-level reliability weighting. The network has specific domain requirements. It contains only about 1200 trainable parameters, which is much smaller than that of conventional deep neural network models.

[0107] The lightweight multilayer perceptron employs a self-supervised training method. The training process of the multilayer perceptron network includes: acquiring point cloud data samples; determining high-reliability points in the point cloud data samples based on a first comparison between the geometric consistency residuals of each point in the point cloud data samples and a preset high-reliability residual threshold; determining low-reliability points in the point cloud data samples based on a second comparison between the geometric consistency residuals of each point in the point cloud data samples and a preset low-reliability residual threshold; determining the point-level feature vectors of the target sample points using the high-reliability and low-reliability points as target sample points; and training the multilayer perceptron network using the point-level feature vectors of the target sample points as input and the point-level reliability weight samples corresponding to the target sample points as output.

[0108] Furthermore, by applying initial extrinsic parameter solving and residual calculation, without introducing point-level reliability weights, the initial extrinsic parameters are used. Perform preliminary convergence calculations and calculate the geometric consistency residuals for all points involved in the optimization. .

[0109] Implement point-level pseudo-label construction. Based on the residuals of each point. The size of the residual determines whether all participating points are divided into a high-reliability set and a low-reliability set; a high-reliability residual threshold is set. and low reliability residual threshold And satisfy For residuals Points that are deemed high-reliability points are assigned a pseudo-label value of 1 (indicating that the contribution of this point to optimization should be the same as that of effective points); for residuals... Points with low reliability are identified and assigned a pseudo-label value of 0 (indicating that the contribution of this point to optimization should be suppressed); for residuals between... and Points between these points are not included in the training to avoid interference from blurred boundary samples on the network's learning. This pseudo-label construction method based on residual size is based on a reasonable assumption: under convergent extrinsic parameters, points with small residuals are usually high-quality points with accurate spatial location and stable reflection intensity, while points with large residuals are usually low-quality points affected by noise, edge effects, or sparse sampling.

[0110] Using constructed pseudo-labels as supervision signals, a lightweight multilayer perceptron is subjected to... Perform training; the loss function during training can be a binary classification cross-entropy loss, which has the following form:

[0111] ;

[0112] in, For constructed pseudo-labels (0 or 1). This is the network's predicted reliability weight for the i-th point; by minimizing this loss function, the network learns to extract the feature vector of high-reliability points. Mapping the feature vectors of low-reliability points to weight values ​​close to 1. Mapped to weight values ​​close to 0.

[0113] neural functions It is a pre-trained multilayer perceptron network that takes point-level feature vectors as input and corresponding point-level reliability weights as output.

[0114] After training, the lightweight multilayer perceptron will be learned. The weighted consistency optimization objective function is embedded to replace the fixed equal-weight strategy; in the subsequent extrinsic parameter refinement and optimization iterations, the network predicts its own point-level reliability weight for each point. This allows for adaptive adjustment of the contribution of different quality points, resulting in the trained multilayer perceptron network.

[0115] Through the self-supervised training mechanism described above, the network can autonomously learn the implicit reliability distribution pattern from the statistical distribution of point cloud geometric residuals, and obtain effective point-level reliability assessment capabilities without manual annotation, which significantly reduces the engineering deployment threshold of the method.

[0116] Applications below Figure 4 The process of optimizing the initial extrinsic parameters is explained.

[0117] Figure 4 The illustration shows a schematic diagram of the initial extrinsic parameter optimization solution method according to an embodiment of this application.

[0118] like Figure 4 As shown, the initial extrinsic parameter optimization solution method includes operations 410 to 490.

[0119] In operation 410, input the initial external parameters.

[0120] In operation 420, point-level feature vectors are extracted.

[0121] In operation 430, input a lightweight multilayer perceptron.

[0122] In operation 430, the point-level feature vector Input a lightweight multilayer perceptron (MLP).

[0123] In operation 440, output the reliability weight.

[0124] In operation 450, calculate the geometric consistency residual.

[0125] In operation 460, a weighted consistency loss is constructed.

[0126] In operation 460, the weighted consistency loss function is used as the consistency optimization objective function.

[0127] In operation 470, minimize the weighted consistency loss.

[0128] In operation 470, the weighted consistency loss is minimized to obtain the extrinsic parameters.

[0129] In operation 480, update the external parameters.

[0130] In operation 490, the optimized results are output.

[0131] This invention introduces a lightweight multilayer perceptron to establish a point-level reliability weight function, transforming the traditional hard binary mask constraint into a continuously learnable weighted constraint. This allows the contribution of point cloud features with different confidence levels to the optimization objective to be adaptively adjusted, thereby improving the ability of the extrinsic parameter optimization process to suppress noise points and outliers. This invention retains the interpretability of the target board geometric constraint method while enhancing optimization robustness through a lightweight learning model. It does not rely on complex large-scale end-to-end networks, making it easy to deploy in resource-constrained platforms or embedded scenarios.

[0132] According to embodiments of this application, the lidar and camera extrinsic parameter calibration method further includes:

[0133] Output the rotation matrix R and translation vector t or homogeneous transformation matrix T from the lidar coordinate system to the camera coordinate system to complete the external parameter calibration; and perform quantitative evaluation on the calibration results.

[0134] Calculate the root mean square value of the reprojection error:

[0135] ;

[0136] Where M represents the number of corner pairs that successfully established a correspondence. This represents the 2D reprojection coordinates of the 3D corner points of the point cloud after extrinsic parameter transformation and camera intrinsic parameter projection. This represents the pixel coordinates of the corresponding corner point detected on the image side, and ||| represents the Euclidean distance. The smaller the RMSE, the higher the pixel-level alignment accuracy.

[0137] When rotation parameter reference true value exists At that time, calculate the rotation parameter deviation: When there is a translation parameter reference true value At that time, calculate the translation parameter deviation: When there is no reference true value, the standard deviation of the rotation parameter is calculated through N repeated calibration experiments. Standard deviation of translation parameters :

[0138] ;

[0139] ;

[0140] in, , These are the calibration results for the nth time. and Let be the mean of N results, and let euler() be the transformation function from rotation matrix to Euler angles. Using reprojection error as the pixel-level accuracy evaluation index and rotation and translation parameter deviations or standard deviations as the spatial accuracy and repeatability evaluation indexes, a quantitative assessment of the calibration quality of lidar and camera extrinsic parameters is achieved.

[0141] This invention is applicable to non-repetitive scanning solid-state lidar scenarios. In such scenarios, due to uneven point cloud distribution and significant fluctuations in reflection intensity, traditional calibration methods based on fixed-plane extraction and equal-weighted loss often suffer from unstable extraction and unrobust optimization. This application, through a collaborative design of front-end spatial feature analysis and back-end endpoint reliability weighting, can obtain more stable and accurate extrinsic parameter calibration results in complex scenarios.

[0142] Figure 5 A block diagram of a lidar and camera extrinsic parameter calibration device according to an embodiment of this application is shown schematically.

[0143] like Figure 5 As shown, the lidar and camera extrinsic parameter calibration device 500 includes a target point set determination module 510, an initial extrinsic parameter determination module 520, and a transformation matrix determination module 530.

[0144] The target point set determination module 510 is used to determine a high-confidence planar point set that meets the preset geometric shape characteristics from the point cloud data of LiDAR point cloud data containing checkerboard calibration board information, based on the planarity index and scatter index of the nearest neighbor point set corresponding to each point at each candidate neighborhood scale. The calibration scene includes LiDAR, camera and checkerboard calibration board.

[0145] The initial extrinsic parameter determination module 520 is used to determine the initial extrinsic parameters between the lidar and the camera based on the high-confidence plane point set.

[0146] The transformation matrix determination module 530 is used to apply the consistency optimization objective function to optimize the initial extrinsic parameters, obtain the extrinsic transformation matrix between the lidar and the camera, and realize the extrinsic parameter calibration of the lidar and the camera.

[0147] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0148] For example, any plurality of the target point set determination module 510, the initial extrinsic parameter determination module 520, and the transformation matrix determination module 530 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the target point set determination module 510, the initial extrinsic parameter determination module 520, and the transformation matrix determination module 530 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the target point set determination module 510, the initial extrinsic parameter determination module 520, and the transformation matrix determination module 530 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0149] It should be noted that the lidar and camera extrinsic parameter calibration device part in the embodiments of this application corresponds to the lidar and camera extrinsic parameter calibration method part in the embodiments of this application. For a detailed description of the lidar and camera extrinsic parameter calibration device part, please refer to the lidar and camera extrinsic parameter calibration method part, which will not be repeated here.

[0150] Figure 6 A block diagram of an electronic device suitable for implementing a lidar and camera extrinsic parameter calibration method according to an embodiment of this application is shown schematically. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0151] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0152] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0153] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0154] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0156] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0158] Embodiments of this application also include a computer program product, which includes a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the lidar and camera extrinsic parameter calibration method provided in the embodiments of this application.

[0159] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0160] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0161] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0163] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for calibrating the extrinsic parameters of a lidar and camera, characterized in that, The method for calibrating the external parameters of the lidar and camera includes: Based on the planarity and scattering indices of the nearest neighbor set corresponding to each point at each candidate neighborhood scale in the lidar point cloud data containing checkerboard calibration board information in the calibration scene, a high-confidence planar point set satisfying the preset geometric shape characteristics is determined from the point cloud data; the calibration scene includes the lidar, camera and checkerboard calibration board. Based on the high-confidence plane point set, determine the initial extrinsic parameters between the lidar and the camera; By applying a consistency optimization objective function, the initial extrinsic parameters are optimized to obtain the extrinsic transformation matrix between the lidar and the camera, thereby achieving extrinsic parameter calibration between the lidar and the camera.

2. The lidar and camera extrinsic parameter calibration method according to claim 1, characterized in that, Based on the planarity and scattering indices of the nearest neighbor set corresponding to each point at each candidate neighborhood scale in the lidar point cloud data containing checkerboard calibration board information in the calibration scenario, a high-confidence planar point set satisfying preset geometric shape characteristics is determined from the point cloud data, including: Based on the planarity and scattering indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scenario at each candidate neighborhood scale, the optimal neighborhood scale of each point in the point cloud data is determined. From the point cloud data, points whose planarity and randomness indices satisfy the preset geometric shape features are determined for the nearest neighbor point set at the optimal neighborhood scale, thus obtaining a high-confidence planar point set.

3. The lidar and camera extrinsic parameter calibration method according to claim 2, characterized in that, The process of determining the planarity and dispersion indices of the nearest neighbor set corresponding to each point at each candidate neighborhood scale in the lidar point cloud data containing checkerboard calibration board information in the calibration scenario includes: Determine the covariance matrix of the nearest neighbor set of each point in the lidar point cloud data containing chessboard calibration board information in the calibration scenario at each candidate neighborhood scale; The covariance matrix is ​​decomposed into eigenvalues ​​to obtain multiple non-negative eigenvalues; Based on the non-negative eigenvalues, the planarity index and scatter index of the nearest neighbor set corresponding to each point in the point cloud data at each candidate neighborhood scale are determined.

4. The lidar and camera extrinsic parameter calibration method according to claim 2, characterized in that, Based on the planarity and dispersion indices of the nearest neighbor set corresponding to each point in the LiDAR point cloud data containing checkerboard calibration board information in the calibration scenario at each candidate neighborhood scale, the optimal neighborhood scale of each point in the point cloud data is determined, including: Based on the planarity index and the randomness index of the nearest neighbor set corresponding to each point in the point cloud data at each candidate neighborhood scale, determine the function value of the information entropy evaluation function based on the planarity index and the randomness index corresponding to each point in the point cloud data at each candidate neighborhood scale. The optimal neighborhood scale of each point in the point cloud data is determined based on the function values ​​of multiple information entropy evaluation functions.

5. The lidar and camera extrinsic parameter calibration method according to claim 3, characterized in that, Based on the correspondence between the calibration features of the target plane point cloud region corresponding to the checkerboard calibration board in the high-confidence plane point set and the calibration features of the camera image containing the checkerboard calibration board information in the calibration scene, the initial extrinsic parameters between the lidar and the camera are determined.

6. The method for calibrating the extrinsic parameters of a lidar and camera according to claim 1, characterized in that, The consensus optimization objective function is: ； ； ； ； Where L is the consistency optimization objective function; N is the number of points in the point cloud data that participate in the optimization of the initial extrinsic parameters; For the validity constraint of the i-th point; Point-level reliability weights; The geometric consistency residual at the i-th point; For point-level feature vectors Point-level reliability weights The function; Let be the projected coordinates of the i-th point on the chessboard calibration board; Let be the laser reflection intensity corresponding to the i-th point; Let be the radial distance from the i-th point to the center of the checkerboard calibration board.

7. The lidar and camera extrinsic parameter calibration method according to claim 6, characterized in that, function It is a pre-trained multilayer perceptron network that takes the point-level feature vector as input and the corresponding point-level reliability weight as output. The training process of the multilayer perceptron network includes: Obtain point cloud data samples; Based on the first comparison result between the geometric consistency residual of each point in the point cloud data sample and the preset high reliability residual threshold, the high reliability points in the point cloud data sample are determined. Based on the second comparison result between the geometric consistency residual of each point in the point cloud data sample and the preset low reliability residual threshold, the low reliability points in the point cloud data sample are determined. Using the high reliability points and the low reliability points as target sample points, determine the point-level feature vector of the target sample points; The multilayer perceptron network is trained by taking the point-level feature vector of the target sample point as input and the point-level reliability weight sample corresponding to the target sample point as output.

8. The lidar and camera extrinsic parameter calibration method according to claim 5, characterized in that, The process of determining the target plane point cloud region corresponding to the checkerboard calibration board in the high-confidence plane point set includes: The point with the smallest curvature in the high-confidence plane point set is used as the initial seed point. The current growth region and seed point queue are determined based on the initial seed point, and the initial seed point is marked. Using the initial seed point as the center, determine the spatial nearest neighbor points of the initial seed point from the set of high confidence plane points to obtain multiple initial neighborhood points; From the plurality of initial neighbor points, determine unlabeled initial nearest neighbor points to obtain at least one first target nearest neighbor point; A first target nearest neighbor point is selected as the second target nearest neighbor point, with the angle between the normal vector of the first target nearest neighbor point and the normal vector of the initial seed point being less than or equal to a preset normal angle threshold. Based on the second target nearest neighbor point whose local curvature at the optimal neighborhood scale is less than or equal to a preset curvature threshold, at least one third target nearest neighbor point is obtained. The normal vector of the first target nearest neighbor point is the eigenvector corresponding to the smallest non-negative eigenvalue of the first target nearest neighbor point. Add the third target nearest neighbor to the current growth region and the seed point queue, update the current growth region and the seed point queue, and mark the third target nearest neighbor; Using the third target nearest neighbor as the initial seed point, the initial neighbor point determination operation, the first target nearest neighbor point determination operation, the third target nearest neighbor point determination operation, the current growth region and seed point queue update operation, and the third target nearest neighbor point marking operation are repeatedly executed until the preset first termination condition is met, and a candidate planar region is obtained. If there are unmarked points in the high-confidence plane point set, the unmarked points are used as the high-confidence plane point set. The operations of determining the current growth region and seed point queue, marking the initial seed point, determining the initial neighbor point, determining the first target nearest neighbor point, determining the third target nearest neighbor point, updating the current growth region and seed point queue, marking the third target nearest neighbor point, and determining the candidate plane region are repeatedly executed until the preset second termination condition is met, resulting in multiple candidate plane regions. The target planar point cloud region is determined from multiple candidate planar regions according to preset rules.

9. A laser radar and camera extrinsic parameter calibration device, characterized in that, The lidar and camera extrinsic parameter calibration device includes: The target point set determination module is used to determine a high-confidence planar point set that satisfies preset geometric shape characteristics from the point cloud data of LiDAR point cloud data containing checkerboard calibration board information, based on the planarity index and scatter index of the nearest neighbor point set corresponding to each point at each candidate neighborhood scale; the calibration scene includes LiDAR, camera and checkerboard calibration board. The initial extrinsic parameter determination module is used to determine the initial extrinsic parameters between the lidar and the camera based on the high-confidence plane point set; The transformation matrix determination module is used to apply a consistency optimization objective function to optimize the initial extrinsic parameters, obtain the extrinsic transformation matrix between the lidar and the camera, and realize the extrinsic calibration of the lidar and the camera.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

Images