A monocular depth auxiliary-based laser radar-camera external parameter calibration method

Abstract

The application discloses a kind of laser radar-camera external parameter calibration methods based on monocular depth auxiliary, including image feature extraction, point cloud feature extraction, SVD-ICP-based external parameter rough calibration, monocular depth map estimation and normalization processing, cross-modal depth constraint construction, joint optimization and external parameter fine solution etc.Steps.The application introduces monocular depth estimation method, realizes the accurate alignment of two-dimensional image and three-dimensional point cloud, effectively breaks through the modal barrier of point cloud and image, and provides a new technical path for the construction of cross-modal feature association.The method can complete the depth information association of image and point cloud without manual intervention, and the association mechanism based on depth features significantly reduces the strong dependence of traditional targetless calibration methods on environmental texture and lighting conditions.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion perception technology, involving computer vision and robotics technology. Specifically, it is applied to the high-precision joint calibration of LiDAR and vision cameras in intelligent navigation systems, autonomous driving and mobile robots, which can achieve spatially accurate alignment of multi-source sensor data in complex scenarios. Background Technology

[0002] Sensor fusion is a core technology for improving the environmental perception and positioning accuracy of navigation systems, and its fusion effect directly depends on the accuracy of extrinsic parameter calibration between sensors. The heterogeneous characteristics of LiDAR and cameras make the calibration system a complex cross-modal adaptation problem. The calibration process is affected by the inherent characteristics of the sensors themselves, including the scanning mode (rotational / non-repetitive scanning) and point cloud sparsity of LiDAR, and the imaging distortion and field-of-view differences of cameras. It is also affected by external random factors such as drastic changes in ambient lighting, differences in material reflectivity, motion blur, and noise interference, making it difficult to establish accurate geometric mapping relationships between sensors, thus affecting the spatial registration quality of multi-source data. Therefore, based on an in-depth analysis of the cross-modal data correlation characteristics and environmental influencing factors, and combining the advantages of cross-modal constraint construction and optimization algorithms, it is necessary to design a reasonable extrinsic parameter calibration model to achieve high-precision solution of extrinsic parameters between sensors.

[0003] Common LiDAR-camera extrinsic parameter calibration methods include target-based direct registration and hand-eye calibration. These models, built upon explicit geometric or motion constraints, offer some physical logic support, but their application conditions are stringent: target-based methods require precise manual placement of the calibration board and demand extremely high feature extraction accuracy, making them prone to failure in scenarios with target occlusion or edge distortion; hand-eye calibration relies on high-precision time synchronization and rich motion trajectories from the sensor, making it susceptible to trajectory noise and difficult to adapt to dynamic and complex environments. In practical calibration scenarios, the modal differences between LiDAR and camera (3D discrete point clouds versus 2D continuous pixels) and multi-factor coupling interference make the aforementioned linear models insufficient for high-precision calibration. Therefore, some calibration methods based on nonlinear constraints have emerged, such as targetless methods based on mutual information, feature matching methods based on deep learning, and joint optimization methods based on composite constraints.

[0004] Target-based methods construct geometric constraint models by placing artificial markers such as checkerboard patterns or ArUco codes in the scene and utilizing salient features like points, lines, and surfaces. However, the cumbersome manual placement process and limited environmental adaptability of these methods restrict their automated application. To overcome this limitation, target-free calibration methods have emerged. These methods are free from the constraints of physical markers and optimize the objective function by extracting edge and semantic features from the natural environment or by utilizing the mutual information between image grayscale and point cloud intensity. While they support online calibration in dynamic scenes, their robustness and versatility face challenges under conditions of sparse texture or drastic lighting changes. Unlike the static feature-based methods mentioned above, motion constraint-based methods transform the calibration problem into a hand-eye calibration framework. They derive extrinsic parameters by back-calculating the pose sequence of sensors moving with the carrier, possessing the unique advantage of not requiring overlapping fields of view. However, their accuracy is highly dependent on the richness of the trajectory and the accuracy of the intrinsic parameters, and they are often susceptible to noise interference. With the development of artificial intelligence, deep learning-based methods have become a research hotspot in recent years, effectively reducing human intervention. This type of method includes both accurate extrinsic parameter estimation by directly regressing parameters through the extraction of global or local features, and relative extrinsic parameter prediction based on retrieval matching mechanisms. Although it performs well in terms of environmental adaptability, it is still in the exploratory stage in terms of model interpretability, generalization and dataset construction system.

[0005] Commonly used methods for constructing cross-modal calibration constraints include gray-intensity information constraint methods and target feature-based geometric constraint methods. Among these, the gray-intensity information constraint method is represented by the calibration tool "General, Single-shot, Target-less, and Automatic LiDAR-Camera Extrinsic Calibration Toolbox" proposed by Koide et al. Its core principle is to quantify the statistical correlation of cross-modal data by calculating the normalized mutual information distance (NID) between image gray levels and point cloud intensities, and then construct an optimization objective function to solve for the extrinsic parameters. The principle involves projecting the LiDAR point cloud intensity into a virtual image, establishing an initial correspondence using feature matching algorithms such as SuperGlue, and then optimizing the extrinsic parameters using the NID index. The NID calculation formula is as follows: ; ; in For information entropy, To achieve mutual information, this method does not require a physical target and is compatible with various sensor models, but it relies on the statistical correlation between grayscale and intensity.

[0006] The geometric constraint method based on target features, "FAST-Calib: LiDAR-Camera Extrinsic Calibration in One Second," employs a customized calibration board integrating a circular aperture array and ArUco markers. It extracts the center feature points of the circular apertures on the calibration board and uses geometric registration methods such as the Kabsch algorithm to solve for extrinsic parameters. Its core is to segment the calibration board plane using the RANSAC algorithm, detect edge points based on angular gaps, compensate for edge distortion through ellipse fitting, and finally establish the geometric correspondence of the feature point set through least-squares optimization. The optimization objective function is as follows: ; in, and These are the center feature points in the camera and LiDAR coordinate systems, respectively. This method is straightforward and efficient, but it relies on the accurate extraction of target features to determine the extrinsic parameters.

[0007] In summary, in the gray-intensity-based information constraint method, the imaging mechanisms of image grayscale and point cloud intensity are fundamentally different. Grayscale is affected by visible light diffuse reflection and ambient light sources, while point cloud intensity depends on laser reflection characteristics. The cross-modal statistical correlation between the two is weak and lacks stability, resulting in low constraint robustness. In scenarios with drastic changes in illumination, material reflection, or missing textures, the objective function is prone to converge to a local optimum, leading to distorted calibration results. In the geometric constraint method based on target features, the sparsity of lidar point clouds, edge scattering effects, and camera image noise can easily cause positioning errors of feature points such as the center of the circular hole and vertices. Furthermore, the vertices of the calibration board may not be accurately identified due to missing point clouds or edge distortion, directly affecting the accuracy of extrinsic parameter calculation. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing lidar-camera extrinsic parameter calibration techniques and propose a monocular depth-assisted lidar-camera extrinsic parameter high-precision calibration algorithm to more accurately achieve feature matching and extrinsic parameter solving between sensors, thereby improving the accuracy and robustness of calibration in complex environments.

[0009] This invention addresses the problems of missing hole edges, insufficient accuracy in center extraction, and poor environmental adaptability in traditional target-based calibration methods. The following improvements are made: To address the issues of missing hole edges and inaccurate center positioning, a calibration board vertex is added as a supplementary feature to the original hole features, forming a dual-feature extraction scheme of "hole + vertex". Simultaneously, a hybrid optimization strategy combining principal component analysis (PCA) and ideal geometric constraints is employed to virtually complete missing vertices. For hole edges affected by edge scattering, angle gap detection and clustering optimization are used to effectively avoid calibration deviations caused by insufficient extraction of single hole features. To effectively solve the problem of poor robustness of traditional image grayscale-point cloud intensity constraints, and addressing the pain point of unreasonable cross-modal constraint construction, the traditional constraint method relying on appearance features is changed. A monocular depth estimation network is introduced to obtain a dense predicted depth map with physical scale. The normalized mutual information distance (NID) of the depth distribution is used as the core constraint, replacing the grayscale-intensity constraint which is easily affected by illumination and material, significantly improving the stability of constraints in complex environments. This method first extracts dual features of "circular hole + vertex" using a composite calibration plate, and then uses a hybrid optimization strategy to achieve accurate matching of cross-modal feature point sets, thus realizing coarse calibration of extrinsic parameters. Next, it constructs cross-modal geometric priors through monocular depth estimation, and establishes a joint optimization function by fusing geometric constraints and depth information constraints to perform fine calibration of extrinsic parameters. Finally, it solves for the optimal extrinsic parameters using a nonlinear optimization algorithm, which to some extent avoids the problems of large feature extraction errors and poor environmental adaptability of traditional target-based methods and traditional constraint methods. This provides an effective method for high-precision extrinsic parameter calibration of LiDAR-camera systems in complex scenarios.

[0010] The technical solution of the present invention is as follows: A joint optimization and calibration method for extrinsic parameters of a lidar-camera system based on monocular depth assistance is proposed. The core of the method consists of two stages: coarse calibration and fine calibration. High-precision extrinsic parameter solving is achieved through feature extraction, cross-modal constraint fusion, and nonlinear optimization. The specific technical solution is as follows: [1] Image feature extraction: Key feature points such as the center of the circular hole and the contour vertex on the calibration plate are accurately extracted from a single frame image and mapped to the camera coordinate system to provide a high-precision image feature set for cross-modal feature matching.

[0011] 【1.1】Based on the predefined ArUco encoding dictionary, the input image is marked for detection, the edge corners marked by ArUco are extracted, and the accuracy of the corner coordinates is improved by the sub-pixel thinning algorithm.

[0012] 【1.2】Construct a local coordinate system with the calibration plate plane as the reference (Z axis is perpendicular to the plate surface), and determine the theoretical three-dimensional coordinates of each ArUco marker corner point in this local coordinate system based on the physical design parameters of the calibration plate.

[0013] [1.3] Utilizing the projection correspondence between the two-dimensional corner points detected in the image and the three-dimensional corner points in the local coordinate system, the pose parameters of the calibration board relative to the camera, including the rotation matrix, are calculated using the perspective N-point (PnP) algorithm. Translation vector .

[0014] [1.4] Based on the physical geometric parameters of the calibration plate, determine the precise three-dimensional position of the center of the circular hole and the vertex of the contour in the local coordinate system. It can be mapped to the camera coordinate system using the following formula: ; In the formula, These are the 3D feature coordinates in the camera coordinate system, including the coordinate information of the center of the circular hole and the vertices of the contour.

[0015] [2] Point cloud feature extraction: Extract the center of the circular hole and the contour vertex features corresponding to the image from the LiDAR point cloud, and fill in the missing vertices through a hybrid optimization algorithm to construct a high-precision feature point set in the LiDAR coordinate system.

[0016] [2.1] Voxel downsampling is performed on the original lidar point cloud to reduce data redundancy; the main geometric planes in the scene are extracted using the Random Sample Consensus (RANSAC) algorithm, and the calibration board plane is accurately separated by combining clustering algorithm and point picking interaction, and then projected onto two-dimensional space.

[0017] [2.2] The center of the circular hole is located by boundary clustering and RANSAC circle fitting algorithm. The prior radius constraint of the circular hole on the calibration plate is used to eliminate false target points and ensure the accuracy of the center extraction of the circular hole.

[0018] [2.3] To address the issue of missing vertices on the calibration board due to sparse point clouds or edge effects, a hybrid optimization algorithm combining principal component analysis (PCA) and ideal geometric constraints is adopted: PCA is used to perform dimensionality reduction analysis on the edge point cloud of the calibration board, and a local coordinate system is constructed with the centroid of the point cloud as the origin and the eigenvector direction as the basis vector to obtain the initial translation vector. t and rotation angle θ An ideal rectangular mathematical model is established based on the physical dimensions of the calibration plate, and the set of parameters to be optimized is defined. Construct the objective function: ; In the formula, These are the edge sampling points of the ideal rectangular model. For point cloud observation by lidar, M The number of sampling points is used. The Nelder-Mead algorithm is employed to minimize the objective function, and a distance threshold is introduced. = 0.02 m: If the distance to the nearest neighbor observation point of the optimized ideal vertex is... If the observed point is true, then the observed point is used as the true vertex; otherwise, the optimized rectangular model vertices are used for virtual completion to ensure geometric consistency of the features.

[0019] [3] SVD-ICP-based coarse extrinsic calibration: By registering cross-modal feature point sets, the initial values ​​of extrinsic parameters between the lidar and the camera are solved, providing highly reliable initial parameters for fine calibration.

[0020] [3.1] After completing [1] and [2], obtain the image feature point set in the camera coordinate system. Point cloud feature point set in lidar coordinate system Then, proceed to the coarse calibration stage of extrinsic parameters, construct the objective function, and minimize the sum of squared Euclidean distances between the two sets of point sets after rigid body transformation: ; In the formula, To find the rotation matrix, To find the translation vector, [3.2] A closed-form solution based on Singular Value Decomposition (SVD) is used to solve the above least squares problem: the geometric centroids of the two sets of feature points are calculated, and the two sets of points are centered to eliminate the influence of translation. The cross-covariance matrix of the centroid-free point sets is constructed, and the optimal rotation matrix is ​​directly solved by SVD decomposition. Combining the centroid deviation of the two sets of points, the translation vector is recovered to obtain the initial values ​​of the extrinsic parameters.

[0021] [4] Monocular depth map estimation and normalization: Obtain dense predicted depth maps and sparse point cloud depth maps with physical scale to provide a data foundation for constructing cross-modal depth constraints.

[0022] [4.1] After obtaining the initial values ​​of the extrinsic parameters in [3], the monocular depth map estimation and normalization process is entered: the monocular depth estimation network DepthPro is used to infer the dense predicted depth map from the camera image. This model is based on the multi-scale Vision Transformer architecture, uses "normalized inverse depth" as the intermediate prediction representation, solves the camera field of view through the parallel focal length estimation branch, and uses the following formula to map the relative depth to the absolute physical scale depth. : ; In the formula, To estimate the focal length, Image width, To standardize the inverse depth.

[0023] [4.2] Initial values ​​of external parameters obtained using coarse calibration and camera internal parameters K Projecting the LiDAR point cloud onto the image plane generates a sparse true depth map: for any point in the point cloud... Transform to the camera coordinate system using the following formula: ; Through camera intrinsic parameter matrix K Projecting points in the camera coordinate system onto the image plane yields the image coordinates. The formula is as follows: ; In the formula, The scaling factor; during projection, to address occlusion issues, only the minimum depth value within the same pixel grid is retained. The point that is greater than 0 and the smallest.

[0024] 【4.3】Robust normalization is performed on the predicted depth map and the point cloud depth map: Calculate the 5th percentile P5 and the 95th percentile P95 of the effective depth values ​​of both, and map the depth values ​​to the [0,1] interval using the following formula to eliminate the influence of outliers, where, To avoid the minimum value where the denominator is 0.

[0025] ; [5] Construction of cross-modal depth constraints (NID constraints): Quantify the distribution similarity between the predicted depth map and the point cloud depth map, and construct cross-modal information constraints that are resistant to modal differences and environmental interference.

[0026] [5.1] After completing the processing in [4], the normalized predicted depth map (X) and point cloud depth map (Y) are obtained, and then cross-modal depth constraints are constructed. The two depth maps are... Quantified as Given a histogram interval, calculate the joint probability distribution of the two. and marginal probability distribution .

[0027] 【5.2】Calculating information entropy and joint entropy based on probability distribution: ; ; ; 【5.3】Calculating Mutual Information A normalized mutual information distance (NID) constraint term is constructed, and the smaller the value, the higher the structural consistency between the two depth maps: ; [6] Joint optimization and extrinsic parameter refinement: By integrating geometric constraints and deep information constraints, the optimal extrinsic parameters are solved through an adaptive weighting strategy and a nonlinear optimization algorithm.

[0028] [6.1] After completing [5], combined with the geometric feature point set extracted in [1] and [2], we enter the stage of multi-constraint joint optimization and extrinsic parameter fine solution. Constructing geometric constraint terms: given N right( N =8) Calculate the geometric root mean square error of the geometric feature point set extracted in the coarse calibration stage. : ; In the formula, and The first and second coordinate systems are respectively the lidar and camera coordinate systems. For feature points with the same name, Let be the extrinsic parameters of the Lie algebra form to be optimized.

[0029] 【6.2】Constraint Normalization: Assume that the lidar ranging error follows a Gaussian distribution. ,in = 0.02 m represents the ranging accuracy of the lidar, where the geometric error is converted into a dimensionless geometric constraint term. : ; 【6.3】Constructing the overall objective function: Introducing weight parameters and Adjusting the contributions of the two types of constraints, the overall objective function is as follows: ; The adaptive strategy for weights is as follows: when When the value is greater than 1, a "deep information guidance" strategy is adopted, and settings are made as follows: = 2, = 1; When 0.5 < When ≤ 1, a "cooperative optimization" strategy is adopted, and the following settings are made: = 1, = 1; when When ≤ 0.5, a "precision locking" strategy is adopted, and the settings are... = 0.5, = 1.

[0030] 【6.4】Nonlinear optimization solution: The Powell conjugate direction method is used to solve the total objective function without derivatives. By constructing conjugate search directions, the multidimensional optimization is decomposed into a one-dimensional linear search, which efficiently approximates the global minimum and obtains the optimal extrinsic parameters.

[0031] Compared with the prior art, the present invention has the following advantages: 1) In the coarse calibration stage, this invention does not require the provision of initial values ​​of external parameters. For the pre-extracted set of corresponding 3D points, the Iterative Closest Point (ICP) algorithm based on Singular Value Decomposition (SVD) is used to quickly solve for the initial values ​​of external parameters and efficiently obtain the global optimal solution. This provides a reliable initial estimate for subsequent fine calibration and effectively avoids the problem of optimization getting trapped in local optima due to improper selection of initial values ​​in traditional methods.

[0032] 2) This invention introduces a monocular depth estimation method in the fine calibration stage to achieve accurate alignment between 2D images and 3D point clouds, effectively breaking through the modal barrier between point clouds and images, and providing a new technical path for constructing cross-modal feature associations. This method can complete the association of depth information between images and point clouds without human intervention, and its depth feature-based association mechanism significantly reduces the strong dependence of traditional targetless calibration methods on environmental texture and lighting conditions.

[0033] 3) This invention proposes a joint constraint optimization architecture that integrates geometric constraints and depth information constraints, constructing a more robust optimization objective function. The distance between corresponding 3D point sets is used as a geometric constraint to ensure the consistency of rigid body motion and guarantee that the calibration results conform to the laws of physical space. Normalized mutual information distance is used as an information constraint to verify the reliability of feature matching by quantifying the similarity of information distribution between the depth measured by LiDAR and the depth predicted by monocular images. This bidirectional constraint design effectively avoids the limitations of single constraints, solving the problem of susceptibility to insufficient feature extraction accuracy and noise interference when relying solely on geometric features, and reducing the direct impact of unilateral errors on the calibration results. Finally, high-precision solution of extrinsic parameters is achieved through joint optimization. Attached Figure Description

[0034] Figure 1 This is a flowchart of the overall technical process of the present invention, which includes two core stages: coarse calibration and fine calibration.

[0035] Figure 2 This is a flowchart of the image feature extraction process (ArUco detection - pose estimation - target extraction).

[0036] Figure 3 The flowchart for lidar point cloud feature extraction (plane fitting - plane extraction - target extraction).

[0037] Figure 4 This is a schematic diagram of the DepthPro monocular depth estimation network structure.

[0038] Figure 5A schematic diagram and dimensional parameters for the ArUco circular hole composite calibration plate.

[0039] Figure 6 This is a schematic diagram of the sensor device installation.

[0040] Figure 7 A comparison of the edge distribution and joint distribution of different features (grayscale-intensity and predicted depth-point cloud depth).

[0041] Figure 8 The NID distribution surface plots under different rotational perturbations demonstrate the convergence of the depth constraint.

[0042] Figure 9 A comparison chart of the mean error of calibration in each direction for different methods.

[0043] Figure 10 This is a comparison of the point cloud-image alignment effect before and after the fine calibration of the method presented in this paper.

[0044] Figure 11 The diagram shows the rotation and translation calibration error distributions of two information constraint methods based on grayscale-intensity and predicted depth-point cloud depth under different offset levels of initial extrinsic parameters.

[0045] Figure 12 The histograms show the distribution of NID values ​​before and after calibration for two information constraint methods: gray-intensity and predicted depth-point cloud depth. Detailed Implementation

[0046] The present invention will now be further described with reference to the accompanying drawings.

[0047] Reference Figure 1 The specific implementation steps of the present invention are as follows: Step 1: For the 3D feature point set of the image, this invention adopts an image feature point detection and 3D coordinate calculation method based on the fusion of ArUco marker array and geometric prior information. The aim is to accurately extract key feature points such as the center of circles and calibration plate vertices of known structural layouts from a single frame image and map them uniformly to the camera coordinate system. The overall feature extraction process is as follows: Figure 2 As shown. First, the input image is detected using a predefined ArUco encoding dictionary to extract the marked edge corners, and a sub-pixel thinning algorithm is introduced to obtain high-precision two-dimensional corner coordinates. Simultaneously, a local coordinate system (Z-axis perpendicular to the plate surface) is constructed based on the calibration plate plane, and the theoretical three-dimensional coordinates of each marked corner point in this coordinate system are established according to the physical design parameters of the calibration plate. Using the projection correspondence between the two-dimensional detected corner points and their three-dimensional coordinates, the pose parameters (rotation matrix) of the calibration plate relative to the camera are calculated using the Perspective-n-Point (PnP) algorithm. Translation vector ).

[0048] Similarly, the precise three-dimensional positions of the center and contour vertices in the local coordinate system of the calibration plate can be determined based on geometric parameters. The obtained pose parameters are then uniformly mapped to the camera coordinate system. ; In the formula, This refers to the three-dimensional feature coordinates in the camera coordinate system. Through this transformation, the three-dimensional coordinates of the center and vertex features in the camera coordinate system can be accurately obtained.

[0049] Step 2: For point cloud feature extraction, the overall process is as follows: Figure 3 As shown. Due to the high redundancy of the original point cloud, this invention first reduces the data dimensionality through voxel downsampling, and then uses the Random Sample Consensus (RANSAC) algorithm to extract the main geometric planes. Combined with clustering algorithms and point picking, the calibration plate plane is extracted and projected into two-dimensional space through coordinate alignment. Subsequently, the center of the circular hole is located through boundary clustering and RANSAC circle fitting, and false targets are eliminated using prior radius constraints.

[0050] Since calibration board vertices are often missing due to sparse point clouds or edge effects, this invention proposes a hybrid optimization algorithm combining Principal Component Analysis (PCA) and ideal geometric constraints. First, PCA is used to perform dimensionality reduction analysis on the calibration board edge point cloud extracted by the aforementioned method, constructing a local coordinate system with the centroid as the origin and the eigenvector directions as the basis vectors. This step provides the initial translation vectors for subsequent optimization. and rotation angle Subsequently, an ideal rectangular mathematical model was established based on the physical dimensions of the calibration plate, and the set of parameters to be optimized was defined. Constructing model edge sampling points Compared with actual observation points The objective function constrained by the nearest neighbor distance is: ; The Nelder-Mead algorithm is used to minimize the distance residual between the ideal rectangular model and the observed point cloud. M This represents the number of sampling points. To balance observational accuracy with model integrity, this invention introduces a distance threshold. = 0.02 m. For the optimized ideal vertex, if the nearest neighbor observation point is... If the observed point is correct, it is considered the true vertex; otherwise, the optimized rectangular model vertices are used for virtual completion. This strategy effectively solves the problems of missing points or edge distortion in the point cloud, ensuring the geometric consistency of the features.

[0051] Step 3: After obtaining the image feature point set in the camera coordinate system Point cloud feature point set in lidar coordinate system Then, the optimal rigid body transformation parameters (rotation matrix) need to be solved. With translation vector This invention aims to minimize the sum of squared Euclidean distances between two paired point sets. Based on the ICP registration method, the objective function is constructed as follows: ; For the aforementioned least squares problem, a commonly used closed-form solution based on SVD decomposition is employed. The core idea of ​​this method is to decouple the rotation and translation solutions: first, the geometric centroids of the two point sets are calculated and centered to eliminate the influence of the translation; then, the cross-covariance matrix of the centroid-free point sets is constructed, and the optimal rotation moment is directly obtained through SVD decomposition; finally, the translation vector is recovered by combining the centroid deviation. This algorithm obtains a unique analytical solution without iteration, providing highly reliable initial values ​​of extrinsic parameters for subsequent fine calibration.

[0052] Step 4: To measure the data consistency between the LiDAR and the camera at the dense pixel level, this invention utilizes the monocular depth estimation network DepthPro to pre-infer a dense predicted depth map from the camera image. This model is based on a multi-scale Vision Transformer architecture, employs "normalized inverse depth" as an intermediate prediction representation, and directly calculates the camera's field of view from image features through a parallel focal length estimation branch. The network structure is as follows: Figure 4 As shown. Ultimately, the model maps relative depth to absolute depth with absolute physical scale based on the geometric relationship shown in the following equation. , in To estimate the focal length, Image width, To standardize the inverse depth.

[0053] ; At the same time, the initial external parameters obtained from the coarse calibration are used and camera internal parameters The point cloud is projected onto the image plane to generate a sparse, true depth map. For any point in the point cloud... Its coordinates in the camera coordinate system and image plane coordinates The calculation is as follows: ; ; in This is the scaling factor. During projection, to address the occlusion problem between objects, only the object with the smallest depth value falling within the same pixel grid is retained. Minimum and Points with depth values ​​greater than 0 were identified. Finally, robust normalization was performed on both the predicted depth map and the point cloud depth map. The 5th percentile of the effective depth values ​​for both was calculated. and 95th percentile The following formula is used to map depth values ​​to the [0, 1] interval to eliminate the influence of outliers: ; Since there is a significant difference in numerical scale between the actual depth measured by lidar and the predicted depth by the depth estimation network, directly using metrics such as root mean square error (MSE) per pixel is not suitable. Therefore, this invention uses normalized mutual information distance (NID) based on information theory as a similarity measure. NID is highly robust to modal differences and can effectively measure the statistical correlation between two different distributions.

[0054] The normalized depth map described above is quantized as follows: Given a histogram interval, calculate the joint probability distribution of the two. and marginal probability distribution From this, we can obtain the information entropy. With joint entropy : ; ; ; Mutual Information Defined as Ultimately, the normalized deep mutual information constraint... The value is defined as follows: the smaller the value, the higher the structural consistency between the two depth maps. ; Step 5: Since geometric reprojection error has a meter-level physical dimension, while normalized mutual information distance (NID) is a dimensionless statistical information quantity, direct linear superposition will cause the optimization gradient to be significantly disturbed by the difference in numerical magnitude. To address this problem, this invention first introduces the lidar measurement noise prior as a standardization benchmark and constructs a joint objective function based on statistical significance. Specifically, given... N right( N =8) The geometric root mean square error of the geometric feature point set extracted in the coarse calibration stage. Defined as the statistical value of the Euclidean distance between a LiDAR feature point and its corresponding image feature point after being transformed to the camera coordinate system using extrinsic parameters, the calculation formula is as follows: ; in, and The first and second coordinate systems are respectively the lidar and camera coordinate systems. For feature points with the same name, Let be the extrinsic parameter in the Lie algebraic form to be optimized. To eliminate dimensional differences, it is assumed that the lidar ranging error follows a Gaussian distribution. ,in The ranging accuracy of the lidar is set at 0.02 m in this invention. The geometric error is transformed into a geometric distance constraint term that is a dimensionless ratio relative to the sensor noise. : ; This metric physically represents the standard deviation multiple of the current geometric residual relative to the sensor's inherent noise, thus achieving normalized alignment of geometric constraints and NID constraints in the numerical dimension.

[0055] Step 6: Based on this, define the overall objective function: ; in, and Weighting parameters for adjusting the contribution of geometric accuracy and information consistency.

[0056] To ensure the optimal convergence path despite fluctuations in feature point extraction quality, this paper proposes an adaptive weighting strategy based on initial geometric residual evaluation for setting the weighting parameters. Before performing nonlinear optimization, the geometric constraint term is... It is used as a criterion for feature credibility to configure weights.

[0057] When the initial When this occurs, it indicates that the feature point pairs contain significant noise, and the reliability of the initial geometric values ​​is low. In this case, the system adopts a "depth information-guided" strategy, assigning dominant weights to the NID constraint terms ( This utilizes depth information constraints to guide extrinsic parameters away from erroneous local extrema caused by insufficient accuracy in geometric feature extraction. This indicates that the geometric error is within a reasonable fluctuation range of sensor noise, and the feature quality is reliable but still requires fine alignment. The system adopts a "co-optimization" strategy, assigning equal weights to both ( This method, while limiting the search space based on geometric features, also utilizes depth information to correct biases. When When the result is high, it indicates that the geometric feature extraction accuracy is high. At this point, a "precision locking" strategy is adopted, reducing the NID weight ( This is to prevent statistical noise in the predicted depth map from interfering with the achieved sub-centimeter geometric registration accuracy.

[0058] Given that the NID term in the objective function involves discrete histogram statistics and joint entropy calculation, resulting in a highly nonlinear and non-differentiable objective function in the parameter space, traditional gradient-based optimization algorithms are no longer applicable. Therefore, this invention employs the Powell conjugate direction method for derivative-free iterative solution. The core idea of ​​this algorithm is to solve the problem without explicitly calculating the Jacobian matrix by... N A set of conjugate search directions is constructed in the parameter space, decomposing the multidimensional optimization problem into a series of one-dimensional linear search subproblems. During the optimization process, the algorithm dynamically updates the set of conjugate directions using a bidirectional search strategy to adapt to the local topological structure of the objective function, thereby efficiently approximating the global minimum. Through the above joint optimization mechanism, the system not only ensures the geometric alignment of key feature points on the calibration board, but also fully utilizes the deep structural information of the environmental background for fine-tuning, enabling the extrinsic parameter calibration results to possess both physical geometric accuracy and cross-modal structural consistency.

[0059] The effectiveness of the method of the present invention can be illustrated by the following experiments: 1. Experimental Environment This experiment utilizes the Xiaomi Mini PC hardware platform and runs on the Ubuntu 20.04 operating system. A customized ArUco-marked composite calibration board with circular holes was fabricated, with specifications as follows: Figure 5 As shown in Figure 6, the sensor system consists of a Livox Mid-360 hybrid semi-solid-state LiDAR and a GC2093 USB camera. The camera is integrated into the AlgoT1 integrated navigation device from Shanghai Algebraic Rhythm Co., Ltd., while the LiDAR is rigidly fixed to the top of the device. During data acquisition, given the non-repeating scanning characteristics of LiDAR, the raw point cloud data can be acquired simply by keeping the device stationary and collecting data for a few seconds. For synchronous image acquisition, the sensors must be rigidly connected and relatively stationary; a single image can be selected within the point cloud acquisition timeframe.

[0060] 2. Experimental Results To evaluate the feasibility of depth information constraints in camera-LiDAR extrinsic parameter calibration methods, a comparative study was first conducted on the statistical distribution characteristics of multimodal data. Experiments were performed to statistically analyze the distribution patterns of image grayscale and point cloud reflectance intensity, and the predicted and measured depths of the image and LiDAR, respectively, under the true values ​​of the extrinsic parameters (e.g.,...). Figure 7As shown in the figure. Normalized mutual information distance is used as a quantitative evaluation index to describe the correlation between feature distributions, aiming to verify the inherent consistency of heterogeneous data in geometric and physical properties from a statistical perspective, providing theoretical support for the proposed method. Experimental results show that, under the true external parameters, the joint distribution of image grayscale and point cloud intensity exhibits significant dispersion (…). Figure 7 (1) The NID is as high as 0.889, reflecting that the correlation between the two types of signals is still weak even when geometrically aligned, making it difficult to form a robust constraint. The root cause lies in the essential difference between the imaging mechanism and the spectral response: the image gray level is affected by the diffuse reflection of visible light and the ambient light source, and the laser reflection characteristics of different materials in different bands are different, resulting in a large number of samples with inconsistent cross-modal statistical distributions. Therefore, the statistical correlation based on gray level information is highly dependent on specific scenes and lacks universality. In contrast, the joint distribution of image prediction depth and point cloud depth shows a clear linear positive correlation ( Figure 7 (2) The NID is approximately 0.516, demonstrating a high degree of consistency. The physical essence of depth is the three-dimensional spatial distance from the object to the sensor, and this geometric consistency constitutes the objective basis for cross-modal data fusion. Therefore, depth features avoid interference from spectral and material properties, and their correlation mechanism originates from the three-dimensional structure itself, possessing stronger reliability and versatility. Using this as the core optimization objective can significantly improve the accuracy and robustness of camera-LiDAR extrinsic parameter calibration.

[0061] To further verify the effectiveness of depth information difference as the objective function for optimization, a set of extrinsic parameter perturbation experiments were designed to systematically verify the variation law of NID value when the extrinsic parameters deviate from the true value. Specifically, based on the known true extrinsic parameters, different degrees of angular perturbation (±10°) were applied along the three rotational degrees of freedom of roll, pitch, and yaw. The NID value between the image predicted depth and the point cloud measured depth under each perturbation combination was calculated, and the variation surface was plotted.

[0062] like Figure 8 As shown, NID exhibits a "quasi-inverted cone" topological characteristic with increasing extrinsic parameter perturbation. This indicates that as the extrinsic parameters approach the true value, NID tends to the global minimum, and the joint distribution of the two modal depths shows a strong correlation. With the introduction of rotational bias, NID monotonically increases along each perturbation axis, demonstrating excellent parameter discrimination performance. This shows that the objective function constructed based on depth information constraints has good convergence robustness. Therefore, using depth information NID constraints as the optimization criterion can effectively guide the optimization process to converge to the global minimum within a relatively wide initial parameter domain.

[0063] To verify the extrinsic parameter calibration accuracy of the method proposed in this invention, seven sets of camera-LiDAR data under different observation poses were collected for testing. The comparison schemes covered three traditional single-target feature extraction methods in the coarse calibration stage and the two-stage calibration method proposed in this paper. The three traditional methods are classic schemes that rely solely on single target features for extrinsic parameter calculation, without additional constraint optimization mechanisms. Specifically, the method based solely on the center point set corresponds to the FAST-Calib method mentioned in the background section. This method calculates extrinsic parameters only by extracting the target center features and is a mature existing traditional calibration scheme. The method based solely on the vertex point set is a supplementary comparison scheme to the traditional single-feature calibration method. The point feature SVD-ICP method based on the center and vertex point sets is the scheme used in the coarse calibration stage of the two-stage calibration method in this paper. Its essence still belongs to the category of traditional point feature matching, which calculates extrinsic parameters by fusing the geometric features of the target center and vertices using the SVD-ICP algorithm. The average calibration errors of each method in all experiments are shown in Table 1 and [Table 2 missing in original text]. Figure 9 As shown.

[0064] Table 1 and Figure 9 The results show that the initial extrinsic parameters provided in the coarse calibration stage can converge the error to a smaller range, with the rotational extrinsic parameters approaching the global optimum. However, the translational amount (especially in the Y-axis direction) is quite sensitive to the accuracy of point feature extraction. After fine calibration by the algorithm in this paper, the extrinsic parameter errors show good convergence consistency. On the translation vector, the overall average error in each axis is reduced to 0.01 m, significantly better than the comparative algorithm. On the rotation vector, the overall average error is stabilized at a high accuracy level of 0.941°. In summary, the method in this paper significantly optimizes the calibration accuracy of the translation component while taking into account the accuracy of rotation calibration, and has better overall calibration performance. The core reason is that relying solely on point feature matching for extrinsic parameter calculation is susceptible to interference from laser point cloud edge scattering and environmental noise, leading to feature extraction deviations that are directly used in the calculation results. In contrast, the method in this paper uses the point feature calculation results as initial values ​​and further introduces depth information to construct information constraints, effectively correcting the extrinsic parameter drift caused by feature extraction errors and achieving higher accuracy spatial alignment.

[0065] Table 1. Mean error of calibration using different methods also, Figure 10 The point cloud coloring effect visually confirms the necessity of precise calibration. In some experiments where the feature extraction accuracy was poor, the image, after projection onto the extrinsic parameters, showed a significant offset from the point cloud, resulting in the target contour not accurately coinciding (e.g., ...). Figure 10(As shown in the red box). After correction by the fine calibration algorithm of this invention, the point cloud and image data are accurately aligned. This further proves that even when the point feature extraction quality is poor, the fine calibration stage can still compensate for the defects of single feature matching by constraining the global geometric information of depth information, ensuring high-precision extrinsic parameters.

[0066] To further investigate the sensitivity of traditional grayscale-based information constraints and the depth-based information constraints proposed in this invention to initial extrinsic parameter deviations, based on the coarse calibration results, the calibration performance of grayscale-based and depth-based schemes was compared by applying multiple levels of perturbation proportionally along different axes. This experiment aims to quantify the calibration error under different offset levels, thereby evaluating the convergence radius of the objective function and its ability to guide extrinsic parameters towards the true value within a wide parameter space.

[0067] from Figure 11 It can be seen that the calibration error distribution based on grayscale information exhibits high uncertainty and significant fluctuations: when the rotation offset is greater than 4°, the Roll axis angle error oscillates dramatically between 1° and 4°; in the translation direction, the Y-axis shows an extreme deviation of more than 20 cm at an 8 cm offset. This disordered fluctuation indicates that grayscale and intensity features are difficult to form a smooth gradient under non-precise alignment, resulting in a limited convergence radius of the objective function and a tendency for the extrinsic parameter results to diverge when a large offset occurs. This further confirms the above argument that, due to the influence of imaging principles and differences in spectral response, this feature cannot provide stable information constraints.

[0068] In contrast, the calibration results based on depth information methods exhibit significant robustness. Their error stability is high: the angular errors of all three axes remain stable within 1.5°; the translational errors of the X and Y axes fluctuate by less than 1 cm, and the Z-axis error only slightly fluctuates within 3 cm. Experimental results further confirm the "quasi-inverted cone" topological feature generated by depth geometric consistency: even with significant deviations in initial values, depth constraints still provide monotonic and clear optimization guidance, ensuring that the calibration process achieves high-precision convergence and effectively compensating for the insufficient accuracy of point feature extraction in the coarse calibration stage.

[0069] To directly verify the superiority of depth constraints from a statistical perspective, Figure 12 The differences in NID value distribution before and after calibration were compared between the two constraint schemes. Figure 12 (Left) As can be seen, before calibration, the NID value based on the grayscale information constraint method was in an extremely high range (0.95-1.0), indicating poor correlation between cross-modal features; after fine calibration, although the NID value decreased slightly, it still remained above 0.85. This reflects the limitation of appearance features being easily affected by lighting, material reflection, and texture loss. Figure 12(Right) After fine calibration, the NID value based on the depth information constraint method significantly decreased from 0.85-0.90 to 0.5-0.7, and the distribution tended to be concentrated. This indicates that the depth constraint effectively guides the extrinsic parameters to be corrected to the optimal parameters, enabling a high degree of matching between the cross-modal depth distribution and significantly improving the similarity between the distribution of the predicted depth in the image and the measured depth in the point cloud.

[0070] In summary, the core advantage of the "coarse calibration + joint constraint fine calibration" method proposed in this invention lies in using depth information constraints instead of traditional grayscale information constraints to construct the optimization architecture. Experimental results fully demonstrate that depth constraints have stronger robustness and stability compared to traditional grayscale constraints, effectively overcoming the limitations of grayscale constraints that are susceptible to illumination, material, and extrinsic parameter offsets. Furthermore, by combining geometric constraints, it not only solves the problem of insufficient feature extraction accuracy and noise interference when relying solely on geometric features, but also corrects initial extrinsic parameter deviations through the precise fine-tuning effect of depth constraints, ensuring high-precision calibration in complex environments.

Claims

1. A method for calibrating the extrinsic parameters of a lidar-camera system based on monocular depth assistance, characterized in that, Includes the following steps: [1] Image feature extraction: Based on the fusion of ArUco marker array and geometric prior information, the center of the circular hole and the contour vertex feature points of the calibration plate are extracted from a single frame image. The pose parameters of the calibration plate relative to the camera are calculated by the perspective N-point PnP algorithm, and the feature points are mapped to the camera coordinate system to obtain the image feature point set. ; [2] Point cloud feature extraction: The original point cloud of the lidar is voxel downsampled, and the calibration plate plane is extracted by the Random Sampling Consensus (RANSAC) algorithm. The center of the circular hole is located by combining boundary clustering and RANSAC circle fitting. The missing calibration plate vertices are virtually completed by using a hybrid optimization algorithm of principal component analysis (PCA) and ideal geometric constraints to obtain the point cloud feature point set. ; [3] Coarse calibration of extrinsic parameters based on SVD-ICP: constructing a minimum objective function and The sum of squared Euclidean distances after rigid body transformation is used to solve for the rotation matrix using a closed-form method of singular value decomposition (SVD). With translation vector We obtain the initial values ​​of the external parameters; [4] Monocular depth map estimation and normalization: The monocular depth estimation network DepthPro is used to infer dense predicted depth maps from camera images. Combined with the initial values ​​of extrinsic parameters and camera intrinsic parameters, the lidar point cloud is projected onto the image plane to generate sparse real depth maps. Robust normalization is performed on the two types of depth maps. [5] Construction of cross-modal depth constraints: The normalized depth map is quantized into histogram intervals, the joint probability distribution and marginal probability distribution are calculated, and the normalized mutual information distance NID constraint term is constructed through information entropy, joint entropy and mutual information. ; [6] Multi-constraint joint optimization and external parameter fine solution: Constructing geometric constraint terms and The joint objective function is obtained by configuring the weight parameters using an adaptive weighting strategy based on the initial geometric residual evaluation, and then iteratively solving the problem using the Powell conjugate direction method to obtain the optimal extrinsic parameters.

2. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The formula for mapping the feature points to the camera coordinate system in step [1] is: ； in, The rotation matrix of the calibration board relative to the camera, It is a translation vector. The three-dimensional positions of the center of the circular hole and the vertex of the contour in the local coordinate system of the calibration plate.

3. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The objective function of the hybrid optimization algorithm described in step [2] is: ； In the formula, For the parameter set to be optimized, These are the edge sampling points of the ideal rectangular model. For point cloud observation by lidar, M The number of sampling points is used; the Nelder-Mead algorithm is employed to minimize the objective function, and a distance threshold is introduced. = 0.02 m, virtual completion is performed on missing vertices.

4. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The objective function mentioned in step [3] is: ； By calculating and centering the geometric centroids of the two point sets, constructing the cross-covariance matrix, and then solving for the optimal rotation matrix through SVD decomposition, the translation vector is recovered by combining the centroid deviation, and the initial values ​​of the extrinsic parameters are obtained.

5. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The formula for robust normalization in step [4] is: ； In the formula, The 5th percentile of the effective depth value. It is at the 95th percentile. To avoid the minimum value where the denominator is 0; Absolute depth The calculation formula is: ； in, To estimate the focal length, Image width, To standardize the inverse depth.

6. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The NID constraint term mentioned in step [5] The calculation formula is: ； In the formula, , For information entropy, For joint entropy, For mutual information, X To predict the depth map, Y This is a point cloud depth map.

7. The monocular depth-assisted lidar-camera extrinsic parameter joint optimization and calibration method according to claim 1, characterized in that: The joint objective function mentioned in step [6] is: ； in, , The geometric root mean square error, = 0.02 m is the ranging accuracy of the lidar; the adaptive weighting strategy is: > 1 hour = 2, = 1; 0.5 < ≤ 1 = 1, = 1; ≤ 0.5 = 0.5, = 1.

Images