Lidar-camera calibration method based on dual-view feature and multi-scale feature matching
Patent Information
- Application Number
- CN202610420585.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-01
- Publication Date
- 2026-10-16
AI Technical Summary
[0005](2)缺乏多视角特征的有效融合，场景特征提取质量有待提高
[0015] The beneficial effects of this invention are as follows: It employs a collaborative mechanism of normal vector denoising and distance-driven inlier back-judgment to suppress outlier interference while preserving edge features; the feature fusion of the forward field of view and bird's-eye view overcomes the dependence of traditional single-view calibration on scene structure, improving the quality of scene feature extraction; and based on adaptive threshold adjustment of point cloud density and image sharpness, it constructs a multi-scale progressive matching strategy from coarse to fine, improving the quality of 2D-3D feature matching. This invention achieves significant reductions in both rotation and translation errors, and its calibration accuracy comprehensively surpasses current mainstream LiDAR-camera calibration methods. Furthermore, the method of this invention exhibits excellent cross-scene and hardware device generalization capabilities, possessing high engineering practical value.
Smart Images
Figure CN122928983A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-sensor fusion, and in particular to a lidar-camera calibration method based on the matching of dual-view features and multi-scale features. Background Technology
[0002] LiDAR and cameras are widely used in autonomous navigation scenarios such as autonomous driving, robotic inspection, and drone mapping. LiDAR data provides high-precision geometric structure information, while camera data provides rich texture information. Fusing the data from both can significantly improve the ability to perceive target objects. The quality of data fusion directly affects the performance of subsequent tasks such as perception, localization, and decision-making, and is the foundation for building a reliable multimodal perception and navigation system.
[0003] Achieving data fusion from LiDAR and cameras requires establishing a geometric transformation relationship between their coordinate systems, i.e., calibrating the parameters of the LiDAR and camera. Feature-based calibration methods are currently the mainstream approach in practical applications. Based on the type of feature, this category can be further divided into three types: edge features, geometric features, and depth features. Edge features are susceptible to noise and viewpoint changes, and errors may occur during matching due to edge breaks or deformation. While depth features utilize the depth information of point clouds, it is difficult to establish reliable correspondences when the point cloud is sparse or the depth information is incomplete. Therefore, from the perspective of generalization and calibration accuracy, geometric feature-based calibration methods remain the most widely used approach.
[0004] Existing geometric feature-based lidar-camera calibration methods have achieved significant improvements in accuracy, but still face the following challenges: (1) The lack of a noise reduction and fault-tolerant recovery mechanism means that the quality of point cloud preprocessing needs to be improved. Existing methods for point cloud denoising generally use the angle between normal vectors as the criterion and a fixed threshold to identify noise points. When the threshold is set too high, a large number of noise points remain. To ensure denoising effectiveness, the threshold is usually set too low, which effectively removes noise but can easily lead to the accidental deletion of interior points, resulting in compromised point cloud integrity. Therefore, current research needs to establish an effective fault-tolerant recovery mechanism to recover accidentally deleted interior points, thereby balancing denoising effectiveness and point cloud integrity.
[0005] (2) The lack of effective fusion of multi-view features means that the quality of scene feature extraction needs to be improved. Existing methods typically extract features from the front-view perspective. While this yields rich texture details, it is susceptible to interference from factors such as occlusion, lighting fluctuations, and viewpoint shifts, making it difficult to guarantee the integrity and discriminability of the features. A top-down view, on the other hand, can acquire global spatial geometric information and structured scene information. Although a few works have attempted feature fusion from these two perspectives, they mostly employ simple concatenation or fixed-weight fusion strategies, resulting in limited improvement in feature extraction quality. Therefore, current research lacks an effective method for feature extraction and fusion of calibration scenes from multiple perspectives to obtain high-quality calibration scene features.
[0006] (3) The lack of consideration for scene feature density means that the quality of 2D-3D feature matching needs to be improved. Existing methods often employ fixed thresholds for 2D-3D feature matching. When the scene has a high density of effective features, these methods are prone to introducing erroneous matching pairs with similar features, interfering with the stability of subsequent optimization. Conversely, when scene features are sparse, they may mistakenly filter out legitimate matching pairs, resulting in insufficient effective interior points and affecting the final calibration quality. Therefore, current research lacks a method that considers scene feature density and utilizes matching thresholds at different scales for high-quality 2D-3D feature matching. Summary of the Invention
[0007] To overcome the aforementioned problems in the prior art, this invention proposes a lidar-camera calibration method based on the matching of dual-view features and multi-scale features.
[0008] The technical solution adopted by this invention to solve its technical problem is: a lidar-camera calibration method based on dual-view feature and multi-scale feature matching, comprising the following steps: Step 1: Acquire camera and LiDAR data. Using RGB images and LiDAR point clouds as input, filter out noise interference and extract effective input features through two-stage point cloud denoising and image preprocessing. Step 2: The dual-view feature extraction module acquires the semantic-geometric features of the front field of view and the spatial-structural features of the bird's-eye view. Through selective feature fusion, it outputs high-quality point cloud features. Step 3: In the multi-scale progressive feature matching module, a multi-scale feature matching method oriented towards density and image quality perception is constructed to obtain high-quality 2D-3D matching point pairs, providing reliable constraints for extrinsic parameter optimization. Step 4: The extrinsic parameter optimization module uses the LM algorithm to solve and optimize the extrinsic parameters, and outputs high-precision extrinsic parameters. The two-stage point cloud denoising in step 1 specifically consists of: normal vector angle denoising and distance optimization recovery.
[0009] The aforementioned LiDAR-camera calibration method based on dual-view feature and multi-scale feature matching, specifically the normal vector angle denoising in step 1, involves: denoising the points in the original point cloud... Take it - Nearest neighbors: Use PCA to calculate the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the normal vector of that point. Then fit the main axis of the scene. Through calculation and The included angle ,Will Greater than the threshold The points are marked as noise points: .
[0010] The aforementioned LiDAR-camera calibration method based on dual-view feature and multi-scale feature matching, specifically the distance optimization recovery in step 1, involves: for a point... The KDTree method is used to generate a product with... The set of nearest neighbors ;calculate and Average distance of middle nodes : = ; in, for interior point in, for and Euclidean distance; if < Points identified as interior points are restored; multiple point clouds are transformed to the world coordinate system, and point cloud registration and fusion are performed to filter out duplicate points and retain high-quality point clouds.
[0011] The aforementioned LiDAR-camera calibration method based on dual-view feature and multi-scale feature matching, specifically the image preprocessing in step 1, involves: projecting the point cloud onto the image plane, statistically analyzing the proportion of projected points falling within the semantic segmentation region, and using this proportion as the consistency score for that frame. This consistency score reflects the degree of consistency between the point cloud and the image's semantic information, and is used for subsequent optimization of the confidence weight of the objective function; converting the static region image into a grayscale image, calculating its Sobel gradient in the x-axis and y-axis directions, and accumulating the gradient magnitudes to obtain the sharpness G. The normalized image sharpness G is used for threshold selection in the subsequent 2D-3D matching stage. The formula for calculating the consistency score is as follows: ; in, This represents the number of projection points within the semantic segmentation region. This represents the total number of valid projected points. The formula for calculating image sharpness G is: = ; in, W , H These are the image width and height, respectively. and The images are respectively in Sobel gradient response in the x and y directions at a pixel.
[0012] In the aforementioned lidar-camera calibration method based on dual-view feature and multi-scale feature matching, step 2 specifically comprises: Step 2.1: Under the forward field of view, extract image feature points using the ORB algorithm, represented as... Geometric feature vectors are extracted based on the point cloud projection depth map, and are represented as follows: ,Will and Perform vector concatenation to form the front field of view feature vector. ; Step 2.2: After preprocessing such as denoising and ground segmentation, the point cloud is projected onto a horizontal plane and mapped onto an image grid at a fixed resolution to obtain a bird's-eye view. From this bird's-eye view, the normalized density features of the grid point cloud are constructed, represented as... The average height feature is calculated based on the height of the grid point cloud, and is expressed as follows: According to the grid point cloud x shaft and y The spatial distribution statistical characteristics are obtained by analyzing the distribution along the axis, and are represented as follows: Using a denoised point cloud, for each point, its k nearest neighbors are first retrieved and principal component analysis is performed to estimate the normal vector. The mean and standard deviation of the normal vectors of all points are then concatenated to represent the result. ,Will , , , Vector concatenation is performed to form the feature vector of the bird's-eye view. ; Step 2.3, using The activation function retains negative values, resulting in the final front field feature vector. and bird's-eye view feature vector : = ; = ; Step 2.4, and Fusion, resulting in fusion characteristics Multi-view features are obtained through channel attention. : ; = ; in, Indicates the feature weights of the front field of view. Indicates the weight of the bird's-eye view features. express Features and A unified dimension for features.
[0013] In the aforementioned lidar-camera calibration method based on dual-view feature and multi-scale feature matching, step 3 specifically comprises: Step 3.1, determine the matching threshold: select the corresponding scene according to the point cloud density ρ, select the matching degree threshold according to the image sharpness G, divide the scene complexity into three levels, corresponding to dense scene, medium density scene and sparse scene respectively, and set three levels of matching threshold for each scene, namely coarse matching threshold, medium matching threshold and fine matching threshold, and set the reprojection error threshold of geometric consistency constraint at the same time. Step 3.2: Implement coarse-to-fine feature matching using a multi-scale progressive matching strategy: If the number of 2D-3D matching pairs after matching at a certain scale level is more than 10, then proceed to the next scale level for matching. When the number of matching pairs at the end of matching at a certain scale level is less than 10 or when matching at all scale levels is completed, the 2D-3D feature matching ends. The set of matching pairs obtained by this multi-scale progressive matching strategy is denoted as M. Step 3.3, perform geometric consistency constraint checks on the matching pairs in M: for points Using initial extrinsic parameters and Set its coordinates in the point cloud =( , , Mapped to coordinates in the camera coordinate system =( , , ), among which, among which Let be the initial rotation matrix. As the initial translation matrix, the camera intrinsic parameters are used to... The pixel coordinates are obtained by projecting them onto the image plane. ,calculate reprojection error , ,in( , (This is a review) The true coordinates in a 2D pixel coordinate system; for Points in the array are sorted by reprojection error, and those exceeding a specified error threshold are included. The matching pairs corresponding to the points are deleted. During the deletion process, if the number of matching pairs is less than 10, the deletion stops. Finally, a sufficient number of matching pairs that have passed geometric consistency filtering are obtained, denoted as . .
[0014] The aforementioned lidar-camera calibration method based on dual-view feature and multi-scale feature matching, specifically step 4, the extrinsic parameter solution model, is as follows: ; ; in, As the confidence level weight, This is the reprojection error.
[0015] The beneficial effects of this invention are as follows: It employs a collaborative mechanism of normal vector denoising and distance-driven inlier back-judgment to suppress outlier interference while preserving edge features; the feature fusion of the forward field of view and bird's-eye view overcomes the dependence of traditional single-view calibration on scene structure, improving the quality of scene feature extraction; and based on adaptive threshold adjustment of point cloud density and image sharpness, it constructs a multi-scale progressive matching strategy from coarse to fine, improving the quality of 2D-3D feature matching. This invention achieves significant reductions in both rotation and translation errors, and its calibration accuracy comprehensively surpasses current mainstream LiDAR-camera calibration methods. Furthermore, the method of this invention exhibits excellent cross-scene and hardware device generalization capabilities, possessing high engineering practical value. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the present invention; Figure 2 This is a point cloud denoising visualization with a fault-tolerant mechanism in an embodiment of the present invention, wherein (a) is the original camera image, (b) is a bird's-eye view of the original LiDAR point cloud, (c) is the dynamic / noise points detected in the first stage, (d) is the final denoising result after optimization and recovery in the second stage, (e) is the original point cloud in the front field of view, (f) is the front view projection of the denoising effect in the first stage, and (g) is the front view projection of the denoising effect in the second stage. Figure 3 This refers to the feature matching results of two consecutive frames of images in this embodiment of the invention; Figure 4This is a flowchart of the adaptive matching method in an embodiment of the present invention, wherein (A) is adaptive parameter adjustment; (B) is multi-scale progressive matching; and (C) is geometric consistency constraint. Figure 5 This is the 2D-3D feature matching result of a frame of image in an embodiment of the present invention; Figure 6 This is pseudocode for the specific calibration steps in an embodiment of the present invention; Figure 7 These are schematic diagrams illustrating the calibration effects in four typical scenarios according to embodiments of the present invention. Detailed Implementation
[0017] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0018] like Figure 1 As shown, the FTDM-Calib LiDAR-camera calibration method disclosed in this embodiment comprises four modules. The data preprocessing module takes RGB images and LiDAR point clouds as input, and filters out noise interference and extracts effective input features through two-stage point cloud denoising and image preprocessing. The dual-view feature extraction module acquires front-view semantic-geometric features (FV) and bird's-eye view spatial-structural features (BEV), and outputs high-quality point cloud features through selective feature fusion. In the multi-scale progressive feature matching module, a multi-scale feature matching method oriented towards density and image quality perception is constructed to obtain high-quality 2D-3D matching point pairs, providing reliable constraints for extrinsic parameter optimization. The extrinsic parameter optimization solution module uses the Levenberg-Marquardt (LM) algorithm to solve and optimize extrinsic parameters, outputting high-precision extrinsic parameters. The specific steps are as follows: Step 1: Data preprocessing with fault-tolerant response In practical applications, lidar and cameras are susceptible to interference from factors such as dust, reflective spots, and sudden changes in illumination, resulting in noise in the generated point clouds and glare artifacts in the acquired images. To ensure the quality of the calibrated data, point cloud denoising and image enhancement are necessary. In this embodiment, an angle threshold is selected. =2.5°, normal vector neighborhood size =12 is used as the default parameter for point cloud fault-tolerant denoising. =2.5 is used as the default distance threshold.
[0019] Step 1.1, point cloud noise reduction.
[0020] Existing calibration methods typically perform point cloud denoising based on normal vector angles. To ensure denoising effectiveness, these methods usually set relatively strict thresholds. In this case, some key structural points may be mistakenly deleted, leading to feature loss. To address this, this embodiment proposes a point cloud denoising method with a fault-tolerant recovery mechanism. This method combines normal vector denoising with a distance-driven inlier back-judgment mechanism, balancing the problems of inlier deletion and insufficient denoising.
[0021] This method comprises two stages: normal vector angle denoising and distance optimization recovery. The first stage performs denoising based on normal vector angles, effectively identifying and labeling dynamic objects and noise points. The specific process is as follows: For points in the original point cloud... Take it - Nearest neighbors: Use PCA to calculate the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the normal vector of that point. Then fit the main axis of the scene. Through calculation and The included angle (Eq.(1)), will Greater than the threshold The points are marked as noise points.
[0022] (1) The second stage involves distance detection of noise points around the internal points to recover the mistakenly deleted critical structural points. For the points... The KDTree method is used to generate a product with... The set of nearest neighbors Calculate using formula (2) and Average distance of middle nodes .if < If it is determined to be an interior point, it will be restored.
[0023] = (2) in, for interior point in, for and The Euclidean distance was calculated. Finally, the multi-frame point cloud was transformed to the world coordinate system, and point cloud registration and fusion were performed to filter out duplicate points and retain high-quality point clouds.
[0024] Figure 2 The visualization results of the proposed two-stage point cloud denoising method in this paper are shown in a certain scene in the KITTI dataset. Figure 2 (a) is the original camera image. Figure 2 (b) is an aerial view of the original LiDAR point cloud, which shows that there are a large number of dynamic objects and noise interference in the point cloud. Figure 2 (c) Shows the dynamic / noise points detected in the first stage (marked in red). Figure 2 (d) is the final denoising result after the second stage of optimization and restoration. Figure 2 (e) Original point cloud in the front field of view Figure 2 (f) and Figure 2 (g) shows the front view projection of the two-stage denoising effect. It can be seen that the proposed method can effectively remove dynamic objects and noise while preserving key structural points of the static environment.
[0025] Step 1.2, Image Processing.
[0026] The SAM model is used to perform semantic segmentation on camera images to obtain a static region mask. This mask is used not only to identify static regions but also to calculate the consistency score as shown in formula (3). Specifically, after projecting the point cloud onto the image plane, the proportion of projected points falling within the semantic segmentation region is counted and used as the consistency score of the frame data. This score reflects the degree of consistency between the point cloud and the semantic information of the image and is used to optimize the confidence weight of the objective function in the subsequent process.
[0027] (3) In formula (3) This represents the number of projection points within the semantic segmentation region. This represents the total number of valid projection points.
[0028] Subsequently, the static region image is converted to grayscale, and its position is calculated. x The sharpness is obtained by summing the Sobel gradients along the y-axis and y-axis and accumulating the gradient magnitudes. As shown in Formula 4. Normalized image sharpness. Threshold selection for subsequent 2D-3D matching stages.
[0029] = (4) in W , H These are the image width and height, respectively. and The images are respectively in Sobel gradient response in the x and y directions at a pixel.
[0030] Step 2: Multi-view feature extraction and fusion.
[0031] To improve the completeness and recognizability of point cloud features, this paper performs feature extraction and fusion from both the front field of view and the bird's-eye view. In the front field of view, the ORB algorithm is used to extract image feature points, represented as... Geometric feature vectors are extracted based on the point cloud projection depth map, and are represented as follows: Subsequently, and Perform vector concatenation to form the front field of view feature vector. .
[0032] To verify the effectiveness of the ORB feature extraction algorithm, feature matching was performed between two consecutive image frames. Figure 3 The matching results are shown, verifying that the extracted feature points can stably establish a correspondence between consecutive frames.
[0033] After preprocessing such as denoising and ground segmentation, the point cloud is projected onto a horizontal plane and mapped onto an image grid at a fixed resolution to obtain a bird's-eye view. From the bird's-eye view, the normalized density features of the grid point cloud are constructed, represented as... The average height feature is calculated based on the height of the grid point cloud, and is expressed as follows: Based on the grid point cloud in x shaft and y The spatial distribution statistical characteristics are obtained by analyzing the distribution along the axis, and are represented as follows: Using a denoised point cloud, for each point, its k nearest neighbors are first retrieved, and principal component analysis is performed to estimate the normal vector. Then, the mean and standard deviation of the normal vectors of all points are concatenated to represent the result. Finally, , , , Vector concatenation is performed to form the feature vector of the bird's-eye view. .
[0034] To preserve negative features such as the depth of depressions below ground level, the following methods are used: The activation function retains negative value information, resulting in the final front field feature vector as shown in formulas (5) and (6). and bird's-eye view feature vector .
[0035] = (5) = (6) After feature extraction is completed, and The fusion process is as follows: First, [the process involves]... Features and Features are unified to the same dimension. When unifying dimensions, zero-padding is used to unify the feature dimensions. ,in Weights are assigned based on the L2 norm. As shown in Eq.(7) and Eq.(8), dynamic weight allocation between fields of view of different frames is realized.
[0036] = (7) = (8) Then, according to formula (9), the weighted features are scaled, processed by the LeakyReLU activation function, and clipped to the range [-1,1] using the clip function to obtain the fused features. .
[0037] = (9) Finally, channel attention is used to obtain the fused multi-view features. Global average pooling is performed on the fused features along the channel dimension to obtain channel-level statistics. This vector can be considered as a compressed channel description vector. Then, the weights of each channel are calculated using tanh activation and fully connected layers to highlight important channels and suppress noise, as shown in Equation (10).
[0038] (10) Step 3: Multi-strategy fusion 2D-3D feature matching.
[0039] Existing methods often employ fixed thresholds for 2D-3D feature matching, resulting in poor scene adaptability. Therefore, this embodiment constructs a dynamic matching threshold based on point cloud density and image quality, and adopts multi-scale progressive matching and geometric consistency constraints to ultimately output high-quality 2D-3D matching pairs, thereby improving calibration accuracy. The 2D-3D matching module is as follows: Figure 4 As shown.
[0040] 2D-3D feature matching consists of three stages. The first stage determines the matching threshold. This is first determined based on the point cloud density. Select the corresponding scene, and then base it on image clarity. Select a matching threshold. Divide scene complexity into three levels, corresponding to dense, medium-density, and sparse scenes. For each scene, set three levels of matching thresholds: coarse matching threshold, medium matching threshold, and fine matching threshold. Simultaneously, set a reprojection error threshold for geometric consistency constraints.
[0041] Based on experience, point cloud density, image sharpness, and matching thresholds at various levels were set for dense and sparse scenes, as shown in the first two rows of Table 1. For the matching thresholds at various levels of medium density, linear interpolation was used for calculation, as shown in formulas (11) and (12).
[0042] Table 1
[0043] = (11) (12) in, and These represent the point cloud density in sparse and dense scenes, respectively. k { c , m , f}
[0044] The second stage involves implementing a multi-scale progressive matching strategy, progressing from coarse to fine feature matching. In the previous stage, we based the feature matching on point cloud density... Determine the scene density and select the corresponding three-level matching threshold based on image sharpness. A matching threshold is selected. During matching, three scales are applied sequentially: coarse matching threshold, medium matching threshold, and fine matching threshold. If the number of 2D-3D matching pairs after matching at a certain scale is greater than 10, matching at the next scale is performed. 2D-3D feature matching ends when the number of matching pairs at a certain scale is less than 10 or when matching at all scales is completed. The set of matching pairs obtained using this multi-scale progressive matching strategy is represented as... .
[0045] The third stage is... Geometric consistency constraint checks are performed on the matching pairs to filter out pairs with excessively large reprojection errors, thereby further improving the quality of the matching pairs. (Point-to-point) Using the initial extrinsic parameters and formula (13), its coordinates in the point cloud are determined. =( , , Mapped to coordinates in the camera coordinate system =( , , ), among which, among which Let be the initial rotation matrix. This is the initial translation matrix.
[0046] (13) Then, through camera intrinsic parameters The pixel coordinates are obtained by projecting them onto the image plane. Calculated using formula Eq.(14) reprojection error ,in( , (This is a review) The actual coordinates in a 2D pixel coordinate system.
[0047] (14) for Points in the array are sorted by reprojection error, and those exceeding a specified error threshold are included. The matching pairs corresponding to the points are deleted. During the deletion process, if the number of matching pairs falls below 10, the deletion stops. Finally, a sufficient number of matching pairs that have passed geometric consistency filtering are obtained, denoted as . .
[0048] Figure 5 This displays the 2D-3D feature matching results for a single image frame. Circles represent 2D image feature points, and squares represent the corresponding 3D point cloud projection locations. Color coding indicates reprojection error (green < 0.5 pixels, yellow 0.5-1.0 pixels, red ≥ 1.0 pixels). Figure 6 It can be seen that the circles and squares are highly overlapping, which verifies that the matching quality and calibration accuracy are good.
[0049] Step 4: Solve for external parameters.
[0050] The external parameter solution model is shown in formula (14), where, The confidence weight can be obtained by normalizing the consistency score, see formula (16). This is the reprojection error.
[0051] (15) (16) The LM algorithm is used to optimize and solve formula (15) to obtain the final external parameters R and t for lidar-camera calibration. For specific calibration steps of FTDM-Calib, please refer to [link to FTDM-Calib calibration documentation]. Figure 6 Lines 1 to 5 are used for data preprocessing, performing point cloud denoising and image enhancement. Dual-field feature extraction and fusion are located in lines 6 to 11 of the algorithm. Lines 12 to 26 perform multi-scale progressive feature matching to extract high-quality 2D-3D matching pairs. Line 27 uses the LM algorithm to solve for the extrinsic parameters of the LiDAR and camera. Line 28 outputs the solved extrinsic parameters.
[0052] To verify the calibration performance of the method in this embodiment, the method in this embodiment was compared with calibration methods such as LCCNet, AdaptiveVoxelization, DEdgeNet, CalibFormer, OL-Reg, EdgeCalib, Full-Lifecycle, Adaptiveplug-and-play, DF-Calib, and Motion+Edge. The results are shown in Table 2.
[0053] As shown in Table 2, the proposed method outperforms the comparative methods in both rotation and translation errors. Compared with the average performance of all comparative methods, our method improves upon the rotation index by 89.2% and the translation index by 85.7%. In terms of rotation accuracy, our method improves upon the suboptimal value (LCCNet) by 11.8%, and in terms of translation accuracy, it improves upon the suboptimal value (DEdgeNet) by 2.4%. These data demonstrate that our method outperforms the comparative methods in all evaluation metrics, exhibiting excellent calibration performance and proving the effectiveness of the method.
[0054] Table 2
[0055] The KITTI dataset contains various scenarios. To verify the calibration accuracy and robustness of FTDM-Calib under different scenarios, four typical scenarios were selected for statistical analysis of calibration performance. The four scenarios mainly consist of traffic signs, vehicle motion, residential buildings, and urban scenes.
[0056] Table 3 shows the calibration results of FTDM-Calib in four scenarios. The data in the table shows that FTDM-Calib achieves excellent calibration results in all scenarios. The maximum difference between the rotation error and the average rotation error of the entire KITTI dataset in the four scenarios is 0.0022°, and the maximum difference in translation error is 0.032cm. The largest difference in rotation error occurs in CityScenario, fluctuating by approximately 14.6% compared to the average rotation error of the KITTI data. The largest difference in translation error occurs in RoadScenario, fluctuating by approximately 13.3% compared to the average rotation error of the KITTI data. In most other scenarios, the error fluctuation is less than 10%, indicating that FTDM-Calib exhibits excellent calibration accuracy and stable calibration performance in different scenarios. Therefore, FTDM-Calib has a strong generalization ability for calibration scenarios.
[0057] Table 3
[0058] Figure 7 The comparison between the original image, the real calibration results, and the calibration results of the proposed method is presented through visualization. The figures clearly show that the point cloud projections calibrated by the proposed method in various scenarios are highly consistent with the real calibration results, verifying the effectiveness and strong generalization ability of the method.
[0059] Table 4 presents the calibration errors under different sensor configurations, covering the mean error and fluctuation range of two typical hardware schemes, which intuitively reflects the method's adaptability to heterogeneous sensors.
[0060] Table 4
[0061] As can be seen from Table 4, in high-beam radar (KITTI) and low-beam radar (nuScenes)
[25] In the dataset, the error did not fluctuate significantly. The rotation errors of KITTI and nuScenes were 0.015° and 0.0173°, respectively, and the translation errors were 0.241cm and 0.2007cm, respectively. This indicates that even with significant changes in the sensor hardware, the calibration error of the method in this embodiment did not fluctuate significantly and remained in an extremely low range.
[0062] The FTDM-Calib method in this embodiment comprises three core modules: Point Cloud Fault-Tolerant Denoising (FTD), Dual-View Feature Extraction (DVFE), and Multi-Scale Progressive 2D-3D Matching (MPM). This embodiment constructs the following FTDM-Calib variant models to verify the function of each module: FTDM-Calibw / oFTD (FTD module removed), FTDM-Calibw / oDVFE (DVFE module removed), and FTDM-Calibw / oMPM (MPM module removed). Table 5 shows the results of the ablation experiments. The data in the table shows that, compared to FTDM-Calib (rotation error 0.015°, translation error 0.241cm), after removing the FTD module, the rotation error of FTDM-Calib increased from 0.015° to 0.075°, and the translation error increased from 0.241cm to 0.625cm, indicating a significant decrease in calibration accuracy. After removing the DVFE module, the rotational error increased to 0.08°, and the translational error rose to 1.21cm, resulting in a larger error in calibration performance. Removing the MPM module further reduced the rotational and translational accuracy of FTDM-Calib by 0.011° and 0.602cm, respectively. These results demonstrate that removing any one of the three modules significantly decreases the rotational and translational accuracy of FTDM-Calib. This indicates that all three modules play a crucial role in the calibration process of FTDM-Calib, and all three modules are essential for its operation.
[0063] Table 5
[0064] This embodiment proposes a calibration method that integrates fault-tolerant denoising, dual-view feature fusion, and multi-scale progressive matching. This method employs a collaborative mechanism of normal vector denoising and distance-driven inlier back-judgment to suppress outlier interference while preserving edge features. Feature fusion of the forward field of view and bird's-eye view overcomes the dependence of traditional single-view calibration on scene structure. Furthermore, an adaptive threshold adjustment based on point cloud density and image sharpness constructs a multi-scale progressive matching strategy from coarse to fine.
[0065] Experimental results show that this proposed method significantly reduces both rotation and translation errors, and its calibration accuracy surpasses that of current mainstream LiDAR-camera calibration methods. More notably, this method demonstrates excellent cross-scene and hardware device generalization capabilities, possessing high engineering practical value. Future work will further improve the feature extraction and fusion quality of dual-field-of-view systems and focus on the joint calibration of multiple cameras and LiDAR systems.
[0066] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.
Claims
1. A lidar-camera calibration method based on dual-view feature and multi-scale feature matching, characterized in that, Includes the following steps: Step 1: Acquire camera and LiDAR data. Using RGB images and LiDAR point clouds as input, filter out noise interference and extract effective input features through two-stage point cloud denoising and image preprocessing. Step 2: The dual-view feature extraction module acquires the semantic-geometric features of the front field of view and the spatial-structural features of the bird's-eye view. Through selective feature fusion, it outputs high-quality point cloud features. Step 3: In the multi-scale progressive feature matching module, a multi-scale feature matching method oriented towards density and image quality perception is constructed to obtain high-quality 2D-3D matching point pairs, providing reliable constraints for extrinsic parameter optimization. Step 4: The extrinsic parameter optimization module uses the LM algorithm to solve and optimize the extrinsic parameters, and outputs high-precision extrinsic parameters. The two-stage point cloud denoising in step 1 specifically consists of: normal vector angle denoising and distance optimization recovery.
2. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 1, characterized in that, The normal vector angle denoising in step 1 specifically involves: denoising the points in the original point cloud... Take it - Nearest neighbors: Use PCA to calculate the covariance matrix; the eigenvector corresponding to the smallest eigenvalue is the normal vector of that point. Then fit the main axis of the scene. Through calculation and The included angle ,Will Greater than the threshold The points are marked as noise points: 。 3. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 2, characterized in that, The distance optimization and recovery in step 1 specifically involves: for point... The KDTree method is used to generate a product with... The set of nearest neighbors ; calculate and Average distance of middle nodes : = in, for interior point in, for and Euclidean distance; if < Points identified as interior points are restored; multiple point clouds are transformed to the world coordinate system, and point cloud registration and fusion are performed to filter out duplicate points and retain high-quality point clouds.
4. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 3, characterized in that, The image preprocessing in step 1 specifically involves: projecting the point cloud onto the image plane, calculating the proportion of projected points falling within the semantic segmentation region, and using this proportion as the consistency score of the frame data. This consistency score reflects the degree of consistency between the point cloud and the semantic information of the image, and is used for subsequent optimization of the confidence weight of the objective function; converting the static region image into a grayscale image, calculating its Sobel gradient in the x-axis and y-axis directions, and accumulating the gradient magnitudes to obtain the sharpness G. The normalized image sharpness G is used for threshold selection in the subsequent 2D-3D matching stage. The formula for calculating the consistency score is as follows: in, This represents the number of projection points within the semantic segmentation region. This represents the total number of valid projected points. The formula for calculating image sharpness G is: = in, W , H These are the image width and height, respectively. and The images are respectively in Sobel gradient responses in the x and y directions at a pixel.
5. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Under the forward field of view, extract image feature points using the ORB algorithm, represented as... Geometric feature vectors are extracted based on the point cloud projection depth map, and are represented as follows: ,Will and Perform vector concatenation to form the front field of view feature vector. ; Step 2.2: After preprocessing such as denoising and ground segmentation, the point cloud is projected onto a horizontal plane and mapped onto an image grid at a fixed resolution to obtain a bird's-eye view. From this bird's-eye view, the normalized density features of the grid point cloud are constructed, represented as... The average height feature is calculated based on the height of the grid point cloud, and is expressed as follows: According to the grid point cloud x shaft and y The spatial distribution statistical characteristics are obtained by analyzing the distribution along the axis, and are represented as follows: Using a denoised point cloud, for each point, its k nearest neighbors are first retrieved and principal component analysis is performed to estimate the normal vector. The mean and standard deviation of the normal vectors of all points are then concatenated to represent the result. ,Will , , , Vector concatenation is performed to form the feature vector of the bird's-eye view. ; Step 2.3, using The activation function retains negative values, resulting in the final front field feature vector. and bird's-eye view feature vector : = ； = ； Step 2.4, and Fusion, resulting in fusion characteristics Multi-view features are obtained through channel attention. : ； = ； in, Indicates the feature weights of the front field of view. Indicates the weight of the bird's-eye view features. express Features and A unified dimension of features.
6. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 4, characterized in that, Step 3 specifically involves: Step 3.1, determine the matching threshold: select the corresponding scene according to the point cloud density ρ, select the matching degree threshold according to the image sharpness G, divide the scene complexity into three levels, corresponding to dense scene, medium density scene and sparse scene respectively, and set three levels of matching threshold for each scene, namely coarse matching threshold, medium matching threshold and fine matching threshold, and set the reprojection error threshold of geometric consistency constraint at the same time. Step 3.2: Implement coarse-to-fine feature matching using a multi-scale progressive matching strategy: If the number of 2D-3D matching pairs after matching at a certain scale level is more than 10, then proceed to the next scale level for matching. When the number of matching pairs at the end of matching at a certain scale level is less than 10 or when matching at all scale levels is completed, the 2D-3D feature matching ends. The set of matching pairs obtained by this multi-scale progressive matching strategy is denoted as M. Step 3.3, perform geometric consistency constraint checks on the matching pairs in M: for points Using initial extrinsic parameters and Set its coordinates in the point cloud =( , , Mapped to coordinates in the camera coordinate system =( , , ), among which, among which Let be the initial rotation matrix. As the initial translation matrix, the camera intrinsic parameters are used to... The pixel coordinates are obtained by projecting them onto the image plane. ,calculate reprojection error , ,in( , (This is a review) The true coordinates in a 2D pixel coordinate system; for Points in the array are sorted by reprojection error, and those exceeding a specified error threshold are included. The matching pairs corresponding to the points are deleted. During the deletion process, if the number of matching pairs is less than 10, the deletion stops. Finally, a sufficient number of matching pairs that have passed geometric consistency filtering are obtained, denoted as . .
7. The lidar-camera calibration method based on dual-view feature and multi-scale feature matching according to claim 6, characterized in that, The specific steps for solving the extrinsic parameter model in step 4 are as follows: ； ； in, As the confidence level weight, This is the reprojection error.