A method for correcting laser radar and camera external parameter drift

Abstract

A laser radar and camera external parameter drift correction method, comprising the following steps: step (1): the three-dimensional point cloud of the laser radar is preprocessed by the pinhole camera model, and a depth map is obtained; step (2): a full-resolution feature fusion module based on FPN; all hierarchical features including the highest resolution layer are extracted and fused respectively, and paired features with consistent semantics and retaining fine-grained geometric details are output; step (3): the rotation and translation components are estimated; step (4): the regression loss is used to constrain the drift prediction value in the parameter space, and the mixed loss is used to weight the re-projection error pixel by pixel according to the distance from the optical center and the point cloud depth in the imaging geometry space; step (5): a once wide-range correction is decomposed into three times of fine correction by a coarse-to-fine multi-range iteration, and the single prediction ability is converted into the final high-precision calibration result. The present application has high precision, strong robustness and good cross-dataset generalization in the estimation of translation and rotation drift.

Description

Technical Field

[0001] This invention relates to the field of lidar-camera extrinsic parameter calibration and drift correction technology, specifically to a lidar-camera extrinsic parameter drift correction method. Background Technology

[0002] Multi-sensor fusion of LiDAR and cameras plays a crucial role in fields such as autonomous driving, robotic environmental perception, and 3D reconstruction. LiDAR can acquire high-precision 3D geometry but lacks texture, while cameras provide fine texture and semantics but have limited depth perception and are sensitive to lighting conditions. The two can be complementary and fused. The foundation for effective fusion is obtaining accurate external parameter calibration between the two sensors (rotation and translation of the LiDAR and camera, referred to as extrinsic parameters). In actual operation, factors such as temperature changes and mechanical vibrations can cause extrinsic parameters to drift over time, resulting in intermodal misalignment and decreased sensing performance. Therefore, precise online extrinsic parameter drift correction methods are needed.

[0003] Existing technologies for lidar-camera extrinsic parameter calibration and drift correction mainly fall into three categories.

[0004] (1) Calibration-based method: Calibration is completed by establishing a 2D-3D point correspondence relationship with special calibration objects such as checkerboard, calibration plate with round hole, ArUco code plate, and acrylic plate. It has high accuracy but depends on specific calibration objects and scene layout, making it difficult to seamlessly integrate into fully autonomous driving system and unsuitable for online scenarios.

[0005] (2) Targetless method: Extract features such as points, lines and surfaces from lidar and camera data, estimate extrinsic parameters by matching algorithms such as Iterative Closest Point (ICP) and Efficient Perspective n-Point (EPnP), or perform calibration by using motion trajectory, intensity consistency and cross-modal structure consistency.

[0006] (3) Learning-based methods: Convolutional neural networks (CNNs) are used to extract, match and fuse features from the depth map obtained by projecting RGB images and point clouds to regress extrinsic parameters. Representative works include RegNet, CalibNet, LCCNet, MRCNet, CalibFormer, RobustCalib, etc.

[0007] Compare this to the paper X. Lv, B. Wang, Z. Dou, D. Ye, S. Wang. LCCNet: LiDAR and Camera Self-Calibration Using Cost Volume Network. IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021: 2888–2895. This paper uses two ResNet branches to extract features from RGB images and depth maps respectively. It calculates the spatial correlation between the two modal features in a predefined local neighborhood and constructs a cost volume to establish pixel correspondence. Then, the extrinsic parameter bias is output by a fully connected regression head. The technical defects are as follows: (a) When fusing multi-scale features, only the downsampled medium and low resolution feature layers are used, and the highest resolution feature layer is discarded. This causes the fine-grained geometric details at the edge of the point cloud projection to be lost during the fusion process, limiting the spatial localization capability of the fused features and resulting in poor spatial consistency of translation component estimation (manifested as a large standard deviation of translation error); (b) The cost volume only establishes correspondences within a fixed-size local neighborhood. Cross-modal interactions are restricted to fixed local modes, making it impossible to model long-range geometric relationships within the image range. When the drift is large, it is difficult to cover the true corresponding position, resulting in insufficient accuracy of rotation component estimation; (c) The supervision signal is only the regression loss of translation and rotation, without considering the influence of the optical characteristics of camera imaging and the depth of the point cloud on the projection error. Long-distance low signal-to-noise ratio points and short-distance high confidence points are treated equally, limiting the training robustness and final accuracy.

[0008] In summary, the technical defects of the existing technology are: (1) Insufficient feature extraction and fusion - the highest resolution feature layer is discarded during multi-scale fusion, fine-grained geometric details are lost, the quality of fused feature representation is limited, and the spatial consistency of translation estimation is affected; (2) Limited cross-modal matching ability - matching is performed in a spatially correlated manner within a predefined neighborhood, the interaction is limited to a fixed local mode, it is difficult to model long-range geometric relationships, and the accuracy of rotation drift estimation is insufficient; (3) Lack of geometric consistency in supervision - the loss is only regressed in the parameter space, ignoring the influence of camera optical characteristics (optical center distance) and point cloud depth on projection error, and low signal-to-noise ratio points at a distance participate in supervision without distinction, amplifying calibration error, and limiting the robustness and accuracy of optimization. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention provides a method for correcting extrinsic drift in lidar and cameras. This method is based on a hybrid loss mechanism that combines full-resolution feature fusion, cross-modal attention, and geometric consistency. It requires no calibration and can run online. In the feature fusion stage, all feature layers, including the highest resolution layer, are preserved without losing fine-grained geometric information. In the feature matching stage, cross-modal attention is used to establish globally adaptive pixel correspondences across the entire feature map. In the training stage, the hybrid loss is used to weight the reprojection error pixel by pixel based on the optical center distance and point cloud depth. In the inference stage, multi-range iterations are used to refine the data step by step from coarse to fine, thereby achieving high accuracy, strong robustness, and good cross-dataset generalization in both translation and rotation drift estimation.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for correcting extrinsic parameter drift in lidar and camera systems includes the following steps: Step (1): Preprocess the 3D point cloud of the LiDAR using the pinhole camera model, and project the 3D point cloud of the LiDAR into a depth map of the same size as the RGB image; the purpose is to transform the 3D-2D calibration problem into a pixel alignment problem between two images, and provide a unified 2D input representation for the remaining steps. Step (2): Full-resolution feature fusion module based on FPN; Taking the RGB image and depth map obtained in step (1) as input, extract and fuse all layer features including the highest resolution layer respectively, with the aim of providing semantically consistent and fine-grained geometric details as paired features for step (3); Step (3): Establish a globally adaptive pixel correspondence on the paired features output in step (2) using cross-modal attention, and regress the extrinsic parameter drift by the global regression layer. The purpose is to break through the limitation of fixed local neighborhood and accurately estimate the rotation and translation components, which is the core link in generating calibration results. Step (4): Without changing the forward inference structure, supervision is provided only for steps (2) and (3) during the training phase. The regression loss constrains the drift prediction value in the parameter space, and the hybrid loss is weighted pixel by pixel in the imaging geometry space according to the optical center distance and the point cloud depth reprojection error. The purpose is to make the features and correspondence learned in the first two steps self-consistent in the imaging geometry and suppress the interference of distant noise points. Step (5): Extrinsic parameter drift correction inference (coarse-to-fine multi-range iteration); in the inference stage, the structure of steps (1) to (3) is reused to decompose a large-range correction into three progressively fine corrections through coarse-to-fine multi-range iteration. The purpose is to transform the single prediction capability of the first four steps into the final high-precision calibration result.

[0011] The specific steps (1) are as follows: Known data includes camera intrinsic parameter matrix Initial extrinsic parameter matrix of lidar to camera and lidar point clouds Based on the pinhole camera model, the point cloud is first transformed from the lidar coordinate system to the camera coordinate system, and then projected onto the image plane: (1) (2) (3) in, The camera intrinsic parameter matrix is ​​3×3. The initial (miscalibrated) extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system is a 4×4 homogeneous transformation matrix; for The 3×3 rotation matrix in for The 3×1 translation vector in the text; For the first The three-dimensional coordinates of a lidar point in the lidar coordinate system These are the 3D coordinates of the point after transformation to the camera coordinate system. This is the projection of the point onto the image plane; variables marked with "^" ( , , ) represents the homogeneous form of the corresponding coordinates, that is, the vector formed by adding 1 to the end of the original coordinates; Represents the three-dimensional real number space; The index of a point in the point cloud. The number of points involved in the calculation; The coordinates of each point along the optical axis in the camera coordinate system are obtained from equations (1) and (2). The distance from the point to the xoy plane of the camera coordinate system is the depth of the point (in meters); the pixel coordinates of the point on the image plane are obtained from equation (3). , , These are the horizontal and vertical coordinates of the pixel (unit: pixels); defining the value of the projected pixel in the depth map. The reciprocal of depth: (4) Repeat the above calculation for all projection points in the point cloud to obtain a depth map of the same size as the RGB image, which is used as the input for the depth map branch in step (2).

[0012] The full-resolution feature fusion module based on FPN in step (2) includes two parallel feature extraction branches: a depth map branch and an RGB image branch. The RGB image branch takes the RGB image captured by the camera as input, and the depth map branch takes the depth map obtained by projecting the point cloud of the lidar onto the image plane as input. The two branches have identical network structures, both including bottom-up and top-down paths, and both built on the ResNet-18 backbone and FPN structure. The only difference is the modality of the input data and the number of input channels in the first convolutional layer. The network parameters of the two branches are independent and not shared. Since the two branches have the same structure, the following explanation uses one branch as an example to illustrate the specific processing of its bottom-up and top-down paths. The processing of the other branch is exactly the same, the only difference being the input image: Bottom-up path: An RGB image of size 3×256×512 is first convolved to obtain a feature map of size 64×128×256. ; After passing through the first to fourth residual layers, 64×128×256 were obtained respectively. 128×64×128 256×32×64 With 512×16×32 The spatial resolution is halved at each level, while the number of channels is doubled at each level. This process can be represented as: (5) (6) in, The input image for this branch is a 3-channel RGB image for the RGB branch and a depth map branch is a depth map obtained by projecting the point cloud. This represents the first convolutional layer of ResNet-18; Indicates the first... One residual layer; For the first The main characteristics of the hierarchy The larger the value, the lower the spatial resolution, the larger the receptive field, the stronger the semantic information, but the more blurred the geometric localization. Top-down path: starting with the coarsest level Starting from this point, the core features of each level are first uniformly converted to the same number of channels through 1×1 horizontal convolutions. Then, from top to bottom, a "sampling-addition-smoothing" process is performed. That is, the fused features of the previous level are upsampled to the spatial resolution of the next finer level through bilinear interpolation, and then added element-wise with the features of that level after horizontal convolution. The result is then smoothed through 3×3 convolutions, resulting in fused features with spatial resolutions of 32×64, 64×128, and 128×256, respectively. , and The process is represented as: (7) (8) (9) in, The function of the 1×1 horizontal convolution applied to each level is to unify the backbone features of different levels with different channel numbers to the same number of channels, so that the features of different levels can be added element by element. "+" indicates bilinear interpolation upsampling, which magnifies the feature map to the next finer spatial resolution; "+" indicates element-wise addition, used to inject deep semantic information into shallow layers. The function of this 3×3 smooth convolution is to suppress the aliasing introduced by upsampling and avoid generating false edges that interfere with subsequent matching. For the first The intermediate fusion features after adding the layers For the first The final fusion features of the hierarchical output; As can be seen from equations (7)-(9), the top-down path extends all the way to... Unlike the baseline, the highest resolution layer (spatial resolution 128×256) is used, so the fine-grained geometric details at the edge of the point cloud projection are fully preserved in the feature matching stage. The fused features have stronger spatial localization capabilities while maintaining multi-scale semantic consistency. This branch ultimately outputs the fused features of each level. , , The number of layers in the depth map branch output is the same as that of the coarsest layer; the outputs of the same layer of the two branches are fed into the cross-modal attention feature matching layer in step (3).

[0013] Step (3) specifically involves: The feature matching layer employs a cross-modal attention mechanism. , , The three matrices are generated by convolution: Obtained by convolution of RGB branch features. , Obtained by convolution of branch features from the depth map; The baseline method calculates the spatial correlation between RGB pixels and depth map points within a predefined neighborhood for matching, restricting the interaction to a fixed local pattern; this reduces computational cost while preserving the global context. , , All are generated by 1×1 convolutions with a stride of 2 (while preserving the number of channels). (While halving the spatial resolution), the relevant formula is as follows: (10) (11) in, A feature map containing RGB image information. A feature map containing depth map point information. To and Homologous value feature maps Indicates matrix transpose. For the number of channels, This indicates the normalized exponent operation performed row by row. It represents the correlation between RGB pixels and depth map pixels (containing rotation and translation information). This is the output of this layer (including calibration information); the global regression layer takes globally relevant features as input and is divided into two parallel branches: translation and rotation. Each branch consists of three fully connected layers, outputting a 3D translation vector. With 4-dimensional rotation quaternions .

[0014] Step (4) specifically involves: GRNet's loss function consists of two parts: regression loss and mixed loss. Regression loss includes translation vector loss and rotation quaternion loss. Translation vector error is measured using SmoothL1 loss, and rotation quaternion error is measured using angular distance. (12) (13) (14) in, For translation vector loss, This represents the smoothing L1 loss function, which takes a quadratic term when the absolute value of the error is less than a threshold, and a linear term otherwise. For rotation quaternion loss, Represents the angular distance (in degrees) between two unit quaternions; This is the true value of the translation vector (unit: m). The truth value of the rotation quaternion; , These are the translation vector and rotation quaternion predicted by the network, respectively; For regression loss; , These are the weights of the translation loss term and the rotation loss term, respectively, and are hyperparameters that are not less than zero. In this embodiment, we take... ; Hybrid Loss: When mapping point clouds to depth maps, the image plane coordinates and the two class weights are calculated as follows: (15) (16) (17) (18) (19) (20) in, , These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image (unit: pixels). , These are the x and y coordinates (in pixels) of the camera's optical center (principal point) on the image plane, all taken from the camera intrinsic parameter matrix. ; , , For the first The three-dimensional coordinates of a point in the camera coordinate system, where That is, the depth of that point; , These are the x and y coordinates of the point on the image plane; Describes the 2-norm of a vector. Indicates all Summing the projection points, This represents the total number of projection points involved in the calculation for this frame. The weights are point-by-point based on the distance from the optical center, which increase as the distance from the image point to the optical center increases, and are summed to 1 over all points; It is a very small positive number, used to prevent the denominator from being zero; It is a depth-based point-by-point weight, which decreases as the depth of the point increases; The point-by-point weights are obtained by combining the two equally weighted combinations. For the first Homogeneous coordinates of the lidar points The truth matrix of the extrinsic parameters. The extrinsic parameter drift matrix predicted by the network. and These are the coordinates obtained by projecting the point onto the image plane using the true extrinsic parameters and the predicted extrinsic parameters; The projection error is the weighted average of all projection points in the frame, i.e., the mixing loss. Both the optical center distance and depth terms originate from the same physical quantity, "distance from a point to the sensor," and are normalized to the same scale. Therefore, they are combined with equal weights (half each). This equal weighting configuration was verified using a homoscedastic uncertainty framework. Upon convergence, the optical center-depth normalized weights stabilized at 0.50:0.50, and the total loss was: (twenty one) in, The mixed loss is determined by the aforementioned equations (15)-(20); The regression loss is determined by the aforementioned equation (14); , , respectively, are the weights of the mixed loss and the regression loss, and are hyperparameters that are not less than zero; This represents the total loss for a single frame sample, calculated during training for all frames within a batch. After averaging, backpropagation is performed, and the Adam optimizer is used to update the network parameters.

[0015] Step (5) specifically involves: GRNet outputs the extrinsic parameter drift between the LiDAR and the camera, and the input during training is... Represents the initial, misaligned extrinsic parameters, and the network prediction correction matrix. Final calibration results Calculate using the following formula: (twenty two) During reasoning, For sensor extrinsic parameters that may drift in real-world scenarios, GRNet predicts the corresponding drift. The above formula is used to obtain the extrinsic parameters of the current alignment, thereby realizing online misalignment correction based on predicted drift; GRNet employs an iterative optimization strategy involving multiple networks and multiple ranges: three networks with identical structures but trained for different mislabeling ranges, with translation and rotation ranges respectively selected. ( ={0.5, 0.2, 0.1} m) and ( ={5°, 2°, 1°}); In the first iteration, the RGB image and projected depth map at the maximum range (0.5 m, 5°) are input into the network for initial prediction and correction. Then, the network is used... The point cloud is reprojected to generate an updated depth map, which is then input into a second smaller range (0.2 m, 2°) network for refinement along with the original RGB image. This process is repeated for a third smallest range (0.1 m, 1°) network to obtain the final extrinsic calibration parameters. This multi-range iterative strategy allows each network to focus on progressively finer corrections, thereby improving the final calibration accuracy in a structured manner.

[0016] The beneficial effects of this invention are: (1) The input preprocessing in step (1) projects the point cloud into a depth map according to the pinhole camera model, and uses... As pixel values, the 3D data of the LiDAR and the 2D data of the camera are uniformly expressed on the same image plane, transforming the 3D-2D calibration problem into a pixel alignment problem between two images, thus enabling online operation without any calibration objects or downtime.

[0017] (2) The full-resolution feature fusion module in step (2) upsamples from top to bottom, adds it element-wise with the lateral features, and smooths it with a 3×3 convolution, while retaining and fusing all feature levels, including the highest resolution layer. Since the highest resolution layer retains the fine-grained geometric details of the point cloud projection edges, the fused features have stronger spatial localization capabilities while maintaining multi-scale semantic consistency, directly improving the consistency of translation component estimation: under the condition that other parts remain unchanged, after the module is returned to the baseline structure, the translation error The standard deviation increased from 0.499 cm to 1.331 cm, and the mean increased from 0.876 cm to 0.993 cm.

[0018] (3) The cross-modal attention in step (3) enables the query (From RGB features) can match keys across the entire feature map. ,value Weighted aggregation (from depth map features) overcomes the limitation of baselines in calculating spatial correlation within a fixed neighborhood, enabling the modeling of long-range geometric relationships within the image range. Rotation drift, however, relies precisely on these long-range geometric relationships for accurate estimation. Removing this mechanism results in rotation error... The mean increased from 0.126° to 0.145°, and the standard deviation increased from 0.115° to 0.613°. Meanwhile... , , All are generated by 1×1 convolutions with a stride of 2, and attention is only applied to ( )×( The calculation is performed at 10 locations, and the number of parameters of this module is controlled to about 0.84 M while preserving the global context (less than 1% of the total network size of 116.59 M).

[0019] (4) The mixing loss in step (4) is weighted according to the optical center distance. With depth weight (Each accounts for ½) Pixel-by-pixel weighted sum of projection errors from point cloud to image: This automatically reduces the error contribution from distant, low signal-to-noise ratio points and suppresses the interference of ranging noise on optimization. Points whose projection positions are more sensitive to changes in external parameters are given greater weight, and supervision is concentrated on pixels with higher information content. Both provide geometrically consistent supervision for depth projection: after removing the depth-weighted term, the mean rotation error increases from 0.126° to 0.149°, and the median increases from 0.090° to 0.152°; after removing the optical center distance term, the translation error... The mean value increased from 0.876 cm to 0.914 cm; removing both values ​​resulted in a degradation to 1.027 cm and 0.151°, indicating that the two values ​​are complementary and indispensable.

[0020] (5) Step (5) involves coarse-to-fine multi-range iterative reasoning according to... By progressively updating extrinsic parameters and reprojecting them to generate new depth maps, each network level learns finer corrections within a smaller range, thus compressing the error step by step: the translation error is shifted after passing through the first-level network (0.5 m, 5°). The mean value is 2.2433 cm, and the rotational error is... The mean was 0.2161°, decreasing to 1.2718 cm and 0.1657° after the second level (0.2 m, 2°), and further decreasing to 0.8760 cm and 0.1269° after the third level (0.1 m, 1°), with the error distribution becoming more concentrated. The standard deviation decreased from 1.0767 cm to 0.4997 cm.

[0021] (6) The five steps mentioned above work together to enable the present invention to achieve a mean absolute error of 0.3487 cm for translation and 0.0382° for rotation on the KITTI odometry dataset (sequence 01–20 for training and validation, sequence 00 for testing). Compared with the baseline MRCNet, the translation drift estimation is improved by about 2.1% and the rotation drift estimation is improved by about 15.1%, which is better than RobustCalib (translation 0.806 cm, rotation 0.046°) under the same range (±0.1 m, ±1°). In the nuScenes (v1.0-mini) cross-dataset experiment, under the same lightweight fine-tuning protocol, the mean rotation error decreased from 6.694° of the baseline to 2.903° (a relative reduction of about 56.6%), and the standard deviations of translation and rotation errors decreased from 13.248 cm and 7.894° to 9.322 cm and 1.592°, respectively, with stronger generalization and prediction stability.

[0022] Note: The above The Euclidean norm of the translation error vector, Let be the 2-norm of the angle error vector; 0.3487 cm and 0.0382° are respectively The average of the absolute errors of the three directions and the three components of roll / pitch / yaw. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall technical solution of the GRNet of the present invention.

[0024] Figure 2 This is a detailed network framework diagram of the GRNet of this invention.

[0025] Figure 3 This is a schematic diagram of the feature extraction and full-resolution feature fusion (FPN) process of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of the global regression layer (regression head) of this invention.

[0027] Figure 5 This is a comparison chart of the visual results of correcting miscalibrated images according to the present invention (each row shows the miscalibrated image, the true value, and the correction result in sequence).

[0028] Figure 6 This is a distribution diagram of the absolute translation error of the present invention on the test set.

[0029] Figure 7 This is a distribution diagram of the absolute rotation error of the present invention on the test set. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings.

[0031] like Figure 1 , Figure 2 The diagram shows the overall technical solution and detailed network framework of the GRNet proposed in this invention. This invention is an end-to-end network, utilizing RGB images, a (miscalibrated) depth map obtained by projecting a LiDAR point cloud, and initial miscalibrated extrinsic parameters. As input, output the predicted translation vector. With rotation quaternions .

[0032] Its core comprises three key components: (a) Full-resolution feature fusion module based on Feature Pyramid Network (FPN); (b) Cross-modal attention feature matching mechanism; (c) Geometrically consistent mixed loss; coupled with a coarse-to-fine multi-range iterative inference strategy.

[0033] The three key components are sequentially connected in the data flow and complementary in function. Component (a) is located at the front end of the network. Its function is to extract and fuse features from the RGB image and depth map at all resolution levels, and output paired multi-scale fused features. This provides fine-grained geometric information and global semantic representation for subsequent matching, mainly affecting the spatial consistency of the translation component. Component (b) takes the paired features output by component (a) as input. Its function is to establish a globally adaptive cross-modal correspondence on the entire feature map and output relevant features containing calibration information. Then, the global regression layer regresses the extrinsic parameter drift, mainly affecting the accuracy of the rotation component. Component (c) does not change the forward inference structure and only acts on the training stage. Its function is to weight the reprojection error pixel by pixel according to the optical center distance and point cloud depth, providing geometrically consistent and noise-resistant supervision for the learning of components (a) and (b), so that the features and correspondences learned by the first two components are self-consistent in imaging geometry. The coarse-to-fine multi-range iterative inference strategy reuses the above structure in the inference stage, decomposing a large-scale correction into three progressively fine corrections. All three are indispensable: any component returning to the baseline design during the ablation experiment will lead to a decrease in accuracy (see the technical effects section below for details).

[0034] Figure 1 In this process, the RGB image and the miscalibrated depth map obtained by projecting the LiDAR point cloud are used as inputs, and are sequentially passed through a full-resolution feature fusion module, a feature matching layer, and a global regression layer to output the predicted translation vector. With rotation quaternions The coordinate mapping constraint module maps the point cloud to the image plane according to the predicted extrinsic parameters and the true extrinsic parameters respectively, and the training is supervised by the hybrid loss and the regression loss. The corrected extrinsic parameters are used to generate a fused image aligned with the camera and the LiDAR. Figure 2 Give and Figure 1 The corresponding detailed network framework: the RGB branch and the depth map branch are respectively processed by the full-resolution feature fusion module to obtain RGB features and LiDAR features at different scales. The features at the same level of the two branches are fed into the attention-based feature matching layer in pairs. Derived from RGB features, , (Features from point cloud depth maps) are used to establish cross-modal correspondences. The outputs are concatenated and rearranged before being regressed by fully connected layers of the global regression layer. and .

[0035] The specific implementation steps are as follows: (1) Input preprocessing.

[0036] Known data includes camera intrinsic parameter matrix Initial extrinsic parameter matrix of lidar to camera and lidar point clouds Based on the pinhole camera model, the point cloud is first transformed from the lidar coordinate system to the camera coordinate system, and then projected onto the image plane: (1) (2) (3) in, The camera intrinsic parameter matrix is ​​3×3. The initial (miscalibrated) extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system is a 4×4 homogeneous transformation matrix; for The 3×3 rotation matrix in for The 3×1 translation vector in the text; For the first The three-dimensional coordinates of a lidar point in the lidar coordinate system These are the 3D coordinates of the point after transformation to the camera coordinate system. This is the projection of the point onto the image plane; variables marked with "^" ( , , ) represents the homogeneous form of the corresponding coordinates, that is, the vector formed by adding 1 to the end of the original coordinates; Represents the three-dimensional real number space; The index of a point in the point cloud. The number of points involved in the calculation.

[0037] From equations (1) and (2), we can obtain the coordinate values ​​of each point along the optical axis in the camera coordinate system. The distance from the point to the xoy plane of the camera coordinate system is the depth of the point (in meters). The pixel coordinates of the point on the image plane can be obtained from equation (3). , , These are the horizontal and vertical coordinates of a pixel (unit: pixels). Defines the value of this projected pixel in the depth map. The reciprocal of depth: (4) Repeat the above calculation for all projection points in the point cloud to obtain a depth map of the same size as the RGB image, which is used as the input for the depth map branch in step (2).

[0038] (2) Full-resolution feature fusion module based on FPN (structure as follows) Figure 3 (As shown).

[0039] Figure 3The structure of this module is given by taking the RGB branch as an example. The structure of the depth map branch is exactly the same, the only difference being the number of input channels of the convolutional layer (3 for the RGB branch and 64 for the depth map branch). Figure 3 In the diagram, red arrows indicate the input and output of this module; the arrow labeled "Conv1" represents the first convolutional layer of ResNet-18 (7×7 kernel, 3 padding, 2 stride); the arrows labeled "Layer1" to "Layer4" represent the first to fourth residual layers of ResNet-18, respectively; "Upsampling" indicates bilinear interpolation upsampling; "Fusion" indicates element-wise addition followed by 3×3 convolution for smoothing; each cube represents a feature map, with the backbone feature size corresponding to the layer to which the cube is located marked below it, in the format of "number of channels × height × width".

[0040] Bottom-up path ( Figure 3 (Upper part): An RGB image of size 3×256×512 is first convolved to obtain a feature map of size 64×128×256. ; After passing through the first to fourth residual layers, 64×128×256 were obtained respectively. 128×64×128 256×32×64 With 512×16×32 The spatial resolution is halved with each successive step, while the number of channels is doubled with each successive step. This process can be represented as: (5) (6) in, The input image for this branch is a 3-channel RGB image for the RGB branch and a depth map branch is a depth map obtained by projecting the point cloud. This represents the first convolutional layer of ResNet-18; Indicates the first... One residual layer; For the first The main characteristics of the hierarchy The larger the receptive field, the lower the spatial resolution, the larger the receptive field, the stronger the semantic information, but the more blurred the geometric localization.

[0041] Top-down path ( Figure 3 (Lower half): at the coarsest level Starting from this point, the core features of each level are first uniformly converted to the same number of channels through 1×1 horizontal convolutions. Then, from top to bottom, a "sampling-addition-smoothing" process is performed. That is, the fused features of the previous level are upsampled to the spatial resolution of the next finer level through bilinear interpolation, and then added element-wise with the features of that level after horizontal convolution. The result is then smoothed through 3×3 convolutions, resulting in fused features with spatial resolutions of 32×64, 64×128, and 128×256, respectively. , and This process can be represented as: (7) (8) (9) in, The function of the 1×1 horizontal convolution applied to each level is to unify the backbone features of different levels with different channel numbers to the same number of channels, so that the features of different levels can be added element by element. "+" indicates bilinear interpolation upsampling, which magnifies the feature map to the next finer spatial resolution; "+" indicates element-wise addition, used to inject deep semantic information into shallow layers. The function of this 3×3 smooth convolution is to suppress the aliasing introduced by upsampling and avoid generating false edges that interfere with subsequent matching. For the first The intermediate fusion features after adding the layers For the first The final fusion feature output of the hierarchy.

[0042] As can be seen from equations (7)-(9), the top-down path extends all the way to... Unlike the baseline, which discards the highest resolution layer (spatial resolution 128×256), fine-grained geometric details at the edge of the point cloud projection are fully preserved in the feature matching stage. The fused features have stronger spatial localization capabilities while maintaining multi-scale semantic consistency.

[0043] The final output of this branch Figure 3 The fusion features at each level are marked with red arrows in the middle. , , The number of layers in the depth map branch output is the same as that of the coarsest layer; the outputs of the same layer of the two branches are fed into the cross-modal attention feature matching layer in step (3).

[0044] (3) Cross-modal attention feature matching mechanism.

[0045] The feature matching layer employs a cross-modal attention mechanism. , , The three matrices are generated by convolution: Obtained by convolution of RGB branch features. , It is obtained by convolution of branch features of the depth map.

[0046] Baseline methods (such as LCCNet and MRCNet) calculate the spatial correlation between RGB pixels and depth map points within a predefined neighborhood for matching, and the interaction is restricted to a fixed local pattern; the attention mechanism of this invention allows... Selectively aggregate from across the entire feature map , This information enables more flexible and adaptive cross-modal associations. To reduce computational cost while preserving global context, , , All are generated by 1×1 convolutions with a stride of 2 (while preserving the number of channels). (At the same time, the spatial resolution is halved). The relevant formula is as follows: (10) (11) in, A feature map containing RGB image information. A feature map containing depth map point information. To and Homologous value feature maps Indicates matrix transpose. For the number of channels, This indicates the normalized exponent operation performed row by row. It represents the correlation between RGB pixels and depth map pixels (containing rotation and translation information). This is the output of this layer (including calibration information). This module is implemented as a single-head scaled dot-product attention mechanism, does not use explicit positional encoding, and is applied independently at multiple levels of the feature pyramid, thus modeling cross-modal correspondences at both coarse and fine scales. The globally relevant features obtained from the matching are processed by a global regression layer (structure as follows). Figure 4 (As shown) Output translation and rotation drift. Figure 4 As shown, the global regression layer takes globally relevant features as input and is divided into two parallel branches: translation and rotation. Each branch consists of three fully connected layers. Figure 4 The vector is composed of "Linear × 3" and outputs a 3D translation vector. With 4-dimensional rotation quaternions .

[0047] (4) Loss function. The loss function of GRNet consists of two parts: regression loss and mixed loss.

[0048] The functions and purposes of the two loss components are as follows. The regression loss directly affects the extrinsic drift parameters of the network output: the translation component is measured using SmoothL1, and the rotation component using quaternion angular distance. Its function is to constrain the difference between the predicted and true values ​​in the parameter space, ensuring that the output translation vector and rotation quaternion converge to the correct values. It also utilizes the characteristic of SmoothL1 approximating L1 at large errors to suppress excessive gradients caused by outliers, enabling the network to converge stably even with large-scale (e.g., 0.5 m, 5°) miscalibrations. The hybrid loss affects the point cloud reprojection error determined by the extrinsic drift: it weights the error of each projection point according to the optical center distance and depth weights. Its function is to constrain the projection consequences of the predicted extrinsic parameters in the imaging geometric space, compensating for the regression loss's deficiency of "constraining only parameters, not geometry"—on the one hand, it automatically reduces the error contribution of distant, low SNR points, suppressing the interference of ranging noise on optimization; on the other hand, it focuses supervision on pixels more sensitive to changes in extrinsic parameters, thereby improving rotation accuracy and optimization robustness. The two are complementary: regression loss provides direct supervision of the parameter space and determines the direction of convergence; hybrid loss provides consistent supervision of the geometric space and determines the achievable accuracy; finally, the weights are applied... , The linear combination is a joint optimization of the total loss.

[0049] Regression loss includes translation vector loss and rotation quaternion loss. Translation vector error is metric using SmoothL1 loss, and rotation quaternion error is metric using angular distance. (12) (13) (14) in, For translation vector loss, This represents the smoothing L1 loss function, which takes a quadratic term when the absolute value of the error is less than a threshold, and a linear term otherwise. For rotation quaternion loss, Represents the angular distance (in degrees) between two unit quaternions; This is the true value of the translation vector (unit: m). The truth value of the rotation quaternion; , These are the translation vector and rotation quaternion predicted by the network, respectively; For regression loss; , These are the weights of the translation loss term and the rotation loss term, respectively, and are hyperparameters that are not less than zero. In this embodiment, we take... .

[0050] Hybrid loss: This is a loss resulting from a combination of depth loss and optical center distance loss. When mapping point clouds to depth maps, errors are mainly introduced from two aspects. First, the depth of the points: distant points in the LiDAR point cloud have low signal strength, are susceptible to noise, and have a low signal-to-noise ratio (SNR). Therefore, weighting the error term by depth can suppress the adverse effects of distant points on the calibration results and enhance the robustness of optimization. Second, the depth is affected by the camera's imaging characteristics. The larger the point The closer to the optical center, the better. The calculation of the planar coordinates and the two types of weights is as follows: (15) (16) (17) (18) (19) (20) in, , These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image (unit: pixels). , These are the x and y coordinates (in pixels) of the camera's optical center (principal point) on the image plane, all taken from the camera intrinsic parameter matrix. ; , , For the first The three-dimensional coordinates of a point in the camera coordinate system, where That is, the depth of that point; , These are the x and y coordinates of the point on the image plane; Describes the 2-norm of a vector. Indicates all Summing the projection points, This represents the total number of projection points involved in the calculation for this frame. The weights are point-by-point based on the distance from the optical center, which increase as the distance from the image point to the optical center increases, and are summed to 1 over all points; It is a very small positive number, used to prevent the denominator from being zero; It is a depth-based point-by-point weight, which decreases as the depth of the point increases; The point-by-point weights are obtained by combining the two equally weighted components. It should be noted that in this step... For point-by-point weighting, the same as the cross-modal correlation representation in the aforementioned attention mechanism. Different meanings; For the first Homogeneous coordinates of the lidar points The truth matrix of the extrinsic parameters. The extrinsic parameter drift matrix predicted by the network. and These are the coordinates obtained by projecting the point onto the image plane using the true extrinsic parameters and the predicted extrinsic parameters; The projection error is the weighted average of all projected points in the frame, i.e., the hybrid loss. Both the optical center distance and depth terms originate from the same physical quantity, "distance from a point to the sensor," and are normalized to the same scale; therefore, they are combined with equal weights (half each). This equal weighting configuration is verified using a homoscedastic uncertainty framework. During convergence, the optical center-depth normalized weights stabilize at 0.50:0.50, proving that equal weighting is optimal. The total loss is: (twenty one) in, The mixed loss is determined by the aforementioned equations (15)-(20); The regression loss is determined by the aforementioned equation (14); , , respectively, are the weights of the mixed loss and the regression loss, and are hyperparameters not less than zero; in this embodiment, is taken as . ; This represents the total loss for a single frame sample, calculated during training for all frames within a batch. After averaging, backpropagation is performed, and the Adam optimizer is used to update the network parameters.

[0051] (5) External parameter drift correction reasoning (multi-range iteration from coarse to fine).

[0052] Unlike conventional models that directly predict extrinsic parameters, GRNet outputs the extrinsic parameter drift between the LiDAR and the camera. During training, the input... Represents the initial, misaligned extrinsic parameters, and the network prediction correction matrix. Final calibration results Calculate using the following formula: (twenty two) During reasoning, For sensor extrinsic parameters that may drift in real-world scenarios, GRNet predicts the corresponding drift. The above formula is used to obtain the extrinsic parameters of the current alignment, thereby realizing online misalignment correction based on predicted drift.

[0053] GRNet employs an iterative optimization strategy involving multiple networks and multiple ranges: three networks with identical structures but trained for different mislabeling ranges, with translation and rotation ranges respectively selected. ( ={0.5, 0.2, 0.1} m) and ( ={5°, 2°, 1°}).

[0054] In the first iteration, the RGB image and projected depth map at the maximum range (0.5 m, 5°) are input into the network for initial prediction and correction. Then, the network is used... An updated depth map is generated by reprojecting the point cloud, and then input into a second, smaller-range (0.2 m, 2°) network along with the original RGB image for refinement. This process is repeated for a third, smaller-range (0.1 m, 1°) network to obtain the final extrinsic calibration parameters. This multi-range iterative strategy allows each network to focus on progressively finer corrections, thereby improving the final calibration accuracy in a structured manner.

[0055] Technical Results: On the KITTI odometry dataset (sequences 01–20 for training and validation, sequence 00 for testing), the present invention achieves a mean absolute error of 0.3487 cm for translation and a mean absolute error of 0.0382° for rotation. Visual contrast before and after correction is shown in the figure. Figure 5 As shown, the absolute error distributions for translation and rotation are respectively as follows: Figure 6 , Figure 7 As shown in the diagram. Comparative experiments show that compared to the baseline MRCNet, translational drift estimation is improved by approximately 2.1%, and rotational drift estimation is improved by approximately 15.1%. It also outperforms RobustCalib (translation 0.806 cm, rotation 0.046°) within the same range (±0.1 m, ±1°). Ablation experiments show that removing the full-resolution fusion module significantly increases translational variance, removing cross-modal attention significantly increases rotational error, and removing depth weighting significantly impairs rotational accuracy, indicating that the three components are complementary and indispensable. In cross-dataset experiments on nuScenes (v1.0-mini), under the same lightweight fine-tuning protocol, the rotational error is reduced by approximately 56.6% relative to the baseline, and the standard deviations of translational and rotational predictions are smaller, indicating stronger generalization and more stable predictions.

[0056] Compared with the prior art, the key differences of this invention (and the technical solution for which protection is sought) are as follows, and it can be implemented individually or in any combination: (1) A full-resolution feature fusion module based on FPN that retains the highest resolution: A symmetrical ResNet-18+FPN dual-branch is used for RGB and depth maps. The modules are upsampled layer by layer from top to bottom and added element by element with the lateral features. Then, they are smoothed by 3×3 convolution. The key is to retain and fuse all layer features, including the highest resolution layer, in order to retain fine-grained geometric information and improve the quality of the fused representation. The principle is that the bottom-up downsampling of ResNet-18 gives deep features a large receptive field and strong semantics, but low spatial resolution and blurred geometric localization. On the contrary, shallow (highest resolution) features retain fine-grained geometric details such as the edge of the point cloud projection, but have weaker semantics. Extrinsic parameter drift correction is essentially a pixel-level geometric alignment problem. Its accuracy depends on whether the features can simultaneously possess the two attributes of "accurate localization" and "semantic consistency". Therefore, this module uses a top-down approach to progressively upsample deep semantics and add them element-wise to shallow features at the same level, ensuring that the fused features at each level carry both semantic and geometric information. Before addition, 1×1 convolutions are used to align channels, and after addition, 3×3 convolutions are used for smoothing, suppressing high-frequency aliasing caused by bilinear upsampling and avoiding false edges interfering with subsequent matching. Crucially, this top-down approach extends all the way to the highest resolution level, unlike the baseline which discards it, thus preserving fine-grained geometric details in the feature matching stage and improving the spatial consistency of translation component estimation.

[0057] (2) A cross-modal attention mechanism for lidar-camera feature matching: Generated from RGB features and Generated from depth map features, all three are downsampled using a 1×1 convolutional projection with a stride of 2; single-head scaling dot product attention is used. , A globally adaptive cross-modal correspondence is established across the entire feature map and applied independently at multiple levels of the feature pyramid. Its principle and technical characteristics are as follows: the correct correspondence between RGB pixels and depth map pixels is determined by the extrinsic parameters to be determined. When the extrinsic parameters drift, the deviation of the imaging position of the same physical point in the two modalities may far exceed that of the local neighborhood. The baseline establishes the correspondence based on the spatial correlation (cost volume) within a fixed neighborhood, essentially assuming that the corresponding point must be located within a predefined neighborhood. Once the drift is large or the scene contains long-range geometric structures, it cannot be covered. The principle of this scheme is to explicitly calculate the global similarity matrix between the two modal features using scaled dot product attention. , Each line gives the correlation between a given RGB position and all positions in the depth map, and then... Weighted aggregation of depth map features based on this relevance is equivalent to searching for correspondences across the entire map in a data-driven manner. This results in the following technical characteristics: First, the effective receptive field is the entire feature map, and correspondences are not constrained by predefined neighborhoods, allowing for the modeling of long-range geometric relationships, which is particularly beneficial for rotation component estimation; second, the aggregation weights are dynamically generated from the input content, exhibiting adaptability rather than fixed relevance templates; third... , , All are generated by 1×1 convolutions with a stride of 2, while maintaining the number of channels. Without changing the spatial resolution, the similarity matrix is ​​halved only when ( )×( The computation is performed at 16 positions, reducing the dot product computation to about 1 / 16 of the original. The module has about 0.84 M parameters, which is less than 1% of the total network size of 116.59 M. Fourth, the convolutional projection and subsequent row-first rearrangement implicitly preserve the spatial order, so there is no need for explicit positional encoding. Fifth, the same attention module can be applied independently at multiple levels of the feature pyramid, and cross-modal correspondences can be established at both coarse and fine scales.

[0058] (3) A geometrically consistent hybrid loss: weights based on optical center distance Weights based on point cloud depth Combined with equal weights (half each) Projection error from point cloud to image Pixel-wise weighting provides geometrically consistent supervision for depth projection. The effect is that the hybrid loss extends supervision from the parameter space to the imaging geometry space, assigning supervision intensity according to the physical reliability of each point. Depth weights This automatically reduces the error contribution from points at long distances and with low signal-to-noise ratios, preventing the network from fitting unreliable ranging noise; optical center distance weighting. The normalized distance from the image point to the optical center (OCC) assigns greater weight to points whose projection positions are more sensitive to changes in external parameters, concentrating supervision on pixels with higher information content. Both weights are derived from the same physical quantity, "distance from the point to the sensor," and normalized to the same scale, hence they are combined with equal weights (half each). This equal weighting configuration is validated using a homoscedastic uncertainty framework (assigning learnable log-variance to each item). At training convergence, the normalized OCC-depth weights stabilize at 0.50:0.50, indicating that equal weighting is optimal and no manual parameter tuning is required. The quantization results are as follows: with other modules remaining unchanged, removing the depth weighting term increases the mean rotation error from 0.126° to 0.149° and the median from 0.090° to 0.152°; removing the OCC distance term increases the translation error... The mean value increased from 0.876 cm to 0.914 cm; removing both values ​​simultaneously degraded the values ​​to 1.027 cm and 0.151°. This demonstrates that the two values ​​are complementary, and their combined use achieves optimal performance on both translation and rotation metrics, providing geometrically consistent and noise-resistant supervision for depth projection.

[0059] (4) An inference scheme for output extrinsic parameter drift and multi-range iterative correction from coarse to fine: network prediction drift ,according to The calibration results were obtained; three isomorphic networks with decreasing ranges (e.g., 0.5 m / 5° → 0.2 m / 2° → 0.1 m / 1°) were cascaded, with each level using the reprojected updated depth map as the input to the next level. The effects of this technical solution are: firstly, the network output reflects the extrinsic parameter drift. Instead of the absolute value of the external parameter, the calibration result is based on The network only needs to learn a bounded quantity, "the deviation between the current extrinsic parameter and the true value," which is independent of the sensor's installation location. This reduces the learning difficulty and allows the method to be directly used for online drift correction—using the potentially drifted current extrinsic parameter as a reference during operation. The corrected extrinsic parameters can be obtained simply by inputting the data, without any calibration materials or requiring system shutdown for calibration. Secondly, the multi-range cascading from coarse to fine allows each network level to learn finer corrections within a smaller range, and each level uses the results of the previous level to reproject the point cloud and generate a depth map with smaller errors as the input for the next level, thus compressing the error step by step: on the KITTI sequence 00 test set, the translation error after passing through the first-level (0.5 m, 5°) network is significantly reduced. The mean value is 2.2433 cm, and the rotational error is... The mean was 0.2161°; after the second-level network (0.2 m, 2°), it decreased to 1.2718 cm and 0.1657°; after the third-level network (0.1 m, 1°), it decreased to 0.8760 cm and 0.1269°, and the error distribution became more concentrated. The standard deviation decreased from 1.0767 cm to 0.4997 cm. Figure 6 , Figure 7 As shown. Third, since each stage of the cascade has the same structure and only the training range is different, a large-range model can be used as the pre-training weight for a small-range model (200 rounds for the large range and 100 rounds for each of the small ranges), reducing training costs; the number of stages, ranges and iterations can be increased or decreased according to accuracy and computing power requirements, making it easy to tailor in engineering.

[0060] (5) Any combination of (1), (2), (3), and (4) above, and using SmoothL1 translation loss and quaternion angular distance rotation loss to form regression loss within this framework, and training it jointly with the hybrid loss. The implementation method of ).

[0061] Description of Implementation and Variations: The specific implementation of the present invention can also be replaced or modified as follows.

[0062] (1) Replacement of backbone network: In addition to ResNet-18, the feature extraction backbone can be replaced by ResNet-34 / 50, other convolutional backbones or Transformer-based backbones; the number of layers and channels of FPN fusion can be adjusted accordingly.

[0063] (2) Variations of the attention mechanism: Single-head scaling dot product attention can be replaced with efficient variants such as multi-head attention, window attention, or linear attention to reduce computational cost; explicit positional encoding can be added; generating , , The convolution stride and kernel size can be adjusted.

[0064] (3) Variation of hybrid loss: In addition to fixed equal weights, the weights of the optical center distance term and the depth term can adopt learnable weights based on homoscedastic uncertainty or other weighting strategies; in addition to the L1 form, hybrid loss can include L2 loss, spectral loss, perceptual loss and their weighted combinations.

[0065] (4) Variations of regression and inference: The global regression layer can be compressed by low-rank decomposition, global pooling, etc.; the number of stages of multi-range iteration, the range of each stage and the number of iterations can be increased or decreased according to the accuracy and computing power requirements; the weights of each stage of the network can be shared or not shared.

[0066] (5) Deformation of input structure: The method of generating depth map (such as different projection and interpolation schemes), the perturbation sampling distribution of initial extrinsic parameters, etc. can all be deformed.

[0067] (6) Adaptation and migration variations: Lightweight fine-tuning can be performed in a few rounds on new sensor configurations or datasets (such as nuScenes) to achieve cross-dataset generalization.

[0068] In summary, any scheme that can regress extrinsic parameter drift from RGB images and depth maps, as long as it incorporates one or more of the full-resolution feature fusion, cross-modal attention, and hybrid loss methods of this invention, falls within the protection scope of this invention.

Claims

1. A method for correcting extrinsic parameter drift in lidar and camera systems, characterized in that, Includes the following steps: Step (1): Preprocess the 3D point cloud of the LiDAR using the pinhole camera model, and project the 3D point cloud of the LiDAR into a depth map of the same size as the corresponding RGB image. Step (2): The full-resolution feature fusion module based on FPN takes the RGB image and depth map as input, extracts and fuses all layer features including the highest resolution layer, and outputs paired features that are semantically consistent and retain fine-grained geometric details. Step (3): Establish a globally adaptive pixel correspondence on the paired features using cross-modal attention, and regress the extrinsic parameter drift by the global regression layer to accurately estimate the rotation and translation components; Step (4): Without changing the forward inference structure, supervision is provided only for steps (2) and (3) during the training phase. The regression loss constrains the drift prediction value in the parameter space, and the hybrid loss is weighted pixel by pixel in the imaging geometry space according to the reprojection error between the optical center distance and the point cloud depth. Step (5): In the inference stage, the structure of steps (1) to (3) is reused to decompose a large-range calibration into three progressively fine calibrations through multi-range iteration from coarse to fine, and the single prediction capability of steps (1) to (3) is transformed into the final high-precision calibration result.

2. The method for correcting extrinsic parameter drift of a lidar and camera according to claim 1, characterized in that, The specific steps (1) are as follows: Known data includes camera intrinsic parameter matrix Initial extrinsic parameter matrix of lidar to camera and lidar point clouds Based on the pinhole camera model, the point cloud is first transformed from the lidar coordinate system to the camera coordinate system, and then projected onto the image plane: (1) (2) (3) in, This is the camera intrinsic parameter matrix; is the initial extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system, and is a 4×4 homogeneous transformation matrix; for The 3×3 rotation matrix in for The 3×1 translation vector in the text; For the first The three-dimensional coordinates of a lidar point in the lidar coordinate system These are the 3D coordinates of the point after transformation to the camera coordinate system. This is the projection of the point onto the image plane; variables with "^" represent the homogeneous form of the corresponding coordinates, that is, the vector formed by adding 1 to the end of the original coordinates; Represents the three-dimensional real number space; The index of a point in the point cloud. The number of points involved in the calculation; The coordinates of each point along the optical axis in the camera coordinate system are obtained from equations (1) and (2). The distance from the point to the xoy plane of the camera coordinate system is the depth of the point; the pixel coordinates of the point on the image plane are obtained from equation (3). , , These are the horizontal and vertical coordinates of the pixel, respectively; defining the value of the projected pixel in the depth map. The reciprocal of depth: (4) Repeat the above calculation for all projection points in the point cloud to obtain a depth map of the same size as the corresponding RGB image, which is used as the input for the depth map branch in step (2).

3. The method for correcting extrinsic parameter drift of a lidar and camera according to claim 2, characterized in that, The full-resolution feature fusion module based on FPN in step (2) includes two parallel feature extraction branches: a depth map branch and an RGB image branch. The RGB image branch takes the RGB image captured by the camera as input, and the depth map branch takes the depth map obtained by projecting the point cloud of the lidar onto the image plane as input. The two branches have the same network structure, both including bottom-up and top-down paths, and both are built on the ResNet-18 backbone and FPN structure. The only difference is the modality of the input data and the number of input channels of the first convolution. The network parameters of the two branches are independent and do not share each other. Bottom-up path: The input image is first processed by the first convolutional layer to obtain the feature map. ; After passing through the first to fourth residual layers in sequence, the following results were obtained: , , and The spatial resolution is halved at each level, while the number of channels is doubled at each level. This process can be represented as: (5) (6) in, The input image for this branch is a 3-channel RGB image for the RGB branch and a depth map branch is a depth map obtained by projecting the point cloud. This represents the first convolutional layer of ResNet-18; Indicates the first... One residual layer; For the first The main characteristics of the hierarchy; Top-down path: starting with the coarsest level Starting from the first level, the core features of each level are first uniformly converted to the same number of channels through 1×1 horizontal convolutions. Then, from top to bottom, they are upsampled, added, and smoothed level by level. That is, the fused features of the previous level are upsampled to the spatial resolution of the next finer level through bilinear interpolation, and then added element by element with the features of that level after horizontal convolution. The result of the addition is then smoothed by 3×3 convolution, and so on, to obtain the fused features. , and The process is represented as: (7) (8) (9) in, This is a 1×1 horizontal convolution applied to each layer; "+" indicates bilinear interpolation upsampling, which magnifies the feature map to the next finer spatial resolution; "+" indicates element-wise addition, used to inject deep semantic information into shallow layers. It is a 3×3 smooth convolution; For the first The intermediate fusion features after adding the layers For the first The final fusion features of the hierarchical output; Each branch ultimately outputs the fused features of each level. , , In addition to the features of the coarsest level, the number of levels of the output of the two branches corresponds one-to-one with the number of channels and spatial resolution of each level; the outputs of the same level of the two branches are sent in pairs to the cross-modal attention feature matching layer in step (3).

4. The method for correcting extrinsic parameter drift of a lidar and camera according to claim 3, characterized in that, Step (3) specifically involves: The feature matching layer employs a cross-modal attention mechanism. , , The three matrices are generated by convolution: Obtained by convolution of RGB branch features. , Obtained by convolution of branch features in the depth map; The baseline method calculates the spatial correlation between RGB pixels and depth map points within a predefined neighborhood for matching, restricting the interaction to a fixed local pattern; this reduces computational cost while preserving the global context. , , All are generated by 1×1 convolutions with a stride of 2, and the relevant formulas are as follows: (10) (11) in, A feature map containing RGB image information. A feature map containing depth map point information. To and Homologous value feature maps Indicates matrix transpose. For the number of channels, This indicates the normalized exponent operation performed row by row. This indicates the correlation between RGB pixels and depth map pixels. This is the output of this layer; the global regression layer takes globally relevant features as input and is divided into two parallel branches: translation and rotation. Each branch consists of three fully connected layers, outputting a 3D translation vector. With 4-dimensional rotation quaternions .

5. The method for correcting extrinsic parameter drift of a lidar and camera according to claim 4, characterized in that, Step (4) specifically involves: GRNet's loss function consists of two parts: regression loss and mixed loss. Regression loss includes translation vector loss and rotation quaternion loss. Translation vector error is measured using SmoothL1 loss, and rotation quaternion error is measured using angular distance. (12) (13) (14) in, For translation vector loss, This represents the smoothing L1 loss function, which takes a quadratic term when the absolute value of the error is less than a threshold, and a linear term otherwise. For rotation quaternion loss, This represents the angular distance between two unit quaternions; The true value of the translation vector. The truth value of the rotation quaternion; , These are the translation vector and rotation quaternion predicted by the network, respectively; For regression loss; , These are the weights of the translation loss term and the rotation loss term, respectively, and are hyperparameters that are not less than zero; Hybrid Loss: When mapping point clouds to depth maps, the image plane coordinates and the two class weights are calculated as follows: (15) (16) (17) (18) (19) (20) in, , These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image, respectively. , These are the x and y coordinates of the camera's optical center on the image plane, respectively, and all four are taken from the camera intrinsic parameter matrix. ; , , For the first The three-dimensional coordinates of a point in the camera coordinate system, where That is, the depth of that point; , These are the x and y coordinates of the point on the image plane; Describes the 2-norm of a vector. Indicates all Summing the projection points, This represents the total number of projection points involved in the calculation for this frame. The weights are point-by-point based on the distance from the optical center, which increase as the distance from the image point to the optical center increases, and sum over all points equals 1. It is a very small positive number, used to prevent the denominator from being zero; It is a depth-based point-by-point weight, which decreases as the depth of the point increases; The point-by-point weights are obtained by combining the two equally weighted combinations. For the first Homogeneous coordinates of the lidar points The truth matrix of the extrinsic parameters. The extrinsic parameter drift matrix predicted by the network. and These are the coordinates obtained by projecting the point onto the image plane using the true extrinsic parameters and the predicted extrinsic parameters; The projection error is the weighted average of all projection points in the frame, i.e., the mixing loss. The total loss is: (21) in, The mixed loss is determined by the aforementioned equations (15)-(20); The regression loss is determined by the aforementioned equation (14); , , respectively, are the weights of the mixed loss and the regression loss, and are hyperparameters that are not less than zero; This represents the total loss for a single frame sample, calculated during training for all frames within a batch. After averaging, backpropagation is performed, and the Adam optimizer is used to update the network parameters.

6. The method for correcting extrinsic parameter drift of a lidar and camera according to claim 5, characterized in that, Step (5) specifically involves: GRNet outputs the extrinsic parameter drift between the LiDAR and the camera, and the input during training is... Represents the initial, misaligned extrinsic parameters, and the network prediction correction matrix. Final calibration results Calculate using the following formula: (22) During reasoning, For sensor extrinsic parameters that may drift in real-world scenarios, GRNet predicts the corresponding drift. The above formula is used to obtain the extrinsic parameters of the current alignment, thereby realizing online misalignment correction based on predicted drift; GRNet employs an iterative optimization strategy involving multiple networks and multiple ranges: three networks with identical structures but trained for different mislabeling ranges, with translation and rotation ranges respectively selected. ( ={0.5, 0.2, 0.1} m) and ( ={5°, 2°, 1°}); In the first iteration, the RGB image and projected depth map at the maximum range (0.5 m, 5°) are input into the network for initial prediction and correction. Then, the network is used... The point cloud is reprojected to generate an updated depth map, which is then input into a second smaller range (0.2 m, 2°) network for refinement along with the original RGB image. This process is repeated for a third smallest range (0.1 m, 1°) network to obtain the final extrinsic calibration parameters. This multi-range iterative strategy allows each network to focus on progressively finer corrections, thereby improving the final calibration accuracy in a structured manner.

Images