Laser radar-camera dynamic calibration method based on multiple attention

Abstract

The invention discloses a laser radar-camera dynamic calibration method based on multiple attention. Laser radar-camera external parameter calibration is carried out through a neural network composed of a multi-frame spatial-temporal feature coding module, a local-global attention space information fusion module, a Transform-based time information fusion module and a calibration external parameter regression module. According to the laser radar-camera dynamic calibration method based on multiple attention, the problem that the receptive field of a traditional convolutional network is limited is effectively solved, sub-centimeter level translation precision and sub-angle level rotation precision are achieved in a complex dynamic scene, and the robustness and accuracy of laser radar-camera system calibration are remarkably improved.

Description

Technical Field

[0001] This invention relates to a dynamic calibration method. In particular, it relates to a dynamic calibration method for a lidar-camera system based on multiple attention. Background Technology

[0002] With the rapid development of intelligent driving and robotics technologies, multimodal perception systems have become a core technological architecture for environmental understanding. LiDAR-camera systems, with their unique advantage of integrating high-precision 3D point clouds and rich texture information, play a crucial role in tasks such as target detection, semantic segmentation, and SLAM. However, the performance boundaries of this system are limited by the extrinsic parameter calibration accuracy of the LiDAR and camera. Small errors in the extrinsic parameter matrix can have a cascading amplification effect in downstream tasks, ultimately affecting the decision-making reliability of the intelligent system.

[0003] While traditional calibration methods have developed a relatively complete theoretical system, their rigid premise of relying on manually extracted features faces fundamental limitations in real-world, complex scenarios. When dealing with dynamic urban scenes, these methods suffer from deficiencies in key technical aspects such as motion occlusion compensation, adaptation to abrupt changes in lighting, and processing of textureless regions, making it difficult to apply the calibration system to production environments.

[0004] Self-supervised calibration methods based on deep learning have gradually become a research focus, but the convolutional neural network architecture used in existing studies has limitations: the local receptive field characteristics of convolution are difficult to model the global spatial correlation of cross-modal data; the static network weight mechanism cannot dynamically adapt to the changing environmental interference in complex urban scenes; and cross-modal feature fusion is confined to simple stitching operations, making it difficult to model the deep semantic correlation between RGB images and depth images.

[0005] Therefore, in order to improve the calibration accuracy of lidar-camera extrinsic parameters and solve the problem of cross-modal spatiotemporal registration error in dynamic scenes, a new extrinsic parameter calibration method is urgently needed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multi-attention-based lidar-camera dynamic calibration method that can achieve sub-centimeter translation accuracy and sub-angle rotation accuracy in complex dynamic scenes.

[0007] The technical solution adopted in this invention is: a dynamic calibration method for LiDAR-camera based on multiple attention, which performs LiDAR-camera extrinsic parameter calibration through a neural network composed of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module; including the following steps:

[0008] 1) Data preprocessing and construction of time series window: Using multi-frame RGB images and LiDAR point cloud sequences, a time series modeling framework based on time series window is constructed, and the LiDAR point cloud is transformed into a projection map through perspective projection;

[0009] 2) Generate perturbation projection map: Apply random perturbation to the original extrinsic parameters to generate perturbation projection map of the point cloud, and construct training samples based on perturbation learning;

[0010] 3) Construct a multi-frame spatiotemporal feature encoding module. The multi-frame spatiotemporal feature encoding module uses an improved ResNet-18 network to extract multi-scale feature maps of RGB images and projection maps from the perturbation learning training samples, and fuses the spatiotemporal information of multi-scale feature maps of different resolutions through a feature pyramid network to generate a fused feature map.

[0011] 4) Construct a spatial information fusion module based on local-global attention. The spatial information fusion module based on local-global attention designs a local-global collaborative self-attention network, enhances the local feature correlation of the fused feature map through a sliding window mechanism, and models global spatial dependencies to generate spatial feature maps.

[0012] 5) Construct a Transformer-based temporal information fusion module. The Transformer-based temporal information fusion module is a Transformer-based encoder-decoder architecture that unfolds the spatial feature map into embedded feature vectors and fuses the cross-frame temporal information of the embedded feature vectors through a multi-head attention mechanism in the temporal dimension to generate a temporal feature map.

[0013] 6) Construct a calibration extrinsic regression module. The calibration extrinsic regression module performs regression on the time feature map through a two-branch fully connected layer, and outputs the translation vector and rotation quaternion of the extrinsic parameters respectively.

[0014] 7) The training of the neural network, consisting of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module, is supervised by combining pose regression loss and point cloud geometric loss.

[0015] This invention presents a multi-attention-based dynamic calibration method for LiDAR-camera systems, effectively addressing the limitation of the receptive field in traditional convolutional networks. It achieves sub-centimeter translational and sub-angle rotational accuracy in complex dynamic scenes, significantly improving the robustness and accuracy of LiDAR-camera system calibration. It offers the following advantages:

[0016] 1. Spatiotemporal Feature Cooperative Perception: Through the collaborative design of temporal sliding window and feature pyramid, the system can capture the continuous motion trend of the sensor in dynamic scenes, enhance the observability of translation vector, and significantly improve the stability of translation solution compared with single frame input.

[0017] 2. Optimization of multiple attention mechanisms: This invention innovatively constructs a local-global collaborative self-attention mechanism, establishes a large receptive field feature association in the spatial dimension, and models cross-frame temporal dependencies in the temporal dimension through the Transformer encoder-decoder architecture, effectively solving the problem of limited receptive field in traditional convolutional networks.

[0018] 3. Strong adaptability to dynamic scenes: The temporal attention in this invention effectively suppresses the uncertainty of single-frame features in dynamic scenes by dynamically weighting the correlation of features at different times, and supports feature completion of dynamically occluded areas by associating historical frame features through cross-frame attention.

[0019] 4. Multi-scene generalization ability: The algorithm proposed in this invention also shows excellent performance in the calibration of solid-state LiDAR-camera systems, proving the robustness and accuracy of the proposed algorithm in complex scenarios. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the architecture of a LiDAR-camera dynamic calibration method based on multiple attention according to the present invention.

[0021] Figure 2 This is a schematic diagram of the expansion of the local self-attention sliding window in this invention;

[0022] Figure 3 This is a schematic diagram of the Transformer architecture in this invention;

[0023] Figure 4 This is a visualization of the calibration results in a specific embodiment of the present invention. Detailed Implementation

[0024] The following detailed description of a multi-attention-based lidar-camera dynamic calibration method of the present invention, with reference to embodiments and accompanying drawings, is provided. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the invention can be practiced in many other ways than those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] This invention provides a dynamic calibration method for LiDAR-camera based on multiple attention. This method uses a neural network comprised of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module to calibrate the extrinsic parameters of the LiDAR-camera. Figure 1 As shown, the specific steps include the following:

[0026] 1) Data preprocessing and construction of time series window: Using multi-frame RGB images and LiDAR point cloud sequences, a time series modeling framework based on time series window is constructed, and the LiDAR point cloud is transformed into a projection map through perspective projection;

[0027] This invention uses the KITTI odometry dataset as the experimental benchmark, with point cloud data (image_2 sequence) from the left-side color camera and (velodyne sequence) from the Velodyne HDL-64E mechanical lidar as multi-source inputs. In the preprocessing stage, the image resolution is uniformly adjusted to 1024×512 using nearest-neighbor interpolation to adapt to the network input, while the lidar point cloud is uniformly sampled to maintain consistent data volume per frame.

[0028] The temporal window construction method described above is as follows: for consecutive frames of RGB images and LiDAR point cloud sequences, a time window containing... The time-series window of the frame constructs a sample containing an RGB image and a LiDAR point cloud for each frame within the time-series window. This embodiment of the invention uses a continuous... A frame is used to construct a serialized input for a time window, and all data is strictly aligned using timestamps to capture motion continuity. For the first frame... Window sample of the frame :

[0029] ;

[0030] in For the first Frame RGB image, These are the height and width of the image, respectively; For the first Resampled point cloud in the frame lidar coordinate system, superscript Indicates the coordinate system of the lidar. For the resampled point cloud One point, This represents the number of points in the resampled point cloud. For the first Frame point cloud reflection intensity;

[0031] The projection map generation process is as follows: Define the lidar coordinate system. With camera coordinate system Spatial transformation relationship, extrinsic parameter matrix Decomposed into rotational components With translation components , Represents the real number field;

[0032] raw point cloud of lidar The first in Points Its homogeneous coordinate form Mapped to the camera coordinate system via rigid body transformation, where For the XYZ coordinate components of a point:

[0033] ;

[0034] in, The first point cloud in the camera coordinate system One point;

[0035] Based on the camera intrinsic parameter matrix Projecting the 3D point cloud onto a 2D pixel coordinate system yields the first... pixel coordinates of points Generate a two-channel projection map containing depth information and reflection intensity information, wherein The first The horizontal and vertical pixel coordinate components of each point.

[0036] 2) Generate perturbation projection map: Apply random perturbation to the original extrinsic parameters to generate perturbation projection map of the point cloud, and construct training samples based on perturbation learning;

[0037] The disturbance is: the point cloud in the camera coordinate system Apply random perturbation Generate perturbed point cloud The perturbation parameters are uniformly sampled within a preset 6-DOF parameter space, and the perturbation transformation is as follows:

[0038] ;

[0039] in, The point after the disturbance; Disturbance The corresponding rotation and translation components;

[0040] The network is trained to learn the inverse transformation of extrinsic parameter calibration by comparing the differences between the projection images and point clouds before and after the perturbation.

[0041] 3) Construct a multi-frame spatiotemporal feature encoding module. The multi-frame spatiotemporal feature encoding module uses an improved ResNet-18 network to extract multi-scale feature maps of RGB images and projection maps from the perturbation learning training samples, and fuses the spatiotemporal information of multi-scale feature maps of different resolutions through a feature pyramid network to generate a fused feature map.

[0042] The improved ResNet-18 network removes the max-pooling layers of the original ResNet-18 and introduces lateral connections to generate multi-scale feature maps. The resolutions of the multi-scale feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input size, respectively. The construction of the multi-scale feature pyramid to fuse spatiotemporal information of different resolutions involves fusing high-level semantics and low-level details step by step by connecting the multi-scale feature maps output by the improved ResNet-18 network through lateral connections and upsampling operations.

[0043] 4) Construct a spatial information fusion module based on local-global attention. The spatial information fusion module based on local-global attention designs a local-global collaborative self-attention network, enhances the local feature correlation of the fused feature map through a sliding window mechanism, and models global spatial dependencies to generate spatial feature maps.

[0044] The local-global collaborative self-attention network is implemented by concatenating local self-attention modules and global self-attention modules, such as... Figure 2 As shown, the local self-attention module enhances the feature correlation of local regions through a sliding window mechanism. A sliding window is used on each feature map layer to calculate the local self-attention weights of the features within the window. :

[0045] ;

[0046] in, Input the feature map for the current layer; This is the weight matrix; For feature dimensions; To expand the feature map into a sliding window operation, For batch size, The time window size, For the number of channels, To adjust the sliding window size; for The inverse operation; The number of sliding windows; For the first Self-attention weights for each sliding window; This refers to all sliding windows after expansion; The output of local self-attention;

[0047] The global self-attention module focuses on modeling dependencies across the entire image. It is based on the sliding window of the local self-attention module, with each window sharing the same global weights. A multi-head attention mechanism is used to calculate the attention weights across the entire image.

[0048] 5) Construct a Transformer-based temporal information fusion module. This module involves building a Transformer-based encoder-decoder architecture to fuse spatial feature maps. Expanded into a sequence form through spatial dimensions , For the height and width of the spatial feature map, a linear transformation is performed. Projected into the embedded space and overlaid with position encoding. The embedded feature vectors are generated, and finally, the cross-frame temporal information of the embedded feature vectors is fused through a multi-head attention mechanism in the temporal dimension to generate a temporal feature map.

[0049] like Figure 3 As shown, the construction of the Transformer-based encoder-decoder architecture includes: the encoder is composed of... It consists of cascaded temporal attention layers, each layer containing a temporal multi-head attention (T-MHA) and a feedforward network (FFN):

[0050] ;

[0051] in, For the first Layer embedding feature vectors, For the first Layer-encoded feature vectors, For the first Layer embedding feature vectors;

[0052] The decoder uses learnable parameters As an initial query, calibration-sensitive features are extracted from the encoded features through cross-attention and associated with historical frame features.

[0053] 6) Construct a calibration extrinsic regression module. The calibration extrinsic regression module performs regression on the time feature map through a two-branch fully connected layer, and outputs the translation vector and rotation quaternion of the extrinsic parameters respectively.

[0054] This includes a bi-branch fully connected layer that reduces the dimensionality of multi-scale feature maps using two parallel fully connected layers, mapping them to 3D and 4D output spaces respectively, corresponding to translation vectors. With rotation quaternions The quaternion output is then normalized. These are the XYZ coordinate components corresponding to the translation. For scalar components of quaternions, These are the vector components of a quaternion.

[0055] 7) Supervise the training of a neural network consisting of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module by combining pose regression loss and point cloud geometric loss; including:

[0056] The training of the neural network is supervised by combining pose regression loss and point cloud geometric loss:

[0057] ;

[0058] in, For the total loss, These are the translation loss weights, rotation loss weights, and point cloud geometric loss weights, respectively. These are translation loss, rotation loss, and point cloud geometric loss, respectively.

[0059] The translation loss is calculated using smoothed L1 loss to predict the translation. With actual translation Differences:

[0060] ;

[0061] Rotational loss calculation and prediction quaternion With real quaternions Differences in angles :

[0062] ;

[0063] ;

[0064] in, Represents quaternion multiplication; They are vectors , This is the quaternion converted to rotational axis components in axis-angle form.

[0065] Point cloud geometric loss calculation predicts the Euclidean distance between corresponding points in the point cloud and the real point cloud:

[0066] ;

[0067] in, To predict external parameters, This represents the number of points in the resampled point cloud.

[0068] 8. Experimental Verification

[0069] The experiment used the PyTorch library to implement the network, and trained it on an Nvidia RTX-3080 GPU using the Adam optimizer. The initial learning rate was set to 1e-4, the batch size was set to 8, and the network was trained for a total of 200 epochs.

[0070] With a perturbation amplitude of 5° / 0.5m, the network achieves sub-centimeter translation accuracy and sub-angle rotation accuracy. The mean absolute error (MAE) of the translation vector is 0.682cm (x, y, z: 0.538cm, 0.639cm, 0.869cm), and the MAE of the Euler angles of rotation is 0.074° (roll, pitch, yaw: 0.051°, 0.093°, 0.078°). The calibration results are visualized as follows: Figure 4 As shown in the figure, (a) is the perturbation projection map, (b) is the predicted projection map, and (c) is the true projection map.

[0071] Finally, it should be noted that the above-described embodiments are merely one implementation method of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be pointed out that those skilled in the art can make various modifications and improvements to the technical solutions of this invention without departing from the essence and scope of the invention.

Claims

1. A dynamic calibration method for lidar-camera based on multiple attention, characterized in that, LiDAR-camera extrinsic parameter calibration is performed using a neural network consisting of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module; the steps include: 1) Data preprocessing and construction of time series window: Using multi-frame RGB images and LiDAR point cloud sequences, a time series modeling framework based on time series window is constructed, and the LiDAR point cloud is transformed into a projection map through perspective projection; 2) Generate perturbation projection map: Apply random perturbation to the original extrinsic parameters to generate perturbation projection map of the point cloud, and construct training samples based on perturbation learning; 3) Construct a multi-frame spatiotemporal feature encoding module. The multi-frame spatiotemporal feature encoding module uses an improved ResNet-18 network to extract multi-scale feature maps of RGB images and projection maps from the perturbation learning training samples, and fuses the spatiotemporal information of multi-scale feature maps of different resolutions through a feature pyramid network to generate a fused feature map. 4) Construct a spatial information fusion module based on local-global attention. The spatial information fusion module based on local-global attention designs a local-global collaborative self-attention network, enhances the local feature correlation of the fused feature map through a sliding window mechanism, and models global spatial dependencies to generate spatial feature maps. 5) Construct a Transformer-based temporal information fusion module. The Transformer-based temporal information fusion module is a Transformer-based encoder-decoder architecture that unfolds the spatial feature map into embedded feature vectors and fuses the cross-frame temporal information of the embedded feature vectors through a multi-head attention mechanism in the temporal dimension to generate a temporal feature map. 6) Construct a calibration extrinsic regression module. The calibration extrinsic regression module performs regression on the time feature map through a two-branch fully connected layer, and outputs the translation vector and rotation quaternion of the extrinsic parameters respectively. 7) The training of the neural network, consisting of a multi-frame spatiotemporal feature encoding module, a local-global attention-based spatial information fusion module, a Transformer-based temporal information fusion module, and a calibration extrinsic parameter regression module, is supervised by combining pose regression loss and point cloud geometric loss.

2. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, The temporal window construction method described in step 1) is as follows: For consecutive frames of RGB images and LiDAR point cloud sequences, a time window containing... A time-series window of frames is used to construct samples containing RGB images and LiDAR point clouds for each frame within the time-series window. For the first frame... Window sample of the frame : ； in For the first Frame RGB image, These are the height and width of the image, respectively; For the first Resampled point cloud in the frame lidar coordinate system, superscript Indicates the coordinate system of the lidar. For the resampled point cloud One point, This represents the number of points in the resampled point cloud. For the first Frame point cloud reflection intensity; The projection map generation process is as follows: Define the lidar coordinate system. With camera coordinate system Spatial transformation relationship, extrinsic parameter matrix Decomposed into rotational components With translation components , Representing the real number field, for the original point cloud of lidar The first in Points Mapped to the camera coordinate system via rigid body transformation, where The XYZ coordinate components of the point are then determined based on the camera intrinsic parameter matrix. The 3D point cloud is projected onto a 2D pixel coordinate system to generate a two-channel projection map containing depth information and reflection intensity information.

3. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, The perturbation mentioned in step 2) is: the point cloud in the camera coordinate system Apply random perturbation Generate perturbed point cloud The perturbation parameters are uniformly sampled within a preset 6-DOF parameter space, and the perturbation transformation is... , The first point cloud in the camera coordinate system One point, The points are perturbed; the network is trained to learn the inverse transformation of extrinsic parameter calibration by comparing the differences between the projection images and point clouds before and after the perturbation.

4. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, The improved ResNet-18 network described in step 3) involves removing the max-pooling layers of the original ResNet-18 and introducing lateral connections to generate multi-scale feature maps. The resolutions of the multi-scale feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input size, respectively. The construction of the multi-scale feature pyramid to fuse spatiotemporal information of different resolutions involves fusing high-level semantics and low-level details step by step by connecting the multi-scale feature maps output by the improved ResNet-18 network through lateral connections and upsampling operations.

5. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, The local-global collaborative self-attention network described in step 4) is implemented by concatenating a local self-attention module and a global self-attention module. The local self-attention module enhances the feature correlation of local regions through a sliding window mechanism. A sliding window is used on each feature map layer to calculate the local self-attention weights of the features within the window. : ； in, Input the feature map for the current layer; This is the weight matrix; For feature dimensions; To expand the feature map into a sliding window operation, For batch size, The time window size, For the number of channels, To adjust the sliding window size; for The inverse operation; The number of sliding windows; For the first Self-attention weights for each sliding window; This refers to all sliding windows after expansion; The output of local self-attention; The global self-attention module focuses on modeling dependencies across the entire image. It is based on the sliding window of the local self-attention module, with each window sharing the same global weights. A multi-head attention mechanism is used to calculate the attention weights across the entire image.

6. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, Step 5) describes constructing a Transformer-based encoder-decoder architecture, including: the encoder is composed of... It consists of cascaded temporal attention layers, each containing a temporal multi-head attention network and a feedforward network. The decoder uses learnable parameters. As an initial query, calibration-sensitive features are extracted from the encoded features through cross-attention and associated with historical frame features.

7. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, Step 6) involves a dual-branch fully connected layer that reduces the dimensionality of the multi-scale feature map using two parallel fully connected layers, mapping it to 3D and 4D output spaces respectively, corresponding to translation vectors. With rotation quaternions The quaternion output is then normalized. These are the XYZ coordinate components corresponding to the translation. For scalar components of quaternions, These are the vector components of a quaternion.

8. The LiDAR-camera dynamic calibration method based on multiple attention as described in claim 1, characterized in that, Step 7) The training of the neural network is supervised by combining pose regression loss and point cloud geometric loss: ； in, For the total loss, These are the translation loss weights, rotation loss weights, and point cloud geometric loss weights, respectively. These are translation loss, rotation loss, and point cloud geometric loss, respectively. The translation loss is calculated using smoothed L1 loss to predict the translation. With actual translation Differences: ； Rotational loss calculation and prediction quaternion With real quaternions Differences in angles : ； ； in, Represents quaternion multiplication; They are vectors , This is the quaternion converted to rotational axis components in axis-angle form. Point cloud geometric loss calculation predicts the Euclidean distance between corresponding points in the point cloud and the real point cloud: ； in, To predict external parameters, This represents the number of points in the resampled point cloud.

Images