Feature-level camera-lidar online calibration method based on Transformer

Abstract

The present invention discloses a Transformer-based feature-level camera-lidar online calibration method. First, a hand-eye estimator is used to coarsely calibrate the camera and lidar. The original lidar point cloud is projected onto the image using the estimated initial six-degree-of-freedom extrinsic parameters to obtain the coarsely calibrated lidar point cloud and camera image. Then, a multi-constrained transformer deep network model is constructed and trained to output more accurate six-degree-of-freedom extrinsic parameters after prediction. The network model includes a token pyramid module, a transformer fuser module, a feature injection module, and a multi-constraint module. The coarsely calibrated lidar point cloud and camera image are input into the multi-constrained transformer deep network for fine calibration. Finally, an iterative optimization strategy is introduced to obtain accurate and robust six-degree-of-freedom extrinsic parameters, further improving the accuracy and robustness of the calibration results.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and deep learning, and mainly relates to a feature-level camera-lidar online calibration method based on Transformer. Background Art

[0002] In recent years, with the advancement of computer vision and simultaneous localization and mapping (SLAM) technology, more and more researchers have begun to focus on unmanned systems composed of lidar and cameras. Lidar has the ability to measure scene depth with high precision, but the point clouds it collects are sparse, with low resolution, and lack color and texture information. Cameras can collect high-resolution color images, but they are sensitive to lighting changes and cannot obtain accurate depth information. Therefore, the fusion of lidar and cameras has been widely used to solve the positioning and perception problems of unmanned systems. As a prerequisite for multimodal fusion, the calibration of extrinsic parameters between lidar and cameras has become a hot research topic.

[0003] External parameter calibration methods between LiDAR and cameras can be categorized as offline and online. The former, also known as target-based calibration, relies on manual calibration using static markers (such as a chessboard or a calibration plate of a specific shape). The latter, also commonly referred to as target-free calibration, does not rely on specific static markers. Instead, it extracts and matches point cloud and image features in overlapping fields of view, estimating extrinsic parameters by constructing an optimization problem.

[0004] However, target-based calibration methods have two major drawbacks in practical applications: First, they rely heavily on manual marking, making the calibration process very time-consuming. Second, when the calibration system begins operating, the position of each sensor may change slightly due to various reasons, such as jitter during drone flight. Over long periods of operation, without online corrections, navigation results will gradually drift. Therefore, accurate and fast online camera-lidar calibration methods are crucial.

[0005] The main idea behind most current targetless methods is to first extract point, line, or surface features from point clouds and images, then design a specific cost function to evaluate the degree of feature matching, and finally iteratively calculate external parameters through optimization. However, features extracted for certain scenarios may fail in other scenarios, and constructing a cost function for matching relationships is also a challenge. This results in insufficient generalization capabilities for online calibration and its inability to adapt to complex environments. Summary of the Invention

[0006] The present invention addresses the existing problems in which the data collected by lidar and camera have large differences in spatial position. Directly inputting them into a later-designed deep network may result in poor data correlation, a lack of in-depth mining of cross-modal information, a lack of attention to the local correlation of lidar and camera data in the global spatial context, and poor generalization ability. The present invention provides a Transformer-based feature-level camera-lidar online calibration method. First, a hand-eye estimator is used to coarsely calibrate the camera and lidar. The lidar raw point cloud is projected onto the image using the estimated initial six-degree-of-freedom extrinsic parameters to obtain the coarsely calibrated lidar point cloud and camera image. Then, a multi-constrained transformer deep network model is constructed and trained to output more accurate six-degree-of-freedom extrinsic parameters after prediction. The network model includes a token pyramid module, a transformer fuser module, a feature injection module, and a multi-constraint module. The coarsely calibrated lidar point cloud and camera image are input into a multi-constrained transformer deep network for fine calibration. Finally, an iterative optimization strategy is introduced to obtain accurate and robust six-degree-of-freedom extrinsic parameters, further improving the accuracy and robustness of the calibration results.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a feature-level camera-lidar online calibration method based on Transformer, comprising the following steps:

[0008] S1, coarse calibration: Use the hand-eye estimator to perform coarse calibration on the camera and lidar. The original lidar point cloud is projected onto the image using the estimated initial six-degree-of-freedom extrinsic parameters to obtain the coarsely calibrated matching lidar point cloud and camera image.

[0009] S2, building a multi-constraint Transformer deep network model: input the matched lidar point cloud and camera image after rough calibration in step S1 into the model, and output a more accurate six-degree-of-freedom external parameter after prediction; the multi-constraint Transformer deep network model includes a Token pyramid module, a Transformer fuser module, a feature injection module and a multi-constraint module, wherein,

[0010] The Token Pyramid module takes the coarsely calibrated matched LiDAR point cloud and camera image as input, generates a Token pyramid, and obtains local features;

[0011] The Transformer Fusion Module is used for multimodal fusion. It takes the local features output by the Token Pyramid Module as input and uses its self-attention mechanism to combine the global context of the camera image and the LiDAR point cloud to provide global features.

[0012] The feature injection module injects the global features output by the Transformer fuser module into local tokens of corresponding scales for feature enhancement;

[0013] The multi-constraint module: weights the six-degree-of-freedom external parameter global constraint, depth projection constraint, and semantic edge constraint to construct a multi-constraint loss function for network model training;

[0014] S3, iterative optimization: Feed the six-degree-of-freedom external parameters predicted in step S2 back to step S1, re-project the lidar point cloud onto the image to obtain a higher matching lidar point cloud and camera image, repeat step S2, and iteratively optimize to obtain the final accurate six-degree-of-freedom external parameters.

[0015] As an improvement of the present invention, the six-degree-of-freedom external parameters in step S1 include the rotation external parameters and translation of external references Among them, C and L represent the camera and lidar coordinate systems respectively, and the rotation extrinsic parameter Specifically:

[0016]

[0017] in, Represents the rotation matrix of the lidar and camera from time i to time i+1, Minimize function, n and m represent the starting frame and the final frame respectively;

[0018] Translation of external parameters The linear least squares method is used to solve the following equation:

[0019]

[0020] in, They represent the translation matrices of the lidar and camera from time i to time i+1, and s represents the scale factor.

[0021] As an improvement of the present invention, the Token pyramid module in the multi-constrained Transformer deep network model in step S2 is composed of stacked MobileNet blocks. The camera image or lidar point cloud is input into the MobileNet block to generate local tokens of different scales. After the local tokens are averagely pooled to a uniform size, they are spliced along the channel dimension to generate new tokens and obtain local features.

[0022] As another improvement of the present invention, in the Transformer fusion module of the multi-constrained Transformer deep network model in step S2, the local features processed by the Token pyramid module are considered to be the tokens {x1, x i …, x n}, a total of n tokens, can be passed through Queries Keys and Values To express:

[0023] Q i =W Q ·x i

[0024] K i =W K ·x i

[0025] V i =W V ·x i

[0026] Among them, W Q ,W K and W V Represents Q i , K i 、V i The weight matrix of the i-th token x i and the jth tokenx j The local attention weight ω between ij As shown in the following formula:

[0027]

[0028] Among them, f represents the dimension of the word vector, Q i The transpose of K j represents the keys of the jth token, and softmax represents the activation function;

[0029] Self-attention mechanism i The output of the jth token can be expressed as ValuesV j The weighted sum of

[0030]

[0031] Through nonlinear transformation, the output of each Transformer block is obtained. After passing through several Transformer blocks, the Transformer fusion module obtains the global features.

[0032] As another improvement of the present invention, in the feature injection module of the multi-constrained Transformer deep network model in step S2, the local tokens of the lidar point cloud and camera image generated by the Token Pyramid module and the global features generated by the Transformer Fusion module are used as inputs, and the local tokens of each dimension are weighted summed with the corresponding global features to obtain the local-to-global information output:

[0033]

[0034] Among them, k represents the kth layer of the Token pyramid; N represents the number of layers of the Token pyramid; δ k It represents the feature weights generated by sending the global features of the kth layer into the 1×1 convolution layer and then undergoing normalization and sigmoid layer; l k g represents the features to be injected after the local tokens of the kth layer are input into the 1×1 convolution layer and normalized; k It means that the global features of the kth layer are processed and normalized by a 1×1 convolution layer, and then upsampled according to the dimensions corresponding to the local tokens.

[0035] As another improvement of the present invention, in the multi-constrained Transformer deep network model in step S2, after obtaining the global features through the Transformer fusion module, it is sent to the pose regressor composed of 2 MLPs, and decoded to obtain the translation extrinsic parameter t containing 1×3 pre and the 1×4 rotation extrinsic quaternion q pre , the six-degree-of-freedom external parameter global constraint loss function L global for:

[0036] L global =L translation +L rotation

[0037] =||t pre -t gt ||2+||q pre -q gt ||2

[0038] Among them, L translation and L rotation represents the loss function of translation and rotation extrinsic parameters, t gt and q gt is the true value of the translation extrinsic parameter and the rotation extrinsic parameter, and ||·||2 represents the two-norm;

[0039] The depth projection loss function Ldepth The calculation is as follows:

[0040]

[0041] Where M represents the number of laser point clouds, It represents the calibration result of the original point cloud of the LiDAR after deep network prediction and rough calibration. Represents the true value point cloud obtained by projecting the true value extrinsic parameter;

[0042] The semantic edge constraint loss function L seg :

[0043]

[0044] Among them, Z represents the number of semantic edge points in the set, F edge represents the set of semantic edge points, represents semantic edges in the point cloud, Represents the corresponding image semantic edge in the image frame, K represents the camera intrinsic parameter matrix, R pre is the rotation extrinsic matrix of the predicted six-degree-of-freedom extrinsic parameters.

[0045] As another improvement of the present invention, the multi-constraint loss function L in the multi-constraint module final Specifically:

[0046] L final =λ1L global +λ2L depth +λ3L seg

[0047] Among them, λ1, λ2, and λ3 are manually set weights.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) In step S1 of the present invention, a hand-eye estimator is used for coarse calibration, which can reduce the cross-modal differences between the lidar and the camera and obtain more matching information in the overlapping field of view.

[0050] (2) The multi-constrained Transformer deep network model constructed in this paper tightly couples the lidar point cloud and camera image, and uses the self-attention mechanism to focus on the local correlation between the lidar and camera in the global spatial context, thereby improving the calibration accuracy.

[0051] (3) The multi-constraint module is introduced into the multi-constraint Transformer deep network model constructed by the present invention, and a loss function including six-degree-of-freedom extrinsic global constraint, depth projection and semantic edge constraint is constructed to combine concrete features with abstract features to improve the robustness and generalization ability of the calibration network. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the working principle of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of the working principle of the Token Pyramid module in the multi-constrained Transformer deep network model of the present invention;

[0054] Figure 3 This is a working principle diagram of the Transformer fuser module in the multi-constrained Transformer deep network model of the present invention;

[0055] Figure 4 This is a working principle diagram of the feature injection module in the multi-constrained Transformer deep network model of the present invention;

[0056] Figure 5 This is a working principle diagram of the multi-constraint module in the multi-constraint Transformer deep network model of the present invention;

[0057] Figure 6 This is a comparative diagram of the external parameter calibration between LCCNet and the method of the present invention on the KITTIOdometry dataset in the test example of the present invention, where

[0058] Figure (a) is a projection comparison of the lidar point cloud under the true value external parameter;

[0059] Figure (b) is the projection of the lidar point cloud under the LCCNet calibration external parameters;

[0060] Figure (c) is a schematic diagram of the projection of the lidar point cloud under the external parameters calibrated by the method of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0062] Example 1

[0063] A feature-level camera-lidar online calibration method based on Transformer, the working principle is as follows Figure 1 As shown, the following steps are included:

[0064] Step S1, coarse calibration: Use the hand-eye estimator to coarsely calibrate the camera and lidar to obtain the initial six-degree-of-freedom extrinsic parameters between the camera and lidar. Use the estimated initial six-degree-of-freedom extrinsic parameters to project the lidar original point cloud onto the image to obtain the coarsely calibrated matching lidar point cloud and camera image.

[0065] Because the data collected by the lidar and camera differ significantly in spatial position, directly inputting them into a later-designed deep network can lead to poor data correlation. Therefore, this step adds an initial extrinsic parameter estimation algorithm based on the hand-eye estimator. During coarse calibration, camera motion is estimated using the visual SFM (structure from motion) algorithm, and lidar motion is estimated using the iterative closest point (ICP) algorithm. The hand-eye estimator then estimates the rigid transformation between the camera and lidar and uses this as the initial six-degree-of-freedom extrinsic parameter. The estimated initial six-degree-of-freedom extrinsic parameters are used to project the original point cloud onto the image to obtain a coarsely calibrated point cloud, reducing the cross-modal differences between the camera and lidar data.

[0066] The six degrees of freedom external parameters that need to be calibrated are divided into rotational external parameters and translation of external references Where C and L represent the camera and lidar coordinate systems, respectively. The camera motion over a period of time is estimated by visual SFM, and the lidar motion over a period of time is estimated by ICP algorithm. The nonlinear optimization is performed by minimizing the following cost function to estimate

[0067]

[0068] in, Represents the rotation matrix of the lidar and camera from time i to time i+1, Minimize function, n and m represent the starting frame and the final frame respectively.

[0069] After being optimized back, It can be solved by the linear least squares method as follows:

[0070]

[0071] in, They represent the translation matrices of the lidar and camera from time i to time i+1, and s represents the scale factor, from which the initial six-degree-of-freedom extrinsic parameters between the camera and lidar can be obtained online.

[0072] Finally, the original lidar point cloud is projected onto the image using the estimated initial six-degree-of-freedom extrinsic parameters to obtain the roughly calibrated and matched lidar point cloud and camera image.

[0073] Step S2: Build a multi-constrained Transformer deep network model: Input the matched lidar point cloud and camera image after coarse calibration in step S1 into the multi-constrained Transformer deep network for fine calibration, and finally predict a relatively accurate six-degree-of-freedom extrinsic parameter between the camera and lidar.

[0074] The multi-constrained Transformer deep network model consists of four components: a token pyramid module, a Transformer fuser module, a feature injection module, and a multi-constraint module. The token pyramid module takes the lidar point cloud and camera image output from step S1 as input and generates a token pyramid. This provides local tokens, reducing the size of the Transformer and lowering computational costs. The Transformer fuser module performs multimodal fusion, tightly coupling the point cloud and image. Its self-attention mechanism combines the global context of the camera image and lidar point cloud to provide global features. These global features are injected into local tokens at the corresponding scale and enhanced by the feature injection module. The feature head decodes semantic features from the camera and lidar branches for subsequent construction of semantic edge constraints. The pose regressor, meanwhile, uses the fused global features as input to construct a global constraint for the six-degree-of-freedom extrinsic parameters. Finally, the multi-constraint module weights the six-degree-of-freedom extrinsic parameters, the depth projection constraint, and the semantic edge constraint to construct a multi-constraint loss function used in network training. The trained network then predicts the six-degree-of-freedom extrinsic parameters online, obtaining relatively accurate six-degree-of-freedom extrinsic parameters for both the camera and lidar.

[0075] The Token Pyramid module consists of stacked MobileNet blocks. MobileNet is a highly efficient convolutional neural network. Its computational complexity and parameter count are relatively small compared to other networks, such as ResNet. Furthermore, the goal of the Token Pyramid module is not to achieve rich semantics or a larger receptive field, but rather to use fewer blocks to build the Token Pyramid, thereby reducing the size of local tokens in the Transformer network.

[0076] like Figure 2 As shown, a frame of camera image in step S1 or LiDAR point cloud is the input, where represents the domain, 3, H, and W represent the RGB channels, height, and width of I, respectively, and 2, H, and W represent the depth and reflection intensity channels, height, and width of P, respectively. Then, I or P is input into the MobileNet block to generate local tokens {A 1 ,…,A N}, where N represents the number of layers in the Token pyramid. Pool these local tokens to a uniform size, e.g. Finally, these local tokens at different scales are concatenated along the channel dimension to generate new tokens, also known as local features. The local features of the image I and the lidar point cloud P are fed into the Transformer fuser module for correlation and generation of global features. Since the number of these tokens is relatively small, the computational cost and parameter size of the Transformer network are significantly reduced.

[0077] In most cases, there is correlation between local features. In order to consider the global spatial context information in the lidar and camera data frames, the present invention designs a Transformer fusion module, which uses the self-attention mechanism of the Transformer to combine the global spatial context of the image and lidar point cloud, taking into account the complementarity between them. Figure 3 As shown in the figure, Transformer takes the local features processed by the Token Pyramid module as input. These local features are considered to be the tokens {x1, x i …, x n}, a total of n tokens, can be passed through Queries Keys and Values To express:

[0078] Q i =W Q ·x i

[0079] K i =W K ·x i

[0080] V i =W V ·x i

[0081] Among them, W Q ,W K and W V Represents Q i , K i 、V i The weight matrix of the i-th token x i and the jth tokenx j The local attention weight ω between ij As shown in the following formula:

[0082]

[0083] Among them, f represents the dimension of the word vector, Q i The transpose of K j represents the keys of the jth token, and softmax represents the activation function. Then, the self-attention mechanism y i The output of the jth token can be expressed as ValuesV j The weighted sum of

[0084]

[0085] Finally, through nonlinear transformations, the output of each Transformer block is obtained. After several Transformer blocks, the Transformer Fusion module obtains global features. A global self-attention mechanism exchanges information between local tokens along the spatial dimension. Furthermore, the Transformer Fusion module deeply fuses lidar and camera data, manipulating local tokens at different scales to obtain complete global features. The resulting global features are fed into the Feature Injection module to further generate rich semantic features and are used to construct six-degree-of-freedom extrinsic global constraints.

[0086] After obtaining the global features, they are added to the local tokens of the Token Pyramid module. 1 ,…,A N There is a significant feature gap between the features and the global features. To solve this problem, a feature injection module is introduced to fuse them. Figure 4 As shown in Figure 2, the local tokens of the lidar point cloud and camera image generated by the Token Pyramid module and the global features generated by the Transformer Fusion module are used as input. The local tokens of the kth layer pass through a 1×1 convolution layer and are then normalized to generate the features to be injected, which are denoted as l k The global features of the kth layer are fed into a 1×1 convolutional layer, and then normalized and processed through a sigmoid layer to generate feature weights, denoted as δ k At the same time, the global features of the kth layer are also processed and normalized by a 1×1 convolution layer, and then upsampled according to the dimensions corresponding to the local tokens of the kth layer, denoted as g k Finally, the local tokens of each dimension are weighted and summed with the corresponding global features to obtain the local-to-global information, which is recorded as output, as shown in the following formula:

[0087]

[0088] Here, k represents the kth level of the token pyramid, and N represents the number of levels in the token pyramid. After feature injection, features from different scales are fused, capturing rich global and local information. The feature head feeds the processed features into a convolutional decoder layer to produce the final semantic features from the lidar and camera.

[0089] In order to deal with the degradation problem in different scenarios, this paper proposes a multi-constraint module, which considers specific features and constructs the loss function in three parts, such as Figure 5 shown.

[0090] Six-DOF external parameter global constraint: After obtaining the global features through the Transformer fusion module, it is sent to the pose regressor composed of 2 MLPs, and decoded to obtain the 1×3 translation external parameter t pre and the 1×4 rotation extrinsic quaternion q pre The six-degree-of-freedom external parameter global constraint loss function L is constructed using the Euclidean distance between the given calibration truth value and the predicted external parameter global , as shown below:

[0091] L global =L translation +L rotation

[0092] =||t pre -t gt ||2+||q pre -q gt ||2

[0093] Among them, L translation and L rotation represents the loss function of translation and rotation extrinsic parameters, t gt and q gt is the true value of the translation extrinsic parameter and the rotation extrinsic parameter, and ||·||2 represents the two-norm.

[0094] Depth Projection Constraints: Raw Point Clouds Generated by LiDAR T can be predicted by deep network pre and the coarsely calibrated T init Calibration is performed and the calibration result is recorded as At the same time, according to the true value external parameter T gt The true point cloud can be projected As shown in the following formula:

[0095]

[0096]

[0097] Where K is the camera intrinsic parameter matrix. In order to optimize the calibration performance of the deep network, it is necessary to quantify the difference between the two point clouds as a constraint. The difference is introduced as the depth projection loss function L depth , can be calculated as follows:

[0098]

[0099] Where M represents the number of laser point clouds.

[0100] Semantic Edge Constraint: After acquiring semantic features through the feature injection module, soft targets (probability distribution) and hard targets (labels) are extracted. The point cloud is then segmented and scanned, and labels are detected for the two nearest neighbors of each scan. If the current point has a different label than the previous or next point, it is classified as an edge point. Finally, the soft targets are added to the detected edge points as semantic edge features.

[0101] If the predicted extrinsic parameters are accurate, then the semantic edges extracted from the lidar and camera data should be aligned. Based on this assumption, the semantic edges in the point cloud are Project it onto the image plane and find the corresponding image semantic edge from the image frame The difference between them is defined as the semantic edge constraint loss function L seg :

[0102]

[0103] Among them, Z represents the number of semantic edge points in the set, F edge represents the set of semantic edge points, represents semantic edges in the point cloud, Represents the corresponding image semantic edge in the image frame, R pre is the rotation extrinsic matrix of the predicted six-degree-of-freedom extrinsic parameters. Using soft targets to construct the loss function can carry more semantic information than hard targets, thereby more reasonably constructing semantic edge constraints.

[0104] Finally, the three constraints are weighted and summed to construct the multi-constraint loss function L final .

[0105] L final =λ1L global +λ2L depth +λ3L seg

[0106] Among them, λ1, λ2, λ3 are manually set weights.

[0107] Step S3, iterative optimization: Feed the six-degree-of-freedom external parameters predicted in step S2 back to step S1, re-project the lidar point cloud onto the image to obtain a higher matching degree between the lidar point cloud and the camera image, repeat step S2, and iteratively optimize to obtain the final accurate six-degree-of-freedom external parameters.

[0108] After predictions by the multi-constrained Transformer network, an iterative optimization strategy is introduced to make the predictions more robust and accurate. The predicted six-degree-of-freedom extrinsic parameters are fed back to the lidar input, and the lidar point cloud is reprojected onto the image, resulting in a more closely matched lidar point cloud and camera image, further reducing the cross-modal discrepancy between the camera and lidar data. The process in step S2 is repeated at least three times to obtain the final calibration result, i.e., the accurate six-degree-of-freedom extrinsic parameters.

[0109] Test Case

[0110] The simulation environment for the Transformer-based feature-level camera-lidar online calibration method experiment is: GPU NVIDIA RTX3060, CPU R7-5800H, Ubuntu 20.04LTS, CUDA 11.0, Pytorch1.8.1, ROS Noetic.

[0111] The KITTI dataset was selected as the dataset, which was collected using a Velodyne-HDLE64 lidar, four Point Gray Flea2 cameras, and an OXTS RT3003 IMU / GNSS integrated navigation system. Sequences 01-20 were used as the training dataset, and sequence 00 was used as the test dataset. To expand the training data, a random deviation ΔT in the range of [-1.5m, 1.5m] / [-20°, 20°] was added to the six-degree-of-freedom external parameters between the lidar and the camera. The true value T of the expanded training data can be obtained. gt =ΔT·T org , where T org is the ground truth of the original training data. The lidar and camera data used in the experiment were calibrated and synchronized. Furthermore, Semantic KITTI, based on the KITTI Odometry dataset, enables the acquisition of large-scale annotated point clouds. The point cloud is projected onto the image plane to generate a sparse depth image with the same field of view. The original RGB image and lidar point cloud are then simultaneously cropped to 960×320 pixels.

[0112] The training parameters of the multi-constrained Transformer deep network are shown in Table 1:

[0113] Table 1 Network model training parameters

[0114] Network parameter category Parameter value Optimizer Adam Learning rate 0.0001 Batch size 64 Epoch 120 Global constraint weight 1 Depth projection constraint weight 0.01 Semantic edge constraint weight 0.2

[0115] In order to make a fair comparison with other methods, the rotation and translation extrinsic errors are used for performance evaluation. The absolute rotation error and absolute translation error are defined as follows:

[0116] e t =||t pre -t gt ||

[0117]

[0118] In addition, the absolute translation errors in the X, Y, and Z directions are calculated separately. Compare with the absolute rotation error in Roll, Pitch, and Yaw directions

[0119] In order to verify the performance of the proposed method, some other state-of-the-art methods were selected for comparison on the KITTI Odometry sequence 00 test set. The results are shown in Tables 2 and 3:

[0120] Table 2 Rotational extrinsic parameter calibration errors of various methods on KITTIOdometry

[0121]

[0122] Table 3. Calibration error of translational extrinsic parameters of various methods on KITTIOdometry

[0123]

[0124] As shown in the table above, on the KITTIOdometry dataset, LCCNet and our method achieve similar performance, but our method still achieves higher overall accuracy than the other methods. LCCNet uses raw data from lidar and cameras to train a deep network, requiring a large amount of data. This results in excessive model parameters and a large model size, making it difficult to apply in engineering projects. Furthermore, this method does not fully account for the close correlation between lidar and camera data.

[0125] The LCCNet and the proposed method are compared in terms of external parameter calibration on the KITTIOdometry dataset. Figure 6Figure 1 is a schematic diagram of the visualization results of the two methods. Figure (a) is a projection comparison diagram of the lidar point cloud under the true value external parameters, Figure (b) is a projection diagram of the lidar point cloud under the LCCNet calibration external parameters, and Figure (c) is a schematic diagram of the projection of the lidar point cloud under the calibration external parameters of the method of the present invention. It can be seen from the figure that compared with LCCNet, the method proposed in the present invention can more accurately project the laser point cloud onto the camera image. From the enlarged area in the figure, it can be seen that the lidar point cloud on the utility pole, tree trunk, and car is better matched to the camera image under the projection of the six-degree-of-freedom external parameters predicted by the method of the present invention, which is closer to the result of the true value external parameter projection in Figure (a) than LCCNet. Therefore, it is more intuitive to verify that the method of the present invention has better performance.

[0126] In summary, the method of the present invention introduces the Transformer to tightly couple the data of the lidar and camera, reduces the network size through the Token pyramid module, and uses the self-attention mechanism to focus on the global spatial context of the lidar and camera, fully considering their complementarity; in addition, the method of the present invention also proposes to construct a multi-constraint loss function, adding depth projection constraints and semantic edge constraints on the basis of the six-degree-of-freedom extrinsic global constraints, further improving the calibration accuracy, making the algorithm more generalizable and applicable to various scenarios.

[0127] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. Transformer-based feature-level camera-lidar online calibration method, characterized by , including the following steps: S1, coarse calibration: Use the hand-eye estimator to perform coarse calibration on the camera and lidar. The original lidar point cloud is projected onto the image using the estimated initial six-degree-of-freedom extrinsic parameters to obtain the coarsely calibrated matching lidar point cloud and camera image. S2, building a multi-constraint Transformer deep network model: input the matched lidar point cloud and camera image after rough calibration in step S1 into the model, and output a more accurate six-degree-of-freedom external parameter after prediction; the multi-constraint Transformer deep network model includes a Token pyramid module, a Transformer fuser module, a feature injection module and a multi-constraint module, wherein, The Token Pyramid module takes the coarsely calibrated matched LiDAR point cloud and camera image as input, generates a Token pyramid, and obtains local features; The Transformer Fusion Module is used for multimodal fusion. It takes the local features output by the Token Pyramid Module as input and uses its self-attention mechanism to combine the global context of the camera image and the LiDAR point cloud to provide global features. The feature injection module injects the global features output by the Transformer fuser module into local tokens of corresponding scales for feature enhancement; The multi-constraint module: weights the six-degree-of-freedom external parameter global constraint, depth projection constraint, and semantic edge constraint to construct a multi-constraint loss function for network model training; S3, iterative optimization: Feed the six-degree-of-freedom external parameters predicted in step S2 back to step S1, re-project the lidar point cloud onto the image to obtain a higher matching lidar point cloud and camera image, repeat step S2, and iteratively optimize to obtain the final accurate six-degree-of-freedom external parameters.

2. The Transformer-based feature-level camera-lidar online calibration method according to claim 1, characterized in that: The six-degree-of-freedom external parameters in step S1 include rotational external parameters and translation of external references Among them, C and L represent the camera and lidar coordinate systems respectively, and the rotation extrinsic parameter Specifically: in, Represents the rotation matrix of the lidar and camera from time i to time i+1, Minimize function, n and m represent the starting frame and the final frame respectively; Translation of external parameters The linear least squares method is used to solve the following equation: in, They represent the translation matrices of the lidar and camera from time i to time i+1, and s represents the scale factor.

3. The Transformer-based feature-level camera-lidar online calibration method according to claim 1, characterized in that: The Token Pyramid module in the multi-constrained Transformer deep network model in step S2 is composed of stacked MobileNet blocks. Camera images or lidar point clouds are input into the MobileNet blocks to generate local tokens of different scales. After the local tokens are average-pooled to a uniform size, they are concatenated along the channel dimension to generate new tokens and obtain local features.

4. The Transformer-based feature-level camera-lidar online calibration method according to claim 3, characterized in that: In the Transformer fusion module of the multi-constrained Transformer deep network model in step S2, the local features processed by the Token pyramid module are considered to be the tokens {x1, x i …, x n }, a total of n tokens, through Queries Keys and Values To express: Q i =W Q ·x i K i =W K ·x i V i =W V ·x i Among them, W Q ,W K and W V Represents Q i , K i 、V i The weight matrix of the i-th token x i and the jth tokenx j The local attention weight ω between ij As shown in the following formula: Among them, f represents the dimension of the word vector, Q i The transpose of K j represents the keys of the jth token, and softmax represents the activation function; Self-attention mechanism i The output of the jth token can be expressed as ValuesV j The weighted sum of Through nonlinear transformation, the output of each Transformer block is obtained. After passing through several Transformer blocks, the Transformer fusion module obtains the global features.

5. The Transformer-based feature-level camera-lidar online calibration method according to claim 4, characterized in that: In step S2, the feature injection module of the multi-constrained Transformer deep network model takes the local tokens of the lidar point cloud and camera image generated by the Token Pyramid module and the global features generated by the Transformer Fusion module as input, performs a weighted summation of the local tokens of each dimension and the corresponding global features, and obtains the local-to-global information output: Where k represents the kth layer of the Token pyramid; N represents the number of layers of the Token pyramid; δ k It represents the feature weights generated by sending the global features of the kth layer into the 1×1 convolution layer and then undergoing normalization and sigmoid layer; l k g represents the features to be injected after the local tokens of the kth layer are input into the 1×1 convolution layer and normalized; k It means that the global features of the kth layer are processed and normalized by a 1×1 convolution layer, and then upsampled according to the dimensions corresponding to the local tokens.

6. The Transformer-based feature-level camera-lidar online calibration method according to claim 1, wherein: In the multi-constrained Transformer deep network model of step S2, after obtaining the global features through the Transformer fusion module, it is sent to the pose regressor composed of 2 MLPs, and decoded to obtain the translation extrinsic parameter t containing 1×3 pre and the 1×4 rotation extrinsic quaternion q pre , the six-degree-of-freedom external parameter global constraint loss function L global for: L global =L translation +L rotation =||t pre -t gt ||2+||q pre -q gt ||2 Among them, L translation and L rotation represents the loss function of translation and rotation extrinsic parameters, t gt and q gt is the true value of the translation extrinsic parameter and the rotation extrinsic parameter, and ||·||2 represents the two-norm; The depth projection loss function L depth The calculation is as follows: Where M represents the number of laser point clouds, It represents the calibration result of the original point cloud of the LiDAR after deep network prediction and rough calibration. Represents the true value point cloud obtained by projecting the true value extrinsic parameter; The semantic edge constraint loss function L seg : Among them, Z represents the number of semantic edge points in the set, F edge represents the set of semantic edge points, s i L represents semantic edges in the point cloud, Represents the corresponding image semantic edge in the image frame, K represents the camera intrinsic parameter matrix, R pre is the rotation extrinsic matrix of the predicted six-degree-of-freedom extrinsic parameters.

7. The Transformer-based feature-level camera-lidar online calibration method according to claim 6, characterized in that: The multi-constraint loss function L in the multi-constraint module final Specifically: L final =λ1L global +λ2L depth +λ3L seg Among them, λ1, λ2, λ3 are manually set weights.

Images