Multi-mode joint calibration method and system and vehicle

Abstract

The invention discloses a multi-modal joint calibration method and system and a vehicle, and belongs to the technical field of automatic driving multi-modal perception.The method comprises the steps that point cloud features in point cloud data are extracted through point cloud branches in a pre-trained double-branch feature extraction network model and converted into point cloud aerial view features, extracting image features in the image data by using image branches in the pre-trained double-branch feature extraction network model, and converting the image features into image aerial view features; performing feature fusion on the point cloud aerial view features and the image aerial view features to obtain fused aerial view features; and encoding and decoding the fused aerial view features, and determining external parameters between the laser radar and the camera. According to the scheme, point cloud and image data are unified into aerial view features through double-branch feature extraction, and joint calibration of multiple cameras and different types of laser radars is supported through an end-to-end process of feature fusion and coding and decoding, so that calibration robustness and calibration precision in a complex scene can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of multimodal perception technology for autonomous driving, specifically to a multimodal joint calibration method, system, and vehicle. Background Technology

[0002] In autonomous driving multimodal perception systems, the collaborative work of LiDAR and cameras relies on high-precision sensor calibration. The accuracy of sensor calibration directly determines the accuracy of key functions of the system, such as obstacle detection and environmental perception.

[0003] Related multi-sensor calibration methods include offline and online calibration. Offline calibration requires the deployment of specialized targets and high-precision automated equipment on the vehicle production line, completing extrinsic parameter calibration through a fixed process. However, due to the monotonous factory calibration environment, the obtained offline parameters are difficult to adapt to complex and changing road scenarios. Furthermore, when sensors are replaced, involved in collisions, or experience parameter drift due to long-term use, recalibration at the factory is necessary, resulting in poor robustness of the calibration results. Online calibration methods primarily employ multi-stage calibration, first selecting point, line, or surface features within a specific environment, and then constructing an optimization function to solve for the extrinsic parameters. This type of approach relies on environmental information and suffers from error accumulation, exhibiting poor robustness in complex scenarios such as those with severe occlusion. Summary of the Invention

[0004] This paper provides a multimodal joint calibration method, system, and vehicle, aiming to solve the problem of poor robustness of calibration results in multi-sensor calibration methods in related technologies.

[0005] Firstly, a multimodal joint calibration method is provided, including the following steps: Acquire point cloud data from the lidar and image data from at least one camera; The point cloud branch extracts point cloud features from the point cloud data and converts them into point cloud bird's-eye view features. The image branch extracts image features from the image data and converts them into image bird's-eye view features. The point cloud branch and the image branch are two branches in a pre-trained dual-branch feature extraction network model. The point cloud bird's-eye view features and the image bird's-eye view features are fused to obtain the fused bird's-eye view features. The features of the fused bird's-eye view are encoded and decoded to determine the extrinsic parameters between the lidar and the camera. These extrinsic parameters are used to jointly calibrate the lidar and the camera.

[0006] In some embodiments, the step of extracting point cloud features from the point cloud data using point cloud branches and converting them into point cloud bird's-eye view features includes: The point cloud data is processed using a sparse convolutional network to generate three-dimensional voxel features; The three-dimensional voxel features are expanded along the height dimension to obtain the point cloud bird's-eye view features.

[0007] In some embodiments, the step of extracting image features from the image data using image branching and converting them into image bird's-eye view features includes: The image data is processed by a neural network with a Transformer architecture to extract image features. Using a preset initial extrinsic parameter matrix and the camera's intrinsic parameters, the image features are projected onto a world coordinate system defined by the coordinate system of the lidar to obtain the projected bird's-eye view features. The projected bird's-eye view features are pooled to obtain the image bird's-eye view features.

[0008] In some embodiments, the step of generating the initial extrinsic parameter matrix includes: Obtain the true extrinsic parameter matrix between the lidar and the camera; Generate a random error matrix, which includes random Euler angle errors and random displacement errors; The initial extrinsic matrix is ​​obtained by multiplying the true extrinsic parameter matrix with the inverse of the random error matrix.

[0009] In some embodiments, encoding and decoding the fused bird's-eye view features to determine the extrinsic parameters between the lidar and the camera includes: Multi-scale feature extraction and upsampling are performed on the fused bird's-eye view features to obtain enhanced bird's-eye view features; The enhanced bird's-eye view features are semantically parsed using a Transformer encoder to obtain semantic features; Based on the semantic features, the extrinsic parameters between the lidar and the camera are determined.

[0010] In some embodiments, the training steps of the dual-branch feature extraction network model include: Collect point cloud data samples from the lidar and image data samples from the camera; An initial point cloud branch is constructed based on a pre-defined convolutional network; an initial image branch is constructed based on a neural network with a Transformer architecture; and an initial dual-branch feature extraction network model is constructed based on the initial point cloud branch and the initial image branch. The initial point cloud branch is trained using the point cloud data samples to obtain the bird's-eye view features of the intermediate point cloud. The initial image branch is trained using the image data samples to obtain the bird's-eye view features of the intermediate image; The intermediate point cloud bird's-eye view features and the intermediate image bird's-eye view features are fused, encoded, and decoded to generate intermediate extrinsic parameters; The loss value corresponding to the intermediate extrinsic parameters is calculated using a preset loss function; Based on the loss value, the initial bi-branch feature extraction network model is iteratively optimized to obtain a pre-trained bi-branch feature extraction network model.

[0011] In some embodiments, the loss function includes one or more of attitude error, displacement error, and point cloud distance error; Wherein, the attitude error represents the difference between the predicted quaternion in the intermediate extrinsic parameters and the true quaternion in the true extrinsic parameters; The displacement error represents the difference between the predicted displacement vector in the intermediate extrinsic parameters and the true displacement vector in the true extrinsic parameters. The point cloud distance error represents the distance deviation between the point cloud after conversion by the intermediate extrinsic parameters and the original point cloud data acquired by the lidar.

[0012] In some embodiments, after acquiring the point cloud data from the lidar and image data from at least one camera, the process includes: The point cloud data and the image data are preprocessed to update the point cloud data and the image data. The preprocessing includes at least one of downsampling the point cloud data, downsampling and distortion correction of the image data, and aligning the point cloud data and the image data with timestamps.

[0013] Secondly, a multimodal joint calibration system is also provided, including: The data acquisition module is used to acquire point cloud data from the LiDAR and image data from at least one camera. The dual-branch feature extraction module is used to extract point cloud features from the point cloud data using the point cloud branch and convert them into point cloud bird's-eye view features, and to extract image features from the image data using the image branch and convert them into image bird's-eye view features. The point cloud branch and the image branch are two branches in a pre-trained dual-branch feature extraction network model. The feature fusion module is used to fuse the point cloud bird's-eye view features and the image bird's-eye view features to obtain the fused bird's-eye view features. The calibration module is used to encode and decode the features of the fused bird's-eye view to determine the extrinsic parameters between the lidar and the camera. The extrinsic parameters are used to jointly calibrate the lidar and the camera.

[0014] Thirdly, a vehicle is also provided, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the steps of any of the methods described above.

[0015] Fourthly, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being loaded by a processor to perform the steps of any of the methods described above.

[0016] Beneficial effects: This application ensures the temporal consistency and quality of multimodal data through preprocessing of data from multiple types of cameras and LiDAR. By using bi-branch feature extraction, point cloud and image data are unified into bird's-eye view features, breaking down the spatial dimensional barriers between different modal data and taking into account the advantages of the three-dimensional structure of point clouds and the semantic details of images. Through an end-to-end process of feature fusion and encoding / decoding, external parameters can be directly output without relying on specific environmental features. This supports joint calibration of multiple cameras and different types of LiDAR and effectively improves the calibration robustness and accuracy in complex scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a simplified flowchart of the multimodal joint calibration method provided by an exemplary embodiment of this disclosure; Figure 2 This is a schematic diagram of the overall process of the multimodal joint calibration method provided by an exemplary embodiment of this disclosure; Figure 3 This is a schematic diagram of the functional modules of the multimodal joint calibration system provided in an exemplary embodiment of this disclosure; Figure 4 This is a schematic diagram of a vehicle structure provided by an exemplary embodiment of this disclosure. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0022] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0023] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0024] On the one hand, this embodiment provides a multimodal joint calibration method applied to autonomous vehicles, wherein the autonomous vehicles are equipped with an autonomous driving multimodal perception system, such as... Figure 1 As shown, it includes the following steps: Step S100: Acquire point cloud data from the LiDAR and image data from at least one camera.

[0025] Specifically, the autonomous driving multimodal perception system of the autonomous vehicle acquires point cloud data output from LiDAR and image data captured by at least one camera. The camera includes one or more of pinhole cameras, fisheye cameras, and telephoto cameras, and multiple types of cameras can also be used in combination; the LiDAR can be a 360° mechanical LiDAR or a fixed-field solid-state LiDAR, etc. This embodiment does not impose specific limitations on the types of LiDAR and cameras.

[0026] Furthermore, after acquiring the point cloud data from the lidar and the image data from at least one camera, the process includes: preprocessing the point cloud data and the image data, updating the point cloud data and the image data, and the preprocessing includes at least one of downsampling the point cloud data, downsampling and distortion correction of the image data, and aligning the point cloud data and the image data with timestamps, in order to optimize data quality and ensure temporal consistency. For example, for point cloud data, voxel sampling is used to divide the three-dimensional space into a regular voxel grid. The point cloud within each voxel is aggregated, and a single representative point replaces the original multiple points, achieving data compression and noise filtering. For image data, OpenCV functions are used to downsample the original-sized image to obtain a size suitable for the subsequent network model input. At the same time, distortion correction is performed on the image to eliminate distortion of edge pixels and improve image fidelity. To address the temporal differences between point cloud and image data, a caching mechanism is used. Based on the timestamp of the LiDAR data, the difference between the timestamp of each frame of point cloud and the timestamp of each camera image is calculated. The data with the smallest difference is selected for alignment, and redundant data is deleted to ensure that the multimodal data matches in the time dimension. The preprocessed point cloud data and image data are then updated.

[0027] Step S200: Extract point cloud features from point cloud data using the point cloud branch and convert them into point cloud bird's-eye view features; extract image features from image data using the image branch and convert them into image bird's-eye view features. The point cloud branch and the image branch are two branches in the pre-trained dual-branch feature extraction network model.

[0028] Specifically, a pre-trained dual-branch feature extraction network model is constructed and trained in advance. This model includes a point cloud branch and an image branch, which are used to process the pre-processed point cloud data and the pre-processed image data through the point cloud branch and the image branch, respectively.

[0029] For example, for the point cloud branch, a sparse convolutional neural network (SCNN) is used as the network structure to extract features from the downsampled point cloud data, generating three-dimensional voxel features. These three-dimensional voxel features are then unfolded along the height dimension to transform them into two-dimensional point cloud bird's-eye view (BEV) features. These features can accurately map the coordinate relationships of the point cloud in physical space. For the image branch, a Transformer-based neural network is used as the backbone network to extract two-dimensional features from the image. Then, the image frustum point cloud features are constructed through the Lift Splat Shoot (LSS) module. Combined with the pre-generated initial extrinsic matrix and camera intrinsic matrix, the image frustum point cloud features are transformed into a world coordinate system based on the LiDAR coordinate system. Finally, through bird's-eye view pooling, the image frustum point cloud features in the world coordinate system are transformed into image bird's-eye view features, achieving spatial alignment between image features and point cloud features.

[0030] Step S300: Perform feature fusion on the point cloud bird's-eye view features and the image bird's-eye view features to obtain the fused bird's-eye view features.

[0031] Specifically, a convolutional layer is used to fuse point cloud bird's-eye view features and image bird's-eye view features. This convolutional layer can integrate and unify the channel information of the two modalities without changing the feature space dimension. During the fusion process, the advantages of point cloud bird's-eye view features in spatial distance and three-dimensional structure are preserved, while the characteristics of image bird's-eye view features in semantic information and texture details are also preserved. Deep fusion of the two modalities is achieved through convolutional operations, and the final output is a fused bird's-eye view feature with unified dimension and complementary information.

[0032] Step S400: Encode and decode the features of the fused bird's-eye view to determine the extrinsic parameters between the LiDAR and the camera. The extrinsic parameters are used to jointly calibrate the LiDAR and the camera.

[0033] Specifically, the fused bird's-eye view features are first encoded. A Sparsely Embedded Convolutional Detection (SECOND) network is used to extract multi-scale bird's-eye view features. Then, Feature Pyramid Networks (FPN) are used to upsample, align, and concatenate the features at each scale. Afterward, convolutional layers are used for fusion and dimensionality reduction to obtain enhanced bird's-eye view features containing richer multi-scale semantic information. Next, the enhanced bird's-eye view features are decoded using a multi-level Transformer encoder to obtain structured semantic features. These features are then flattened, transforming the multi-dimensional features into a one-dimensional vector. Finally, two fully connected linear layers are used to perform parameter regression on the one-dimensional vector. One fully connected layer outputs a quaternion representing the relative rotation relationship between the LiDAR and the camera, while the other fully connected layer outputs a translation vector representing their relative displacement relationship. These two parameters together constitute the extrinsic parameters between the LiDAR and the camera, enabling joint calibration of the LiDAR and camera.

[0034] In this embodiment, the temporal consistency and quality of multimodal data are ensured through preprocessing of data from multiple types of cameras and LiDAR. By using bi-branch feature extraction, point cloud and image data are unified into bird's-eye view features, breaking down the spatial dimensional barriers between different modal data and taking into account the advantages of the three-dimensional structure of point clouds and the semantic details of images. Through the end-to-end process of feature fusion and encoding / decoding, external parameters can be directly output without relying on specific environmental features. This not only supports the joint calibration of multiple cameras and different types of LiDAR, but also effectively improves the calibration robustness and calibration accuracy in complex scenarios, while avoiding the cumbersome process of returning to the factory for calibration. This can significantly improve the calibration efficiency and ease of use of the autonomous driving multimodal perception system.

[0035] The following exemplary example details the working principle of the multimodal joint calibration method proposed in this application. It takes lidar point cloud data and 3D image data from multiple cameras as input, and outputs the relative extrinsic parameters between the lidar and each camera. Figure 2 As shown, firstly, the point cloud data and 3D image data are preprocessed, including downsampling of the point cloud data and downsampling, distortion correction, and timestamp alignment of the image data. Then, a dual-branch architecture model is constructed to process the input data of different modalities; the image branch uses an image encoder to extract 2D image features. And using initial extrinsic parameters Bird's-eye view features generated from the camera intrinsic parameter matrix K Point cloud branches use a point cloud encoder to extract downsampled point cloud features. And convert it into point cloud bird's-eye view features. Subsequently, and After encoding by the bird's-eye view encoder, the fused bird's-eye view features are obtained. and using a self-attention decoder to Perform decoding to obtain the decoded features. Finally, linear network layers are used to regress the extrinsic parameters between the camera and the LiDAR. .

[0036] During network training and inference, for the camera Using the true extrinsic parameters between the lidar and the camera and random error matrix Generate initial matrix ,in Includes random Euler angle errors and random displacement error The network's output maps to external disturbances. The correction capability, that is, after network processing, the true extrinsic parameters between the LiDAR and the camera are ,in, This is the network's predicted value.

[0037] Based on the above examples, in some embodiments, step S200, which involves extracting point cloud features from the point cloud data using point cloud branches and converting them into point cloud bird's-eye view features, includes: Step S210: Process point cloud data using a sparse convolutional network to generate three-dimensional voxel features.

[0038] Specifically, the point cloud branch in this embodiment includes a point cloud encoder, which uses a sparse convolutional network as its architecture to adapt to the three-dimensional characteristics of the point cloud data. First, the preprocessed and downsampled point cloud data is input into the sparse convolutional network. The sparse convolutional network, designed for the sparse distribution of point cloud data, performs convolution operations only on regions with valid points, avoiding invalid computations on blank areas, thus improving processing efficiency while preserving key spatial information. Through multiple rounds of sparse convolution operations, the three-dimensional spatial structure features in the point cloud data are gradually extracted, including the relative positional relationships between points and density distribution features, generating three-dimensional voxel features that accurately characterize the three-dimensional spatial attributes of the point cloud. ,in, Indicates the batch size of the input during training or testing; , , Represents the spatial voxel coordinates, Indicates the number of features corresponding to the coordinates, subscript Lidar is an abbreviation for radar. This three-dimensional voxel feature is presented in the form of a voxel grid, and each voxel grid contains point cloud feature information of the corresponding spatial region.

[0039] Step S220: Unfold the three-dimensional voxel features along the height dimension to obtain the point cloud bird's-eye view features.

[0040] Specifically, to adapt to subsequent fusion with the bird's-eye view features of the image, after obtaining the 3D voxel features, the 3D voxel features are transformed into 2D point cloud bird's-eye view features. Specifically, using the LiDAR coordinate system as the world coordinate system, the spatial distribution of the 3D voxel features on the horizontal plane (XY plane) remains unchanged; only the 3D voxel features are expanded along the height dimension (Z-axis). By sequentially stitching and integrating voxel features at different height levels in a preset order, the voxel features in 3D space are mapped onto the 2D plane, ultimately forming the 2D point cloud bird's-eye view features. This point cloud bird's-eye view feature retains the spatial location information and structural features of the point cloud on the horizontal plane, while also achieving dimensional simplification, enabling it to be used for fusion operations with image bird's-eye view features in the same dimension.

[0041] In this embodiment, the three-dimensional voxel features of the point cloud are extracted by a sparse convolutional network and transformed into point cloud bird's-eye view features. This not only adapts to the sparse characteristics of point cloud data, but also fully preserves its three-dimensional spatial structure information while efficiently compressing data and filtering out noise. Furthermore, by unfolding along the height dimension, the three-dimensional features are transformed into a two-dimensional bird's-eye view form, which can achieve dimensional consistency with the bird's-eye view features of the image. This also helps to improve the efficiency and adaptability of point cloud feature processing.

[0042] In some embodiments, step S200, which involves extracting image features from image data using image branching and converting them into bird's-eye view features, includes: Step S230: Use a neural network with a Transformer architecture to extract features from the image data to obtain image features.

[0043] Specifically, the image branch in this embodiment includes an image encoder, which uses a Transformer-based neural network as its backbone to adapt to the feature extraction requirements of multi-camera image data. Preprocessed, downsampled, and distortion-corrected multi-camera 3D image data is input into this neural network. Through a hierarchical window shifting mechanism and attention calculation, the network fully captures local texture details, global semantic information, and inter-pixel relationships in the image, resulting in image features that can effectively extract and accurately represent the image content. ,in Indicates the number of feature channels.

[0044] Step S240: Using the preset initial extrinsic parameter matrix and the camera's intrinsic parameters, project the image features onto the world coordinate system defined by the coordinate system of the LiDAR to obtain the projected bird's-eye view features.

[0045] Specifically, after obtaining the image features, they are first constructed into image frustum point cloud features using the LSS module. The image's frustum cloud features can simulate the visual extension range of an image in three-dimensional space. Among them, B This indicates the batch size input during training or testing. Indicates the number of cameras. Indicates the number of feature channels. Indicates the feature map height. Indicates the width of the feature map. The preset spatial distance representing the view cone is: , in, and These represent the shortest and longest distances, respectively. s The step size.

[0046] Then, the preset initial extrinsic parameter matrix is ​​called. and the camera's intrinsic parameter matrix K The initial extrinsic parameter matrix is ​​generated based on the true extrinsic parameters between the LiDAR and the camera, and a random error matrix containing random Euler angle errors and random displacement errors. This matrix is ​​used to establish the spatial relationship between image features and the LiDAR coordinate system. Using these two matrices, coordinate transformation operations are performed to project the constructed view frustum feature from the camera coordinate system to a world coordinate system based on the LiDAR coordinate system, resulting in the view frustum feature in world coordinates. This achieves alignment of image features and point cloud features in the same spatial dimension, resulting in a projected bird's-eye view feature.

[0047] Step S250: Perform pooling operation on the projected bird's-eye view features to obtain the image bird's-eye view features.

[0048] Specifically, for the bird's-eye view features projected onto the world coordinate system, a bird's-eye view pooling operation is performed, including: using the horizontal plane of the world coordinate system as a reference, dividing the projected features according to a preset spatial grid, aggregating the feature information within each grid, and eliminating redundant information and strengthening the representation of key spatial features by integrating the effective feature data within the grid, thereby generating the image bird's-eye view features in the world coordinate system. The bird's-eye view features of this image have the same dimension as the bird's-eye view features of the point cloud generated by the point cloud branch, so that subsequent fusion operations can be performed with the point cloud bird's-eye view features in the same dimension.

[0049] In this embodiment, image features are extracted using Transformer, combined with extrinsic parameters, projected to the world coordinate system, and generated as bird's-eye view features through bird's-eye view pooling. This not only fully captures the semantic details and global correlations of the image, but also aligns the image features with the point cloud features in the spatial dimension. At the same time, it adapts to the feature differences of multiple camera types, preserving the semantic advantages of the image while allowing the image features to be efficiently fused with the point cloud bird's-eye view features. This provides accurate and unified image feature support for the joint calibration of multiple cameras and LiDAR, thereby effectively improving the effectiveness of multimodal feature fusion and the generalization of calibration.

[0050] In some embodiments, the initial extrinsic parameter matrix generation step S240 includes: Step S241: Obtain the true extrinsic parameter matrix between the lidar and the camera.

[0051] Specifically, by deploying high-precision calibration equipment on the vehicle production line to calibrate the lidar and cameras offline, a true extrinsic parameter matrix containing real relative rotation information and real relative translation information is obtained, so as to accurately characterize the relative pose between the lidar and each camera.

[0052] Step S242: Generate a random error matrix, which contains random Euler angle errors and random displacement errors.

[0053] Specifically, a random error matrix is ​​generated to simulate initial deviations in real-world scenarios, such as sensor installation misalignment, collisions, or parameter drift after long-term use. This random error matrix includes random Euler angle errors and random displacement errors. The random Euler angle errors are converted into corresponding rotation matrices to simulate random deviations in the relative attitudes between sensors; the random displacement errors characterize random offsets in the relative positions between sensors. Furthermore, to enable the model to adapt to different levels of initial deviations, this embodiment sets the maximum random Euler angle error to ±20° and the maximum random displacement error to ±1.5m, ensuring that the generated error matrix covers common deviation ranges in practical applications.

[0054] Step S243: Calculate the product of the true extrinsic parameter matrix and the inverse of the random error matrix to obtain the initial extrinsic parameter matrix.

[0055] Specifically, firstly, the inverse of the random error matrix is ​​obtained, which is used to reasonably incorporate the preset random bias into the true extrinsic parameter matrix. Then, the product of the true extrinsic parameter matrix and the random error inverse matrix is ​​calculated. Matrix operations are then used to integrate the random bias into the true extrinsic parameter matrix, ultimately yielding the initial extrinsic parameter matrix, which simulates the initial bias in actual calibration. This initial extrinsic parameter matrix retains the spatial correlation logic inherent in the true extrinsic parameters while incorporating the initial bias from the simulated actual calibration scenario. It provides a biased initial transformation basis for the projection of image features from the camera coordinate system to the world coordinate system, helping the model learn to correct initial biases and accurately regress the true extrinsic parameters.

[0056] In this embodiment, the initial extrinsic parameter matrix is ​​generated by combining the true extrinsic parameters with the random error matrix. This ensures the spatial correlation logic of the extrinsic parameters based on the accurate true values ​​of the offline calibration of the production line. Furthermore, by introducing random deviations, it simulates the installation errors and parameter drift of sensors in real-world scenarios. This allows the image feature projection to adapt to the biased initial conditions, which helps the model learn to correct the initial deviations and effectively improves the adaptability of the dual-branch network to real-world calibration scenarios.

[0057] In some embodiments, step 400 involves encoding and decoding the fused bird's-eye view features to determine the extrinsic parameters between the LiDAR and the camera, including: Step S410: Perform multi-scale feature extraction and upsampling on the fused bird's-eye view features to obtain enhanced bird's-eye view features.

[0058] Specifically, regarding the features of the fused bird's-eye view First, multi-scale feature extraction is performed using the SECOND network to effectively capture feature information at different spatial scales and generate feature maps of multiple dimensions. ,in, The number of feature maps is used to comprehensively cover the details and global semantics of the bird's-eye view features. These multi-scale features are then input into the FPN network. First, upsampling is performed on features at each scale to ensure all feature maps have the same spatial size. Next, channel alignment is used to unify the channel dimensions of features at different scales to the same spatial size, facilitating subsequent fusion. Then, the aligned multi-scale features are concatenated to integrate feature information from each level. Finally, a 1x1 convolutional layer is used to fuse and reduce the dimensionality of the concatenated features, removing redundant information and enhancing key feature representations, ultimately resulting in a more information-rich and enhanced bird's-eye view feature. .

[0059] Step S420: Use a Transformer encoder to perform semantic parsing on the enhanced bird's-eye view features to obtain semantic features.

[0060] Specifically, the enhanced bird's-eye view features are input into a multi-level Transformer encoder. The encoder's self-attention mechanism decodes these features, capturing the relationships between distant pixels in the feature map and deeply mining the global semantic information of the features. Simultaneously, a hierarchical encoding structure is used to progressively refine the feature representation, performing deep semantic parsing on the enhanced bird's-eye view features, filtering out invalid information, strengthening effective feature associations, and ultimately outputting structured semantic features. .

[0061] Step S430: Determine the extrinsic parameters between the lidar and the camera based on semantic features.

[0062] Specifically, the semantic features are first flattened, reducing the dimension to [missing information]. The feature matrix is ​​transformed into ( The flattened one-dimensional vector is then adapted to fit the input requirements of the linear fully connected layers. This flattened vector is then input into two independent linear fully connected layers. One of these layers is responsible for regressing the relative displacement relationship between the LiDAR and the camera, outputting a displacement vector representing that displacement. Another fully connected layer is used to regress the learning of the relative rotation relationship between the two layers, outputting a quaternion vector representing that rotation. By combining displacement vector and quaternion vector information, the extrinsic parameters containing relative rotation and relative displacement information between the lidar and each camera are determined.

[0063] In this embodiment, the multi-scale feature extraction and enhancement of the fused bird's-eye view features are achieved through the collaborative encoding process of the SECOND network and the FPN network. The global semantic association of the enhanced bird's-eye view features is deeply mined through the decoding process of the Transformer encoder. Finally, the displacement and quaternion vectors are accurately regressed through the fully connected layer. This not only fully integrates the advantages of point cloud and image multimodal features, effectively improving the feature representation ability and robustness, but also achieves end-to-end extrinsic parameter output, effectively avoiding the problem of error accumulation. At the same time, it adapts to the joint calibration requirements of multiple cameras and LiDAR, thereby significantly improving the regression accuracy and calibration efficiency of extrinsic parameters. In this way, it can provide accurate and reliable sensor pose reference for autonomous driving multimodal perception systems.

[0064] In some embodiments, the training step of the dual-branch feature extraction network model in step S200 includes: Step M100: Collect point cloud data samples from the lidar and image data samples from the camera.

[0065] Specifically, the training requires the collection of sample data, including point cloud data samples output by the LiDAR and image data samples captured by at least one camera. For example, image data samples from all cameras with a uniform resolution can be collected, where the resolution can be 704×256 or other resolutions. Simultaneously, the effective range of the point cloud data samples is set, for example, a horizontal range of ±100m and a vertical range of ±3m, to ensure that the sample data meets the input requirements for subsequent model training. Furthermore, high-precision extrinsic parameters obtained from offline calibration on the vehicle production line are used as ground truth extrinsic parameters.

[0066] Step M200: Based on the preset convolutional network, construct the initial point cloud branch; based on the neural network of the Transformer architecture, construct the initial image branch; and based on the initial point cloud branch and the initial image branch, construct the initial dual-branch feature extraction network model.

[0067] Specifically, an initial dual-branch feature extraction network model is constructed, which includes an independent but collaborative initial point cloud branch and an initial image branch. For the initial point cloud branch, a point cloud encoder is built based on a sparse convolutional network to process 3D point cloud data, adapting to the sparse distribution characteristics of point clouds to achieve efficient extraction of spatial features. For the initial image branch, an image encoder is built based on a Transformer architecture neural network (such as a Swing Transformer architecture neural network) and integrates an LSS module to process 2D image data from multiple cameras, effectively capturing local details and global semantic relationships of the image. The two initial branches are combined to form a complete initial dual-branch feature extraction network model.

[0068] Step M300: Train the initial point cloud branch using point cloud data samples to obtain the bird's-eye view features of the intermediate point cloud.

[0069] Specifically, the collected point cloud data samples are first preprocessed. A voxel sampling method is used to divide the 3D space into a regular voxel grid. Points within each voxel are aggregated, and a single representative point replaces the original multiple points, achieving data compression and noise filtering. Then, the preprocessed point cloud data samples are input into the initial point cloud branch. Multiple rounds of feature extraction are performed through a sparse convolutional network within the branch to generate 3D voxel features that characterize the 3D spatial structure of the point cloud. These 3D voxel features are then expanded along the height dimension, transforming them into 2D intermediate point cloud bird's-eye view features, completing the first round of training data processing and feature output for the initial point cloud branch.

[0070] Step M400: Train the initial image branch using image data samples to obtain the bird's-eye view features of the intermediate image.

[0071] Specifically, the acquired image data samples are first preprocessed. Downsampling is performed using OpenCV functions to adapt to the model input size, and distortion correction is performed to eliminate pixel distortion at image edges. Then, a caching mechanism is used to align the timestamps of the image data and point cloud data based on the LiDAR data timestamps. Considering that the typical acquisition frequency of LiDAR is approximately 10Hz and that of the camera is approximately 30Hz, this embodiment uses the LiDAR data timestamps as a reference to align the image data and point cloud data from each camera. Let the set of point cloud data cached at each time point be denoted as... The set of image data from the camera is denoted as ,in, , Indicates the time of data collection. Indicates the first Road camera, , This represents the total number of camera channels. The difference between the point cloud timestamp and the camera image timestamp for each frame is calculated by iterating through the data. Data alignment is achieved by selecting the data with the smallest timestamp difference, and redundant data is removed.

[0072] The preprocessed image data samples are input into the initial image branch. Two-dimensional image features are extracted through the Transformer backbone network. The image view frustum features are constructed through the LSS module. Combined with the camera intrinsic parameter matrix and the initial extrinsic parameter matrix generated by the ground truth extrinsic parameters and the random error matrix, the view frustum features are transformed into the world coordinate system based on the LiDAR coordinate system. Then, through the bird's-eye view pooling operation, two-dimensional intermediate image bird's-eye view features are generated, completing the first round of training data processing and feature output of the initial image branch.

[0073] Step M500: Fuse, encode, and decode the intermediate point cloud bird's-eye view features and the intermediate image bird's-eye view features to generate intermediate extrinsic parameters.

[0074] Specifically, convolutional layers are used to fuse the bird's-eye view features of the intermediate point cloud and the intermediate image, integrating the feature information of the two modalities without changing the spatial dimension to obtain fused bird's-eye view features. The fused features are then encoded: first, multi-scale features are extracted using a SECOND network, then upsampling, channel alignment, and feature concatenation are performed using an FPN network, and finally, 1×1 convolutional fusion and dimensionality reduction are used to obtain enhanced bird's-eye view features. A multi-level Transformer encoder is then used to perform semantic parsing on the enhanced bird's-eye view features to obtain structured semantic features. After flattening the semantic features, they are regressed through two linear fully connected layers to generate intermediate extrinsic parameters containing displacement vectors and quaternion vectors.

[0075] Step M600: Calculate the loss value corresponding to the intermediate extrinsic parameters using a preset loss function.

[0076] Specifically, a pre-defined multi-dimensional loss function is used to calculate the loss values ​​corresponding to the intermediate extrinsic parameters. This loss function includes three parts: attitude error, displacement error, and point cloud distance error. The attitude error represents the difference between the predicted quaternions in the intermediate extrinsic parameters and the true quaternions in the ground truth extrinsic parameters, reflecting the prediction deviation of rotational relationships. The displacement error represents the difference between the predicted displacement vector in the intermediate extrinsic parameters calculated by the Smooth-L1 function and the true displacement vector in the ground truth extrinsic parameters, representing the prediction deviation of translational relationships. The point cloud distance error represents the distance deviation between the point cloud transformed by the intermediate extrinsic parameters and the original acquired LiDAR point cloud, reflecting the deviation of the overall application effect of the extrinsic parameters. The three errors are weighted and summed using weighting coefficients to obtain the final loss value, comprehensively evaluating the prediction accuracy of the intermediate extrinsic parameters.

[0077] For example, the final loss value is obtained by weighted summing of the attitude error, displacement error, and point cloud distance error, and its expression is: , in, Indicates the total number of camera channels; Indicates the first Road camera; , , These represent attitude deviation, displacement deviation, and point cloud distance deviation, respectively. , , These represent the weight coefficients of the corresponding loss function. The loss functions described below are as follows: For attitude error, since the model outputs attitude representation as quaternions, this example uses the predicted quaternions from the intermediate extrinsic parameters. Truth quaternions in truth extrinsic parameters The difference between them represents the attitude error. For normalized quaternions... , respectively and Let the real and imaginary parts of the quaternion be represented, then the attitude error can be defined as: .

[0078] For displacement error, the smoothed-L1 function is used to represent the predicted displacement vector in the intermediate extrinsic parameters. With the truth displacement vector in the truth extrinsic parameters The error between them, then the displacement error can be defined as: .

[0079] For point cloud distance error, assume the point cloud set in the lidar coordinate system is as follows: External parameters output via network Rotation yields a new set of point clouds. To calculate the distance between two point cloud sets, the point cloud distance error can be defined as: , in, This represents the number of points in the point cloud acquired at each time step; This represents the true value of the extrinsic parameters between the lidar and the camera; This represents the external parameters output by the network.

[0080] Step M700: Based on the loss value, iteratively optimize the initial bi-branch feature extraction network model to obtain a pre-trained bi-branch feature extraction network model.

[0081] Specifically, based on the loss value, an iterative optimization process is initiated using a pre-defined framework and optimizer. For example, the batch size is set to 32, the initial learning rate is 5e-5, and the learning rate is dynamically updated using a cosine annealing function. Training is then performed in a pre-built distributed image processing environment. In each iteration, the gradients of the network's parameters at each layer are calculated using the backpropagation algorithm based on the loss value. All trainable parameters of the initial point cloud branch and the initial image branch are updated according to the gradient descent direction to reduce the loss value. This training process is repeated until a pre-defined number of iterations (e.g., 500) or a pre-defined accuracy requirement is achieved, resulting in a performance-optimized pre-trained dual-branch feature extraction network model.

[0082] In this embodiment, by constructing and iteratively optimizing the complete training process of the dual-branch feature extraction network model, it is possible to achieve separate training and joint optimization of the point cloud branch and the image branch without relying on special targets or any high-precision measurement tools. This ensures the feature extraction capability of the dual-branch network for multimodal data. Furthermore, by introducing ground truth extrinsic participation random error matrices to construct training samples that fit the actual scene, and combining multidimensional loss functions and optimizers, the model achieves efficient iterative convergence. The resulting pre-trained dual-branch feature extraction network model has the adaptability to different types of LiDAR and multi-camera data, as well as strong initial bias correction capability and extrinsic parameter regression accuracy. This effectively improves the model's generalization and robustness, thereby providing high-precision and high-reliability online calibration support for autonomous driving multimodal perception systems, and helping to reduce calibration time and manpower costs.

[0083] On the other hand, this embodiment provides a multimodal joint calibration system, such as Figure 3 As shown, it includes: The data acquisition module is used to acquire point cloud data from the LiDAR and image data from at least one camera. The dual-branch feature extraction module is used to extract point cloud features from point cloud data using the point cloud branch and convert them into point cloud bird's-eye view features, and to extract image features from image data using the image branch and convert them into image bird's-eye view features. The point cloud branch and the image branch are two branches in the pre-trained dual-branch feature extraction network model. The feature fusion module is used to fuse the point cloud bird's-eye view features and the image bird's-eye view features to obtain the fused bird's-eye view features. The calibration module is used to encode and decode the features of the fused bird's-eye view and determine the extrinsic parameters between the LiDAR and the camera. The extrinsic parameters are used to perform joint calibration of the LiDAR and the camera.

[0084] This embodiment also provides a vehicle, such as Figure 4 As shown, it includes a memory and a processor. In a specific example, the memory stores a computer program, which, when executed by the processor, implements the method of any of the above embodiments.

[0085] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of any of the methods in the above embodiments.

[0086] In the embodiments of this application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] The above provides a detailed description of a multimodal joint calibration method, system, and vehicle provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal joint calibration method, characterized in that, Includes the following steps: Acquire point cloud data from the lidar and image data from at least one camera; The point cloud branch extracts point cloud features from the point cloud data and converts them into point cloud bird's-eye view features. The image branch extracts image features from the image data and converts them into image bird's-eye view features. The point cloud branch and the image branch are two branches in a pre-trained dual-branch feature extraction network model. The point cloud bird's-eye view features and the image bird's-eye view features are fused to obtain the fused bird's-eye view features. The features of the fused bird's-eye view are encoded and decoded to determine the extrinsic parameters between the lidar and the camera. These extrinsic parameters are used to jointly calibrate the lidar and the camera.

2. The multimodal joint calibration method according to claim 1, characterized in that, The step of extracting point cloud features from the point cloud data using point cloud branches and converting them into point cloud bird's-eye view features includes: The point cloud data is processed using a sparse convolutional network to generate three-dimensional voxel features; The three-dimensional voxel features are expanded along the height dimension to obtain the point cloud bird's-eye view features.

3. The multimodal joint calibration method according to claim 1, characterized in that, The step of extracting image features from the image data using image branching and converting them into bird's-eye view features includes: The image data is processed by a neural network with a Transformer architecture to extract image features. Using a preset initial extrinsic parameter matrix and the camera's intrinsic parameters, the image features are projected onto a world coordinate system defined by the coordinate system of the lidar to obtain the projected bird's-eye view features. The projected bird's-eye view features are pooled to obtain the image bird's-eye view features.

4. The multimodal joint calibration method according to claim 3, characterized in that, The steps for generating the initial extrinsic parameter matrix include: Obtain the true extrinsic parameter matrix between the lidar and the camera; Generate a random error matrix, which includes random Euler angle errors and random displacement errors; The initial extrinsic matrix is ​​obtained by multiplying the true extrinsic parameter matrix with the inverse of the random error matrix.

5. The multimodal joint calibration method according to claim 1, characterized in that, The process of encoding and decoding the fused bird's-eye view features to determine the extrinsic parameters between the lidar and the camera includes: Multi-scale feature extraction and upsampling are performed on the fused bird's-eye view features to obtain enhanced bird's-eye view features; The enhanced bird's-eye view features are semantically parsed using a Transformer encoder to obtain semantic features; Based on the semantic features, the extrinsic parameters between the lidar and the camera are determined.

6. The multimodal joint calibration method according to claim 4, characterized in that, The training steps of the dual-branch feature extraction network model include: Collect point cloud data samples from the lidar and image data samples from the camera; An initial point cloud branch is constructed based on a pre-defined convolutional network; an initial image branch is constructed based on a neural network with a Transformer architecture; and an initial dual-branch feature extraction network model is constructed based on the initial point cloud branch and the initial image branch. The initial point cloud branch is trained using the point cloud data samples to obtain the bird's-eye view features of the intermediate point cloud. The initial image branch is trained using the image data samples to obtain the bird's-eye view features of the intermediate image; The intermediate point cloud bird's-eye view features and the intermediate image bird's-eye view features are fused, encoded, and decoded to generate intermediate extrinsic parameters; The loss value corresponding to the intermediate extrinsic parameters is calculated using a preset loss function; Based on the loss value, the initial bi-branch feature extraction network model is iteratively optimized to obtain a pre-trained bi-branch feature extraction network model.

7. The multimodal joint calibration method according to claim 6, characterized in that, The loss function includes one or more of the following: attitude error, displacement error, and point cloud distance error; Wherein, the attitude error represents the difference between the predicted quaternion in the intermediate extrinsic parameters and the true quaternion in the true extrinsic parameters; The displacement error represents the difference between the predicted displacement vector in the intermediate extrinsic parameters and the true displacement vector in the true extrinsic parameters. The point cloud distance error represents the distance deviation between the point cloud after conversion by the intermediate extrinsic parameters and the original point cloud data acquired by the lidar.

8. The multimodal joint calibration method according to claim 1, characterized in that, After acquiring the point cloud data from the lidar and image data from at least one camera, the process includes: The point cloud data and the image data are preprocessed to update the point cloud data and the image data. The preprocessing includes at least one of downsampling the point cloud data, downsampling and distortion correction of the image data, and aligning the point cloud data and the image data with timestamps.

9. A multimodal joint calibration system, characterized in that, include: The data acquisition module is used to acquire point cloud data from the LiDAR and image data from at least one camera. The dual-branch feature extraction module is used to extract point cloud features from the point cloud data using the point cloud branch and convert them into point cloud bird's-eye view features, and to extract image features from the image data using the image branch and convert them into image bird's-eye view features. The point cloud branch and the image branch are two branches in a pre-trained dual-branch feature extraction network model. The feature fusion module is used to fuse the point cloud bird's-eye view features and the image bird's-eye view features to obtain the fused bird's-eye view features. The calibration module is used to encode and decode the features of the fused bird's-eye view to determine the extrinsic parameters between the lidar and the camera. The extrinsic parameters are used to jointly calibrate the lidar and the camera.

10. A vehicle, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-8.

Images