A multi-modal sensor self-supervised alignment method and system without physical calibration

Abstract

The application provides a multi-modal sensor self-supervised alignment method and system without physical calibration, the multi-modal sensor self-supervised alignment method comprising: acquiring multi-dimensional data and vehicle dynamics data in real time; generating a kinematics pseudo-label according to the vehicle dynamics data based on an integral kinematics model; performing cross-modal feature fusion on the multi-dimensional data based on a cross-modal alignment network and outputting an external parameter transformation matrix; training the cross-modal alignment network by using the kinematics pseudo-label and the external parameter transformation matrix based on a self-supervised loss function; and performing online update on the external parameter transformation matrix by using an EMA mechanism on a vehicle side and performing model iteration and OTA update by using differential privacy federated learning on a cloud side. The application solves the technical problem that the prior art cannot realize self-supervised calibration and online adaptive update only by relying on CAN bus signals without calibration objects and external true values.

Description

Technical Field

[0001] This invention relates to the field of calibration technology for autonomous driving perception systems, and more specifically, to a self-supervised alignment method and system for multimodal sensors that do not require physical calibration. Background Technology

[0002] In multi-sensor fusion perception for autonomous driving, the calibration of extrinsic parameters of LiDAR, cameras, and millimeter-wave radar is a prerequisite for environmental perception. The accuracy of extrinsic parameters directly determines the reliability of tasks such as 3D target detection, BEV perception, and path planning. Existing calibration solutions are mainly divided into two categories: traditional manual calibration and data-driven self-supervised calibration. Neither of these can meet the requirements of automation, full-scenario coverage, low cost, and long-term stability for automotive-grade mass production. Traditional calibration requires physical calibration objects such as checkerboards and calibration boards, as well as manual on-site operation. The process is cumbersome, costly, and time-consuming. It can only be completed offline and cannot be adapted to mass production and large-scale deployment.

[0003] While existing self-supervised calibration methods eliminate some calibration objects, they generally rely on GPS, RTK, and high-precision IMU to provide external ground truth trajectories. This makes them completely ineffective in GNSS-denied areas such as tunnels, underground parking garages, and urban areas where tall buildings obstruct the view, resulting in poor scenario applicability. At the same time, all existing solutions treat external parameters as static fixed values, which cannot compensate for dynamic offsets caused by vehicle vibration, temperature drift, minor collisions, and assembly stress release during vehicle operation, thus compromising long-term operational accuracy. Furthermore, most existing solutions have complex models, high computational consumption, do not support real-time inference on the vehicle side, and lack online updates, self-checking warnings, and cloud iteration mechanisms, making it impossible to automatically repair calibration failures.

[0004] The most prominent technical problem is that existing methods cannot achieve end-to-end self-supervised calibration and online adaptive updates by relying solely on the vehicle's CAN bus signal and without depending on any external truth values ​​or physical calibration objects. At the same time, it is difficult to balance dynamic robustness, automotive-grade deployment efficiency, data privacy compliance, and full lifecycle maintenance capabilities, which seriously restricts the mass production and deployment of advanced autonomous driving perception systems. Summary of the Invention

[0005] This invention solves the technical problem that existing technologies cannot achieve self-supervised calibration and online adaptive updating solely based on CAN bus signals, without calibration objects or external truth values.

[0006] To address the aforementioned issues, this invention provides a self-supervised alignment method for multimodal sensors that does not require physical calibration. The method includes: real-time acquisition of multi-dimensional data and vehicle dynamics data; generation of kinematic pseudo-labels based on an integral kinematics model and the vehicle dynamics data; cross-modal feature fusion of the multi-dimensional data using a cross-modal alignment network and outputting an extrinsic parameter transformation matrix; training the cross-modal alignment network using a self-supervised loss function, leveraging the kinematic pseudo-labels and the extrinsic parameter transformation matrix; online updating of the extrinsic parameter transformation matrix at the vehicle-mounted end using an EMA mechanism, and model iteration and OTA updates using differential privacy federated learning in the cloud. The multi-dimensional data includes LiDAR point cloud data, camera image data, and Radar ranging data; the vehicle dynamics data includes wheel speed, steering angle, and IMU yaw rate.

[0007] Compared with existing technologies, the technical effects achieved by this solution are as follows: calibration can be completed using only raw LiDAR, Camera, and Radar data and CAN bus dynamic signals, without any manual calibration objects; pure self-supervised learning is achieved based on kinematic pseudo-labels, without relying on external ground truth values ​​such as GPS / IMU; online updates of the vehicle EMA and cloud-based differential privacy federated learning and OTA upgrades are supported; it is applicable to all road scenarios, including tunnels, underground parking garages, and other areas without GNSS; the two-level collaborative learning mechanism of continuous online learning on the vehicle and cloud-based differential privacy federated model updates enables dynamic adaptive compensation of the extrinsic parameter transformation matrix and iterative upgrades throughout the model's entire lifecycle without uploading raw sensor data, relying on external ground truth values, or interrupting vehicle operation.

[0008] In one embodiment of the present invention, the method of performing cross-modal feature fusion on multi-dimensional data based on the cross-modal alignment network and outputting the extrinsic transformation matrix includes: extracting point cloud features using a point cloud encoder and extracting image features using an encoder based on the multi-dimensional data; inputting the point cloud features and image features into the cross-modal fusion module for cross-modal fusion to obtain fused features; inputting the fused features into the extrinsic prediction layer and outputting the extrinsic transformation matrix that conforms to the SE(3) manifold constraint.

[0009] Compared with existing technologies, the technical effects achieved by this technical solution are as follows: a dedicated encoder is used to extract point cloud features and image features respectively, thereby improving the feature expression capability; cross-modal fusion module is used to achieve intermodal information alignment, suppressing noise and dynamic target interference; and the output extrinsic transformation matrix strictly satisfies the SE(3) manifold constraint to ensure physical realizability.

[0010] In one embodiment of the present invention, the fused features are input to the extrinsic parameter prediction layer, and the extrinsic parameter transformation matrix that meets the SE(3) manifold constraint is output. This includes: mapping the fused features to a 6-dimensional Lie algebra vector through global pooling and two fully connected layers; decomposing the 6-dimensional Lie algebra vector into a 3-dimensional translation component and a 3-dimensional rotation component; performing a modulo operation on the 3-dimensional rotation component to obtain the rotation angle θ, and calculating an orthogonal rotation matrix based on the antisymmetric matrix and the Rodrigues formula; constructing a G matrix based on the rotation angle θ and the antisymmetric matrix, and transforming the 3-dimensional translation component through the G matrix to obtain a translation vector; and combining the rotation matrix and the translation vector to form an extrinsic parameter transformation matrix that meets the SE(3) manifold constraint.

[0011] Compared with existing technologies, the technical effects achieved by this technical solution are as follows: the fused features are mapped to 6-dimensional Lie algebra vectors, which are simplified in dimension and easy to learn; a strictly orthogonal rotation matrix is ​​generated based on the antisymmetric matrix and the Rodrigues formula; a reasonable translation vector is obtained through G matrix transformation to ensure that the extrinsic matrix is ​​physically legal; and a standard 4×4SE(3) extrinsic matrix is ​​output, which can be directly used for point cloud projection and perception fusion.

[0012] In one embodiment of the present invention, a self-supervised loss function is constructed with a triple loss function as its core; the triple loss function includes temporal consistency loss, kinematic consistency loss and geometric consistency loss; the self-supervised loss function uses an uncertainty-based adaptive weight learning mechanism to complete network training and optimization of the extrinsic parameter transformation matrix.

[0013] Compared with existing technologies, the technical effects achieved by this solution are as follows: temporal consistency loss ensures smooth, non-jumping extrinsic parameter transformations between consecutive frames, conforming to motion continuity; kinematic consistency loss ensures that predicted extrinsic parameters conform to the actual motion laws of the vehicle and are physically consistent with the CAN bus signals; geometric consistency loss ensures that the LiDAR point cloud is accurately projected onto the camera image after extrinsic parameter transformation; temporal consistency loss ensures temporal smoothness, kinematic consistency loss ensures physical reliability, and geometric consistency loss ensures spatial accuracy. These three factors are dynamically weighted through an adaptive weight learning mechanism to jointly constitute a complete self-supervised loss function, achieving end-to-end multimodal sensor alignment without calibration objects or external truth values.

[0014] In one embodiment of the present invention, the kinematic consistency loss satisfies: ; ; ;in, For the rotator loss, These are the weighting coefficients. For yaw rate loss; For the predicted rotation matrix, The linear velocity vector measured on the CAN bus. The linear velocity vector measured by LiDAR. For the predicted yaw rate, Yaw rate measured by the IMU.

[0015] Compared with existing technologies, the technical effects achieved by adopting this technical solution are as follows: introducing dual verification of rotation constraints and yaw rate constraints to make external parameters conform to the vehicle kinematics law; forming a closed-loop supervision between CAN signals and LiDAR observations, without relying on any external truth values; and significantly improving the accuracy of calibration results in motion scenarios such as steering, acceleration and deceleration.

[0016] In one embodiment of the present invention, the geometric consistency loss satisfies: ;in, To sum over all matching points, For the projection function of the pinhole camera, Let be the extrinsic transformation matrix. Let i be the coordinates of the i-th point in the LiDAR coordinate system. Let be the projection of the i-th point in the camera coordinate system.

[0017] Compared with existing technologies, the technical effects achieved by this solution are as follows: using point cloud reprojection error as the core monitoring signal to directly optimize spatial alignment accuracy; constraining the point cloud projection to strictly match the image pixels, improving the intuitiveness and effectiveness of calibration; and directly improving the accuracy of downstream tasks such as 3D detection and BEV segmentation.

[0018] In one embodiment of the present invention, kinematic pseudo-labels are generated based on vehicle dynamics data using an integral kinematics model, including: integrating wheel speed to obtain displacement and displacement change, integrating IMU yaw rate to obtain yaw angle and yaw angle change; constructing pseudo-label transformation matrix using the integral kinematics model based on displacement change, yaw angle change, and yaw angle; and using the pseudo-label transformation matrix as kinematic pseudo-labels.

[0019] Compared with existing technologies, the technical effects achieved by this technical solution are as follows: kinematic pseudo-labels can be generated by integrating wheel speed and yaw rate; the SE(3) pseudo-label transformation matrix is ​​constructed, which has physical consistency and feasibility; the entire process is generated offline, without relying on the network or ground truth devices.

[0020] In one embodiment of the present invention, real-time acquisition of multi-dimensional data and vehicle dynamics data includes: acquiring multi-dimensional data in real time through a multi-modal sensor; and synchronously acquiring vehicle dynamics data through a CAN bus.

[0021] Compared with existing technologies, the technical effects achieved by adopting this technical solution are as follows: dynamic data is acquired synchronously through the CAN bus, ensuring timing consistency; multimodal data and vehicle motion data timestamps are aligned, improving calibration stability; and it is suitable for the standard hardware architecture of mass-produced vehicles without additional modifications.

[0022] In one embodiment of the present invention, after training the cross-modal alignment network based on the self-supervised loss function using kinematic pseudo-labels and extrinsic transformation matrices, the multimodal sensor self-supervised alignment method further includes: verifying the calibration quality using direct and indirect metrics; wherein, the direct metrics include: reprojection error and effective point cloud ratio; and the indirect metrics include: 3D detection mAP and BEV segmentation quality.

[0023] Compared with existing technologies, the technical effects achieved by this solution are as follows: direct indicators quickly determine whether external parameters are accurate, achieving millisecond-level online evaluation; indirect indicators verify effectiveness at the task level, avoiding the problem of "accurate calibration but poor perception"; the two-layer indicators form a closed loop, supporting automatic early warning of calibration failure, online recalibration, and OTA iteration; no human intervention is required, the entire process is automated, and it meets the long-term operation requirements of mass-produced vehicles.

[0024] On the other hand, embodiments of the present invention also provide a self-supervised alignment system for multimodal sensors that does not require physical calibration. This system employs the self-supervised alignment method for multimodal sensors that does not require physical calibration, as described in any embodiment of the present invention. The self-supervised alignment system for multimodal sensors includes: a data acquisition module for real-time acquisition of multi-dimensional data and vehicle dynamics data; a kinematic pseudo-label generation module for generating kinematic pseudo-labels based on an integral kinematics model and vehicle dynamics data; a cross-modal alignment module for performing cross-modal feature fusion on multi-dimensional data based on a cross-modal alignment network and outputting an extrinsic parameter transformation matrix; a self-supervised training module for training the cross-modal alignment network based on a self-supervised loss function, using kinematic pseudo-labels and the extrinsic parameter transformation matrix; and an online continuous learning module for online updating of the extrinsic parameter transformation matrix at the vehicle end using an EMA mechanism, and for model iteration and OTA updates at the cloud using differential privacy federated learning.

[0025] Compared with the prior art, the technical effects achieved by adopting this technical solution are as follows: The self-supervised alignment system for multimodal sensors without physical calibration in this embodiment is used to implement the self-supervised alignment method for multimodal sensors without physical calibration as in any embodiment of the present invention. Therefore, it has all the beneficial effects of the self-supervised alignment method for multimodal sensors without physical calibration as in any embodiment of the present invention, which will not be repeated here.

[0026] By adopting the technical solution of the present invention, the following technical effects can be achieved: (1) Calibration can be completed using only raw data from LiDAR, Camera, and Radar and dynamic signals from the CAN bus, without the need for any manual calibration materials; (2) Pure self-supervised learning is achieved based on kinematic pseudo-labels, without relying on external ground truth values ​​such as GPS / IMU; (3) Supports online updates of vehicle EMA and cloud-based differential privacy federated learning and OTA upgrades; (4) The two-level collaborative learning mechanism of vehicle-mounted online continuous learning + cloud-based differential privacy federated model update enables dynamic adaptive compensation of external parameter transformation matrix and iterative upgrade of model throughout its entire life cycle without uploading original sensor data, relying on external truth values, or interrupting vehicle operation. (5) Temporal consistency loss ensures temporal smoothness, kinematic consistency loss ensures physical reliability, and geometric consistency loss ensures spatial accuracy. The three are dynamically weighted through an adaptive weight learning mechanism to form a complete self-supervised loss function, achieving end-to-end multimodal sensor alignment without calibration objects or external truth values. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a flowchart of a self-supervised alignment method for multimodal sensors that does not require physical calibration, provided in Embodiment 1 of the present invention. Figure 2 This is a schematic block diagram of a self-supervised alignment system for multimodal sensors that does not require physical calibration, provided in Embodiment 2 of the present invention.

[0028] Explanation of reference numerals in the attached figures: 100 - Multimodal sensor self-supervised alignment system; 101 - Data acquisition module; 102 - Kinematic pseudo-label generation module; 103 - Cross-modal alignment module; 104 - Self-supervised training module; 105 - Online continuous learning module. Detailed Implementation

[0029] To make the above-mentioned objectives, features, and advantages of the present invention more apparent and understandable, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1 See Figure 1 This is a flowchart of a self-supervised alignment method for multimodal sensors without physical calibration provided in the first embodiment of the present invention. The self-supervised alignment method for multimodal sensors includes: S100: Real-time acquisition of multi-dimensional data and vehicle dynamics data; S200: Based on an integral kinematics model, kinematic pseudo-labels are generated from vehicle dynamics data; S300: Based on a cross-modal alignment network, it performs cross-modal feature fusion on multi-dimensional data and outputs an extrinsic parameter transformation matrix; S400: Based on a self-supervised loss function, a cross-modal alignment network is trained using kinematic pseudo-labels and extrinsic transformation matrices; S500: The EMA mechanism is used on the vehicle to update the external parameter transformation matrix online, and differential privacy federated learning is used in the cloud for model iteration and OTA updates; The multi-dimensional data includes: LiDAR point cloud data, camera image data, and Radar ranging data; the vehicle dynamics data includes: wheel speed, steering angle, and IMU yaw rate.

[0031] In a specific embodiment, the specific steps of the multimodal sensor self-supervised alignment method include: multimodal data acquisition and preprocessing, kinematic constraint pseudo-label generation, cross-modal feature fusion and extrinsic parameter prediction, self-supervised training, online update and cloud iteration, and dual index verification. The core idea of ​​this scheme is: taking the vehicle kinematics realizability as a rigid prior constraint, using only low-cost CAN bus signals such as wheel speed, steering angle, and IMU yaw rate, generating physically consistent SE(3) pseudo-labels (i.e., kinematic pseudo-labels in the text) through kinematic integration, constructing a time-geometric dual-constraint self-supervised loss function, driving the neural network to regress the sensor extrinsic parameters end-to-end, and realizing fully automatic self-supervised calibration without external truth values, calibration objects, or human intervention.

[0032] Finally, the trained cross-modal alignment network is deployed on the vehicle edge computing platform to perform online forward inference on the real-time input sensor data (i.e., multi-dimensional data), outputting the current optimal LiDAR-camera extrinsic parameter calibration result (i.e., the current optimal extrinsic parameter transformation matrix). An exponential moving average (EMA) is used to smooth and update the extrinsic parameters across multiple consecutive frames. Differential privacy federated learning and OTA updates are implemented: the vehicle-mounted device only uploads model gradients / model parameters to the cloud, without uploading raw privacy data such as images and point clouds; the cloud uses differential privacy federated learning to securely aggregate local models uploaded from multiple vehicles to generate a globally optimal model; the cloud remotely distributes the optimized model to all vehicles via OTA, completing a unified upgrade. This application performs adaptive loss weight adjustment on the vehicle edge device, combined with differential privacy cloud-based federated model updates, supporting differential update distribution and OTA remote correction of neural network weights.

[0033] Furthermore, the S300 includes: S310: Based on multi-dimensional data, a point cloud encoder is used to extract point cloud features, and an encoder is used to extract image features; S320: Input point cloud features and image features into the cross-modal fusion module for cross-modal fusion to obtain fused features; S330: Input the fused features into the extrinsic prediction layer and output the extrinsic transformation matrix that conforms to the SE(3) manifold constraint.

[0034] Specifically, the cross-modal alignment network includes: a point cloud encoder, an image encoder, a cross-modal fusion module, and an extrinsic parameter prediction layer.

[0035] For LiDAR point cloud data, a point cloud encoder is used for feature extraction: the encoder consists of three SetAbstraction layers. The first layer takes N×4 point cloud data (xyz coordinates with added intensity) as input, samples 1024 center points, and outputs 1024×64-dimensional features; the second layer samples 256 center points and outputs 256×128-dimensional features; the third layer samples 64 center points and outputs 64×256-dimensional features; finally, global pooling is used to obtain 256-dimensional global features.

[0036] For camera image data, an image encoder is used for feature extraction: the encoder contains four residual layers and finally outputs a B×512×H / 32×W / 32-dimensional feature map, which is then downsampled by 8 times.

[0037] For multimodal feature fusion, a cross-modal fusion module is designed: This module uses camera features as the query and LiDAR features as the key and value, performs weighted fusion through a gating mechanism, and outputs 64-dimensional fused features.

[0038] Finally, the fused features are input into the extrinsic prediction layer, and the extrinsic transformation matrix that conforms to the SE(3) manifold constraint is output.

[0039] Preferably, the point cloud encoder uses a PointNet++ encoder with a PointNet++ structure to perform multi-level abstraction on the input LiDAR point cloud data (including xyz coordinates and reflection intensity) and extract semantically rich global and local point cloud features; the image encoder uses a ResNet-18 encoder with a ResNet-18 structure to extract features from the input camera image data and obtain high-dimensional image features; the cross-modal fusion module uses a GatedCross-Attention fusion module, which introduces a GatedCross-Attention mechanism, using image features as the query and point cloud features as the key and value, and associating corresponding regions of the two modalities through an attention mechanism. At the same time, a gating unit is introduced to dynamically control the amount of information flowing from the point cloud modality to the fused features, reducing interference from low-quality or dynamic point clouds.

[0040] Furthermore, the S330 includes: S331: The fused features are mapped to a 6-dimensional Lie algebra vector through global pooling and two fully connected layers; S332: Decompose a 6-dimensional Lie algebra vector into a 3-dimensional translation component and a 3-dimensional rotation component; S333: Perform modulo operation on the 3D rotation components to obtain the rotation angle θ, and calculate the orthogonal rotation matrix based on the antisymmetric matrix and the Rodrigues formula; S334: Construct the G matrix based on the rotation angle θ and the antisymmetric matrix, and use the G matrix to transform the 3D translation components to obtain the translation vector; S335: Combine the rotation matrix and translation vector into an extrinsic transformation matrix that satisfies the manifold constraint of SE(3).

[0041] Specifically, the detailed steps of S330 are as follows: ①. Global pooling and two fully connected layers mapping: The high-dimensional fusion features output by the cross-modal fusion module are globally pooled and compressed into a fixed-dimensional feature vector. Then, the dimensionality is transformed through two fully connected layers, finally outputting a 6-dimensional Lie algebra vector. This 6-dimensional Lie algebra vector is a Lie algebra representation, where the first 3 dimensions represent translation components and the last 3 dimensions represent rotation components. It has the characteristics of being differentiable, easy to learn, and without redundancy; ②. 6-dimensional vector decomposition: The 6-dimensional Lie algebra vector is decomposed into: the first 3 dimensions translation components and the 3D rotation component These are used respectively for subsequent calculations of translation vectors and rotation matrices; ③. Calculation of rotation angle and antisymmetric matrix: for rotation components The rotation angle θ is obtained by taking the modulus: Constructing an antisymmetric matrix based on 3D rotational components ④. Calculate the orthogonal rotation matrix based on the Rodrigues formula: The 3D rotation components are mapped to a 3×3 orthogonal rotation matrix R using Rodrigues, satisfying: R×Rᵀ=I (orthogonality), det(R)=1 (right-handed system, no reflection) to ensure that the output rotation matrix is ​​physically valid and has no singular orientation; ⑤. Construct the G matrix and calculate the translation vector: Based on the rotation angle θ and the antisymmetric matrix Construct the G matrix and convert the original translation components The translation component is mapped to the final 3×1 translation vector by the G matrix, so that the translation component and the rotation component maintain geometric consistency on the SE(3) manifold; ⑥. Combine to obtain the SE(3) extrinsic parameter matrix: Combine the rotation matrix R and the translation vector into a 4×4 homogeneous extrinsic parameter transformation matrix T (i.e., the extrinsic parameter transformation matrix in this scheme). This matrix strictly satisfies the SE(3) manifold constraint and is the standard extrinsic parameter from LiDAR to Camera. It can be directly used for point cloud projection, reprojection error calculation and perception fusion, ensuring that the predicted extrinsic parameter is always located on the SE(3) manifold and satisfies the physical constraint of rigid body transformation. The G matrix is ​​used to transform the first 3 dimensions of the translation component in the 6-dimensional Lie algebra vector. The key matrix is ​​mapped to the translation vector, ensuring that the rotation and translation are geometrically consistent and physically self-consistent on the SE(3) manifold.

[0042] Preferably, the extrinsic parameter prediction layer is a SoftSVD prediction layer. A SoftSVD layer is designed. Since the predicted rotation matrix must satisfy the manifold constraints of orthogonality and positive determinant, this layer does not directly output 9 rotation matrix elements, but outputs a 6-dimensional Lie algebra vector (3-dimensional translation + 3-dimensional rotation). Then, the Rodrigues formula is used to accurately map it onto the SE(3) manifold, which ensures the mathematical rigor of the output and facilitates subsequent gradient backpropagation.

[0043] Furthermore, a self-supervised loss function is constructed with a triple loss function as its core; the triple loss function includes temporal consistency loss, kinematic consistency loss, and geometric consistency loss; the self-supervised loss function uses an uncertainty-based adaptive weight learning mechanism to complete network training and optimization of the extrinsic parameter transformation matrix.

[0044] Specifically, the temporal consistency loss constrains the continuity of motion by calculating the Frobenius norm difference between transformation matrices at adjacent time points, avoiding instantaneous infinite acceleration and penalizing drastic jumps in the transformation matrix, thus conforming to the physical laws of vehicle motion. The kinematic consistency loss comprises two parts: rotational constraint and yaw rate constraint. It creatively correlates the CAN signal with the prediction result. The rotational constraint ensures that the CAN speed after the predicted rotational matrix transformation is consistent with the speed measured by LiDAR, and the yaw rate constraint ensures that the predicted yaw rate is consistent with the yaw rate measured by the IMU sensor. The yaw rate loss... Defined as a scalar with a value greater than or equal to zero; the geometric consistency loss is based on a pinhole camera model, which projects LiDAR point clouds onto an image plane and calculates the reprojection error.

[0045] The specific steps for implementing the adaptive weight learning mechanism are as follows: ①. Assign learnable uncertainty parameters to the three losses respectively. , , ②. Through exponential functions Automatically generated adaptive weights: higher confidence level → smaller σ → larger weight; lower confidence level → larger σ → smaller weight. ③. Construct the self-supervised loss function: (The weight coefficients for the i-th task / sample are provided.) ,in, For learnable parameters, Let be the variance of the i-th task / sample. For the exponentially decaying weighting function, the first term For data items, the second item For regularization terms, ④. Continuously update with training to achieve adaptive weight adjustment: automatically increase the kinematic loss weight in high-speed scenarios, automatically increase the geometric loss weight in urban congestion scenarios, and automatically increase the temporal consistency loss weight in tunnels without GNSS scenarios.

[0046] The adaptive weight learning mechanism enables cross-modal alignment networks to automatically learn and assign weights for temporal consistency loss, kinematic consistency loss, and geometric consistency loss during training, without the need for manual pre-setting. This allows the model to maintain optimal calibration accuracy in complex scenarios such as high speeds, congestion, tunnels, and urban areas, solving problems such as uneven constraint strength of the triple loss function in different scenarios, difficulty in manual parameter tuning, and unstable training.

[0047] Furthermore, the kinematic consistency loss satisfies: ; ; ;in, For the rotator loss, These are the weighting coefficients. The loss is due to the yaw rate. For the predicted rotation matrix, The linear velocity vector measured on the CAN bus. The linear velocity vector measured by LiDAR. For the predicted yaw rate, Yaw rate measured by the IMU.

[0048] Specifically, the kinematic consistency loss includes rotation constraint loss and yaw rate constraint loss. The rotation constraint loss ensures that the predicted rotation matrix... The converted CAN speed is consistent with the speed measured by LiDAR: , ,in, The linear velocity vector measured by LiDAR. Here is the linear velocity vector measured on the CAN bus, and × represents matrix-vector multiplication. For the rotational loss; the yaw rate constraint ensures that the predicted yaw rate is consistent with the yaw rate measured by the IMU sensor: , ,in, Let f(·) be the predicted yaw rate, and f(·) be the yaw rate calculation function. Let Δt be the change in yaw angle at time k, and Δt be the time interval. Yaw rate measured by the IMU This is the loss due to yaw rate.

[0049] Total kinematic loss is ,in, For loss of kinematic consistency, The weighting coefficients are used; the kinematic consistency loss is used to calibrate the velocity transformation of LiDAR and CAN sensors, verify the consistency between IMU yaw rate measurement and vehicle motion model, and optimize the rotation matrix. and / or model parameters.

[0050] Furthermore, the geometric consistency loss satisfies: ;in, To sum over all matching points, For the projection function of the pinhole camera, The extrinsic transformation matrix, Let i be the coordinates of the i-th point in the LiDAR coordinate system. Let be the projection of the i-th point in the camera coordinate system.

[0051] Specifically, the geometric consistency loss is based on a pinhole camera model, projecting LiDAR point clouds onto the image plane and calculating the reprojection error: the projection of the point cloud onto the image is... , , Where u and v are image pixel coordinates, and X, Y, and Z are point cloud world coordinates (LiDAR coordinate system). For the camera's focal length (x / y direction), Here, π(·) is the principal point / image center, and π(·) is the projection function (3D→2D). Let them be homogeneous coordinate column vectors. The coordinates of the projected image are given; the reprojection error is... By minimizing reprojection error To calibrate the extrinsic parameter transformation between LiDAR and camera ,in, For geometric reprojection loss, To sum over all matching points, This is the transformation matrix (i.e., the extrinsic parameter transformation matrix) of the LiDAR to camera transformation. Let i be the coordinates of the i-th point in the LiDAR coordinate system. Let π(·) be the projection of the i-th point in the camera coordinate system, where π(·) is the pinhole camera projection function.

[0052] Preferably, the time consistency loss satisfies: , Where F represents the Frobenius norm, For time consistency loss, , Let k and k-1 represent the extrinsic transformation matrices, respectively. External parameter transformation matrix The element in the i-th row and j-th column, where i and j are matrix indices (e.g., i,j=1,2,3,4). To sum over all time steps, The summation is performed on all matrix elements; the temporal consistency loss maintains smooth temporal changes by calculating the squared difference of the Frobenius norm of the extrinsic transformation matrix between two adjacent frames, penalizes drastic jumps in the extrinsic transformation matrix, and constrains the continuity of motion (avoiding instantaneous infinite acceleration). It is often used as a regularization term in pose optimization.

[0053] Furthermore, the S200 includes: S210: Integrate the wheel speed to obtain the displacement and displacement change, and integrate the IMU yaw rate to obtain the yaw angle and yaw angle change; S220: Based on the displacement change, yaw angle change, and yaw angle, construct the pseudo-label transformation matrix using an integral kinematics model; S230: Use the pseudo-label transformation matrix as the kinematic pseudo-label.

[0054] Specifically, based on the vehicle's integral kinematic model, kinematic pseudo-labels for sensor extrinsic parameters are generated using CAN bus signals. Specifically: first, the vehicle displacement is obtained through the integral wheel speed signal. , representing the displacement up to time , is equal to the integral of the velocity over time from time 0 to the present. Let be the displacement at time t. For a moment instantaneous speed, Let be the dummy variable for integration (representing the time variable at intermediate moments), and t be the current moment (the upper limit of integration). Let t be the definite integral from 0 to t. To The derivative; the vehicle yaw angle is obtained by integrating the IMU yaw rate signal. This represents the cumulative IMU yaw rate over time. Let yaw angle be the yaw angle at time t. For a moment Yaw rate (z-axis angular velocity).

[0055] In the case of discrete sampling, the change in displacement can be approximated by the product of the average velocity and the time step: , Let k be the displacement change at step k. , These are the displacements at times k and k+1, respectively. , The velocities at times k and k+1 are respectively. For time step The change in yaw angle can be approximately calculated by multiplying the average yaw rate by the time step: , Δψ k Let yaw angle be the change in the k-th step. , Let be the yaw angles at times k and k+1, respectively. , These are the yaw rates at times k and k+1, respectively.

[0056] Based on the displacement and yaw angle changes calculated above, the SE(3) relative transformation matrix between adjacent frames is constructed: The matrix The transformation matrix represents the robot's motion transformation from time k to k+1, and is a homogeneous transformation matrix at time k. It is used for pose update, LiDAR odometry, and state prediction in inertial navigation. Through chained transformations, a sequence of extrinsic parameter transformation matrices between the LiDAR and the camera is derived, serving as a kinematic pseudo-label for self-supervised learning. This kinematic pseudo-label describes the vehicle's own trajectory within an extremely short time, indirectly constraining the geometric relationship of relative attitude changes between sensors. Let yaw angle be the cumulative yaw angle at time k.

[0057] Furthermore, the S100 includes: acquiring multi-dimensional data in real time through multi-modal sensors; and simultaneously acquiring vehicle dynamics data through the CAN bus.

[0058] Specifically, the multimodal sensors include: LiDAR (point cloud), Camera (image), and Radar (range measurement) three-modal sensors; LiDAR is used to scan the 3D point cloud of the surrounding environment, Camera is used to capture high-definition RGB images, and Radar is used to measure the distance and speed of the target; CAN bus is used to read vehicle dynamics data such as wheel speed, steering angle, and IMU yaw angle; the multimodal data is timestamped with the vehicle dynamics data to improve calibration stability.

[0059] Due to the different sampling frequencies of LiDAR, Camera, and Radar, there is a time-asynchronous problem. For the acquired asynchronous sensor data, a linear time interpolation method is used to achieve data synchronization: for point cloud data, a linear interpolation method is used to calculate the predicted position of the midpoint in time; for image data, the nearest frame or a linear interpolation method is used for alignment, unifying all sensor data to the same timestamp and eliminating asynchronous deviation.

[0060] Furthermore, after step S400, the multimodal sensor self-supervised alignment method also includes: verifying the calibration quality using direct and indirect indicators; wherein, the direct indicators include: reprojection error and effective point cloud ratio; and the indirect indicators include: 3D detection mAP and BEV segmentation quality.

[0061] Specifically, a two-layer indicator system is used to verify the calibration quality. Direct indicators include reprojection error and effective point cloud ratio. Direct indicators directly reflect the calibration accuracy. Reprojection error directly reflects the spatial alignment accuracy between LiDAR and Camera. The smaller the error, the more accurate the calibration. Effective point cloud ratio characterizes the rationality of the extrinsic parameter matrix and the point cloud utilization rate. The higher the ratio, the more complete the alignment. Indirect indicators include performance evaluation of downstream tasks such as 3D detection mAP and BEV segmentation quality. Indirect indicators verify the effectiveness at the task level and support failure warning. 3D detection mAP verifies the improvement of system performance by calibration at the perception task level. BEV segmentation quality verifies the practical engineering value of calibration in autonomous driving panoramic perception.

[0062]

Example 2

[0063] In one specific embodiment, the data acquisition module 101, kinematic pseudo-label generation module 102, cross-modal alignment module 103, self-supervised training module 104, and online continuous learning module 105 of the self-supervised alignment system 100 for multimodal sensors without physical calibration work together to implement the self-supervised alignment method for multimodal sensors without physical calibration as described in the first embodiment above, which will not be elaborated here.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised alignment method for multimodal sensors without physical calibration, characterized in that, The multimodal sensor self-supervised alignment method includes: Real-time acquisition of multi-dimensional data and vehicle dynamics data; Based on the integral kinematics model, kinematic pseudo-labels are generated according to the vehicle dynamics data; The multi-dimensional data is fused using a cross-modal alignment network, and an extrinsic parameter transformation matrix is ​​output. The cross-modal alignment network is trained using the self-supervised loss function, the kinematic pseudo-labels, and the extrinsic transformation matrix. The EMA mechanism is used to update the extrinsic transformation matrix online at the vehicle end, and differential privacy federated learning is used for model iteration and OTA updates in the cloud. The multi-dimensional data includes: LiDAR point cloud data, camera image data, and Radar ranging data; the vehicle dynamics data includes: wheel speed, steering angle, and IMU yaw rate.

2. The self-supervised alignment method for multimodal sensors according to claim 1, characterized in that, The method of fusing cross-modal features of the multi-dimensional data based on a cross-modal alignment network and outputting an extrinsic transformation matrix includes: Based on the multi-dimensional data, point cloud features are extracted using a point cloud encoder, and image features are extracted using an encoder. The point cloud features and the image features are input into the cross-modal fusion module for cross-modal fusion to obtain fused features; The fused features are input into the extrinsic parameter prediction layer, and the extrinsic parameter transformation matrix that conforms to the SE(3) manifold constraint is output.

3. The self-supervised alignment method for multimodal sensors according to claim 2, characterized in that, The step of inputting the fused features into the extrinsic parameter prediction layer and outputting the extrinsic parameter transformation matrix that conforms to the SE(3) manifold constraint includes: The fused features are mapped to a 6-dimensional Lie algebra vector through global pooling and two fully connected layers; The 6-dimensional Lie algebra vector is decomposed into a 3-dimensional translation component and a 3-dimensional rotation component; The rotation angle θ is obtained by performing a modulo operation on the 3D rotation components, and the orthogonal rotation matrix is ​​calculated based on the antisymmetric matrix and the Rodrigues formula. A G matrix is ​​constructed based on the rotation angle θ and the antisymmetric matrix, and the translation vector is obtained by transforming the 3D translation components using the G matrix. The rotation matrix and the translation vector are combined into the extrinsic transformation matrix that satisfies the SE(3) manifold constraint.

4. The self-supervised alignment method for multimodal sensors according to claim 1, characterized in that, The self-supervised loss function is constructed with the triple loss function as its core. The triple loss function includes time consistency loss, kinematic consistency loss, and geometric consistency loss; The self-supervised loss function employs an uncertainty-based adaptive weight learning mechanism to complete network training and optimize the extrinsic parameter transformation matrix.

5. The self-supervised alignment method for multimodal sensors according to claim 4, characterized in that, The kinematic consistency loss satisfies: ; ； ； in, For the rotator loss, These are the weighting coefficients. For yaw rate loss; For the predicted rotation matrix, The linear velocity vector measured on the CAN bus. The linear velocity vector measured by LiDAR. For the predicted yaw rate, Yaw rate measured by the IMU.

6. The self-supervised alignment method for multimodal sensors according to claim 4, characterized in that, The geometric consistency loss satisfies: ; in, To sum over all matching points, For the projection function of the pinhole camera, Let be the extrinsic transformation matrix. Let i be the coordinates of the i-th point in the LiDAR coordinate system. Let be the projection of the i-th point in the camera coordinate system.

7. The self-supervised alignment method for multimodal sensors according to claim 1, characterized in that, The generation of kinematic pseudo-labels based on the vehicle dynamics data using the integral kinematics model includes: The displacement and displacement change are obtained by integrating the wheel speed, and the yaw rate of the IMU is obtained by integrating the yaw angle and yaw angle change. Based on the displacement change, the yaw angle change, and the yaw angle, a pseudo-label transformation matrix is ​​constructed using the integral kinematics model. The pseudo-label transformation matrix is ​​used as the kinematic pseudo-label.

8. The self-supervised alignment method for multimodal sensors according to claim 1, characterized in that, The real-time acquisition of multi-dimensional data and vehicle dynamics data includes: The multi-dimensional data is acquired in real time using a multimodal sensor; The vehicle dynamics data is acquired synchronously via the CAN bus.

9. The self-supervised alignment method for multimodal sensors according to claim 1, characterized in that, After training the cross-modal alignment network based on the self-supervised loss function using the kinematic pseudo-labels and the extrinsic transformation matrix, the multimodal sensor self-supervised alignment method further includes: The calibration quality was verified using both direct and indirect indicators. The direct metrics include reprojection error and effective point cloud ratio; the indirect metrics include 3D detection mAP and BEV segmentation quality.

10. A self-supervised alignment system for multimodal sensors that does not require physical calibration, characterized in that, The multimodal sensor self-supervised alignment system employs the self-supervised alignment method for multimodal sensors without physical calibration as described in any one of claims 1 to 9, and the multimodal sensor self-supervised alignment system comprises: The data acquisition module is used to acquire multi-dimensional data and vehicle dynamics data in real time; A kinematic pseudo-label generation module is used to generate kinematic pseudo-labels based on the vehicle dynamics data according to an integral kinematic model. A cross-modal alignment module is used to perform cross-modal feature fusion on the multi-dimensional data based on a cross-modal alignment network and output an extrinsic parameter transformation matrix; A self-supervised training module is used to train the cross-modal alignment network based on a self-supervised loss function, using the kinematic pseudo-labels and the extrinsic transformation matrix. The online continuous learning module is used to update the extrinsic parameter transformation matrix online at the vehicle end using the EMA mechanism, and to perform model iteration and OTA updates in the cloud using differential privacy federated learning.

Images