Calibration method, device and equipment of intelligent driving sensor and storage medium

Abstract

The application relates to a kind of intelligent driving sensor calibration method, device, equipment and storage medium, according to the multidimensional data obtained in advance, extract dynamic environment characteristics, based on dynamic environment characteristics, construct initial extrinsic parameter matrix, combine vehicle kinematics model and Lie group optimization algorithm, the initial extrinsic parameter matrix is iteratively updated, according to the temperature parameter and vibration parameter obtained in advance, the initial extrinsic parameter matrix after updating is compensated, obtain final extrinsic parameter matrix, based on millimeter wave radar detection data and the fusion perception result after calibration, cross validation is carried out, if the verification result does not satisfy the preset condition threshold, then trigger dynamic re-calibration. The above method solves the problem of low efficiency and poor adaptability of traditional calibration, and also improves the robustness and long-term stability of sensor calibration through dynamic compensation and online verification mechanism.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and specifically to a calibration method, apparatus, device, and storage medium for intelligent driving sensors. Background Technology

[0002] The intelligent driving system is a multi-sensor fusion perception system. After the vehicle equipped with this system rolls off the production line, each sensor needs to be calibrated to determine the position and attitude of each sensor in the vehicle coordinate system, so as to avoid perception and positioning errors caused by installation position deviations.

[0003] Existing technologies typically rely on fixed sensor calibration methods based on calibration boards, which depend on dedicated calibration sites and high-precision calibration patterns.

[0004] However, the above method is only applicable to calibration after the product is off the production line. It lacks flexibility and has significant limitations, resulting in low calibration efficiency and insufficient accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a calibration method, apparatus, device, and storage medium for driving sensors, which addresses the problem of decreased or insufficient accuracy of intelligent driving sensors in dynamic environments.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A calibration method for intelligent driving sensors, comprising:

[0008] Based on pre-acquired multi-dimensional data, dynamic environmental features are extracted. The multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters.

[0009] Based on the dynamic environment characteristics, an initial extrinsic parameter matrix is ​​constructed;

[0010] The initial extrinsic parameter matrix is ​​iteratively updated by combining the vehicle kinematics model and the Lie group optimization algorithm;

[0011] The updated initial extrinsic parameter matrix is ​​compensated based on the pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix.

[0012] In one possible implementation, the method further includes:

[0013] The timing deviation of different sensors is compensated by the IMU pre-integration algorithm;

[0014] Spatiotemporal alignment is performed based on a unified time reference to obtain the timestamp-aligned multi-dimensional data, wherein the unified time reference is a GNSS second pulse signal.

[0015] In one possible implementation, the dynamic environment features include LiDAR point cloud features and camera image features. The step of extracting dynamic environment features based on pre-acquired multi-dimensional data includes:

[0016] Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude calculation to obtain the corrected lidar point cloud data.

[0017] The corrected lidar point cloud data is segmented on the ground to obtain the lidar point cloud features.

[0018] Lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0019] In one possible implementation, constructing the initial extrinsic parameter matrix based on the dynamic environment characteristics includes:

[0020] Based on the lidar point cloud features and the camera image features, feature alignment is performed to obtain 3D-2D corresponding point pairs;

[0021] Based on the 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

[0022] In one possible implementation, the iterative update of the initial extrinsic parameter matrix, combining the vehicle kinematics model and the Lie group optimization algorithm, includes:

[0023] In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initialization extrinsic matrix by minimizing the pre-defined joint calibration objective function;

[0024] By combining the vehicle kinematics model with the wheel speed signals obtained during vehicle operation, an extended Kalman filter is used to correct the cumulative pose error.

[0025] In one possible implementation, the extrinsic parameter matrix includes a rotation matrix and a translation vector. The compensation of the updated initial extrinsic parameter matrix based on pre-acquired temperature and vibration parameters includes:

[0026] The translation vector is linearly adjusted based on the temperature parameter;

[0027] The rotation matrix is ​​corrected based on the vibration parameters.

[0028] In one possible implementation, the method further includes:

[0029] Cross-validation is performed based on millimeter-wave radar detection data and calibrated fusion sensing results;

[0030] If the verification result does not meet the preset threshold, dynamic recalibration is triggered.

[0031] A calibration device for intelligent driving sensors, comprising:

[0032] The extraction module is used to extract dynamic environmental features based on pre-acquired multi-dimensional data, including lidar point cloud data, camera image data, and IMU motion parameters.

[0033] The construction module is used to construct an initial extrinsic parameter matrix based on the dynamic environment characteristics;

[0034] The update module is used to iteratively update the initial extrinsic parameter matrix by combining the vehicle kinematics model and the Lie group optimization algorithm;

[0035] The first compensation module is used to compensate the updated initial extrinsic parameter matrix based on the pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix.

[0036] In one possible implementation, the device further includes:

[0037] The second compensation module is used to compensate for timing deviations of different sensors through the IMU pre-integration algorithm;

[0038] The acquisition module is used to perform spatiotemporal alignment based on a unified time reference and acquire the timestamp-aligned multi-dimensional data, wherein the unified time reference is a GNSS second pulse signal.

[0039] In one possible implementation, the extraction module specifically includes:

[0040] Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude calculation to obtain the corrected lidar point cloud data.

[0041] The corrected lidar point cloud data is segmented on the ground to obtain the lidar point cloud features.

[0042] Lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0043] In one possible implementation, the building module specifically includes:

[0044] Based on the lidar point cloud features and the camera image features, feature alignment is performed to obtain 3D-2D corresponding point pairs;

[0045] Based on the 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

[0046] In one possible implementation, the update module specifically includes:

[0047] In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initialization extrinsic matrix by minimizing the pre-defined joint calibration objective function;

[0048] By combining the vehicle kinematics model with the wheel speed signals obtained during vehicle operation, an extended Kalman filter is used to correct the cumulative pose error.

[0049] In one possible implementation, the first compensation module specifically includes:

[0050] The translation vector is linearly adjusted based on the temperature parameter;

[0051] The rotation matrix is ​​corrected based on the vibration parameters.

[0052] In one possible implementation, the device further includes:

[0053] The verification module is used to perform cross-validation based on millimeter-wave radar detection data and calibrated fusion sensing results;

[0054] The recalibration module is used to trigger dynamic recalibration if the verification result does not meet the preset threshold conditions.

[0055] An electronic device includes: a storage unit and a processing unit;

[0056] The storage unit stores computer-executed instructions;

[0057] The processing unit executes the computer execution instructions stored in the storage unit, causing the processing unit to perform the method described above.

[0058] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0059] A computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0060] The beneficial effects of this invention are:

[0061] By synchronizing LiDAR, camera, and IMU data during vehicle operation, geometric and semantic features (such as lane lines and traffic signs) in the road environment are extracted. The sensor extrinsic parameter matrix is ​​iteratively updated using a Lie group optimization algorithm and a vehicle kinematic model (Ackermann steering model). A dynamic compensation mechanism for temperature and vibration parameters is introduced. Finally, independent observation of static obstacles by millimeter-wave radar verifies the calibration accuracy and triggers recalibration. This solution overcomes the limitations of traditional fixed calibration, achieving high-precision sensor calibration and adjustment in non-fixed scenarios, improving calibration flexibility, and addressing issues such as insufficient calibration accuracy and consistency during trial production and after-sales maintenance for intelligent driving systems. Attached Figure Description

[0062] Figure 1 A schematic diagram of the calibration architecture for an intelligent driving system;

[0063] Figure 2 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 1 ;

[0064] Figure 3 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 2 ;

[0065] Figure 4 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 3 ;

[0066] Figure 5 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 4 ;

[0067] Figure 6 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 5 ;

[0068] Figure 7 A schematic diagram of the structure of the calibration device for intelligent driving sensors provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0070] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0071] Intelligent driving systems are multi-sensor fusion perception systems. After a vehicle equipped with this system rolls off the production line, each sensor needs to be calibrated to determine its position and orientation in the vehicle body coordinate system, thus avoiding perception and positioning deviations caused by installation position deviations. However, traditional calibration equipment is mostly fixed in structure, lacking flexibility, and in the following scenarios, temporary calibration methods are required, resulting in problems such as low efficiency, insufficient calibration accuracy, and insufficient consistency: (1) During the trial production of new factories and new models, the calibration room has not been completed, and vehicles that have rolled off the production line can only be calibrated temporarily; (2) After-sales 4S stores do not have complete calibration room equipment, and temporary calibration methods can only be used after replacing sensors; (3) After the sensors have been used for a long time, their accuracy decreases, and they need to be returned to the factory or 4S store for recalibration. Existing technologies based on fixed sensor calibration boards require dedicated calibration sites and high-precision calibration patterns, making them suitable for calibration after the sensor rolls off the production line. However, they suffer from the aforementioned lack of flexibility and are unsuitable for trial production and after-sales service at 4S stores. Alternatively, workshop-based collaborative calibration requires multiple vehicles and has low efficiency (time ≥ 1 hour), and it cannot solve the parameter drift problem caused by sensor aging. Another option is offline calibration triggered by a diagnostic tool, which is currently a common calibration method used in the after-sales service phase at 4S stores. However, this method lacks a dynamic environmental feedback mechanism, and the calibration results show significantly increased errors under complex road conditions.

[0072] To address the aforementioned issues, this application provides a calibration method, device, equipment, and storage medium for intelligent driving sensors. This enables high-precision sensor calibration and adjustment in non-fixed scenarios, improving calibration flexibility and resolving problems such as insufficient accuracy and consistency in intelligent driving calibration during trial production and after-sales maintenance. Specifically, existing fixed sensor calibration methods based on calibration boards require dedicated calibration sites and high-precision calibration patterns, suitable for calibration after the vehicle leaves the factory. However, this method lacks flexibility and is unsuitable for trial production and after-sales service at 4S stores. Alternatively, workshop-based collaborative calibration requires multiple vehicles and has low efficiency (≥1 hour), but it cannot solve the parameter drift problem caused by sensor aging. Offline calibration triggered by a diagnostic tool is a common method used in 4S store after-sales service, but this method lacks a dynamic environmental feedback mechanism, leading to significantly increased errors in calibration results under complex road conditions. Considering these issues, the inventors investigated whether high-precision sensor calibration and adjustment in non-fixed scenarios could be achieved by integrating natural road feature extraction, Lie group optimization algorithms, and online dynamic compensation mechanisms, thereby improving calibration flexibility. Based on this, this solution is proposed.

[0073] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0074] Figure 1 A schematic diagram of the architecture for a smart driving system, such as... Figure 1 As shown, the calibration system hardware mainly includes: lidar, millimeter-wave radar, camera, inertial navigation module, time synchronization controller, computing unit, etc.; the software modules mainly include: dynamic calibration engine, visualization interface, etc.

[0075] LiDAR is one of the sensor units in an intelligent driving system. By using the point cloud data it scans, road condition information within the perception field of view can be realized, ensuring safe driving of intelligent vehicles.

[0076] Millimeter-wave radar is one of the sensor units in intelligent driving systems. It detects targets by emitting and receiving high-frequency electromagnetic waves and uses the echoes to calculate information such as the target's distance, speed, and angle.

[0077] The camera is one of the sensor units in the intelligent driving system. By processing the images captured by the camera, it can detect lane lines, traffic signs, etc., and assist the intelligent driving system in making decision-making and control.

[0078] An inertial navigation module (IMU) provides high-precision positioning and relative vehicle displacement information for intelligent driving vehicles. It is used in conjunction with a positioning module (GNSS) to assist intelligent driving vehicles in driving within the expected path according to navigation.

[0079] The time synchronization controller, based on software modules, achieves microsecond-level time synchronization (1µs timestamp resolution) for various sensors such as LiDAR, millimeter-wave radar, camera, and IMU; it integrates a PPS signal decoding module to trigger synchronous acquisition by LiDAR, camera, and IMU.

[0080] The computing unit (also called an electronic device) employs an embedded GPU and deploys a lightweight CNN model (accelerated by TensorRT) and an optimized algorithm library. It runs the environmental feature extraction network and optimized algorithm mentioned in this embodiment.

[0081] The dynamic calibration engine performs coarse calibration, optimization, and online compensation algorithms.

[0082] Coarse calibration module: Performs EPnP+RANSAC initial extrinsic parameter estimation and outputs the initial extrinsic parameter matrix and confidence score.

[0083] Optimized kernel: Implements nonlinear least squares solution with parameterized Lie algebras, and supports switching between LM and Doggleg algorithms.

[0084] Compensation logic: Integrated temperature-vibration compensation lookup table (LUT) and EKF filter library.

[0085] The visualization interface displays a calibration error heatmap (color-mapped calibration residual distribution) and 3D point cloud-image alignment status (OpenGL rendering). It also provides a calibration report generation function (PDF / CSV format), including statistics on key parameters (such as reprojection error RMS and extrinsic parameter matrix condition number).

[0086] Figure 2 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method specifically includes:

[0087] S201: Extract dynamic environment features based on pre-acquired multi-dimensional data.

[0088] In this step, in order to improve the accuracy of sensor calibration, multi-dimensional data corresponding to multiple sensors can be used to provide a data foundation for extrinsic parameter estimation. The multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters.

[0089] Specifically, based on IMU motion parameters, the lidar point cloud data is corrected through quaternion attitude calculation to obtain corrected lidar point cloud data. Ground segmentation is performed on the corrected lidar point cloud data to obtain lidar point cloud features. Lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain camera image features.

[0090] S202: Construct the initial extrinsic parameter matrix based on dynamic environment characteristics.

[0091] In this step, after extracting the dynamic environment features, the initial extrinsic parameter matrix is ​​constructed based on the dynamic environment features.

[0092] Specifically, feature alignment is performed based on the lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs. Based on the 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

[0093] S203: Combining the vehicle kinematics model and the Lie group optimization algorithm, the initial extrinsic parameter matrix is ​​iteratively updated.

[0094] In this step, in order to improve the calibration accuracy, the vehicle kinematics model can be combined with the Lie group optimization algorithm to achieve iterative optimization and update of the initial extrinsic parameter matrix.

[0095] Specifically, the Gauss-Newton method is used in the SE(3) manifold space to optimize the initial extrinsic parameter matrix by minimizing the pre-set joint calibration objective function. The cumulative pose error is corrected by using the extended Kalman filter in combination with the wheel speed signal obtained during vehicle driving through the vehicle kinematic model.

[0096] S204: Compensate the updated initial extrinsic parameter matrix based on the pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix.

[0097] In this step, in order to suppress parameter drift caused by temperature and vibration and ensure the long-term temperature of the calibration results, after obtaining the updated initial extrinsic parameter matrix, it is necessary to compensate the updated initial extrinsic parameter matrix based on the temperature and vibration parameters to obtain the final extrinsic parameter matrix.

[0098] Specifically, the translation vector is linearly adjusted based on temperature parameters, and the rotation matrix is ​​corrected based on vibration parameters. Temperature compensation suppresses installation position offset caused by thermal expansion and contraction, while the rotation matrix correction based on vibration parameters includes performing FFT analysis on the IMU vibration spectrum and introducing bandpass filtering (20-50Hz) in pitch angle optimization to suppress parameter drift caused by high-frequency noise.

[0099] Optionally, the method may further include:

[0100] S205: Cross-validation is performed based on millimeter-wave radar detection data and calibrated fusion sensing results.

[0101] S206: If the verification result does not meet the preset condition threshold, dynamic recalibration is triggered.

[0102] To achieve online monitoring and adaptive maintenance of the calibration status, static obstacles, such as streetlights, are monitored by millimeter-wave radar. The results are then compared with the fusion perception results after final extrinsic parameter matrix calibration to calculate lateral deviation and heading angle deviation. Pre-set threshold conditions, such as a lateral deviation threshold of 5 cm and a heading angle deviation threshold of 0.5 degrees, trigger local or global recalibration if the calculated lateral deviation is greater than 5 cm and / or the calculated heading angle deviation is greater than 0.5 degrees.

[0103] The intelligent driving sensor calibration method provided in this application extracts dynamic environmental features from pre-acquired multi-dimensional data, constructs an initial extrinsic parameter matrix based on these features, iteratively updates the initial extrinsic parameter matrix using a vehicle kinematics model and a Lie group optimization algorithm, and compensates the updated initial extrinsic parameter matrix based on pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix. Cross-validation is then performed based on millimeter-wave radar detection data and the fused perception results after calibration. If the validation results do not meet a preset threshold, dynamic recalibration is triggered. This method solves the problems of low efficiency and poor adaptability in traditional calibration methods and significantly improves the robustness and long-term stability of sensor calibration through dynamic compensation and online verification mechanisms.

[0104] Figure 3 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 2 ,like Figure 3 As shown, based on the above embodiments, the method further includes:

[0105] S301: Compensates for timing deviations of different sensors through IMU pre-integration algorithm.

[0106] S302: Perform spatiotemporal alignment based on a unified time base to obtain multi-dimensional data of timestamp alignment.

[0107] Due to differences in hardware sampling rates and transmission delays, multiple sensors can cause data timestamp misalignment, which directly affects calibration accuracy. To ensure calibration accuracy, it is necessary to solve the problem of data asynchrony among multiple sensors. This can be achieved by first compensating for timing deviations, and then using GNSS second pulses to trigger synchronization of various sensors such as lidar, cameras, and IMUs, thereby obtaining multi-dimensional data with timestamp alignment.

[0108] Specifically, between adjacent lidar frames, the high-frequency angular velocity and acceleration of the IMU are integrated to calculate the relative pose change, thereby obtaining the vehicle motion increment between the two lidars. The camera image data is then interpolated onto the lidar timestamp through the IMU pre-integration result, so that the timestamps of all sensor data streams are aligned to the lidar frame time.

[0109] Using the GNSS second pulse signal as a global time reference triggers the hardware timestamp alignment of each sensor. The LiDAR and camera are sampled on the rising edge of the GNSS second pulse, and the IMU data is aligned with the GNSS pulse via timestamps. This eliminates clock skew and ensures strict synchronization of multi-sensor data under a global time reference.

[0110] Optionally, for scenarios without GNSS, the V2X roadside unit clock synchronization or the IMU internal clock can be switched.

[0111] The intelligent driving sensor calibration method provided in this application compensates for timing deviations between different sensors using an IMU pre-integration algorithm, performs spatiotemporal alignment based on a unified time reference, and obtains multi-dimensional data with timestamp alignment. This method solves the timing drift problem caused by differences in sensor sampling rates in traditional calibration, providing a highly consistent data foundation for dynamic calibration and real-time perception.

[0112] Figure 4 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 3 ,like Figure 4 As shown, based on the above embodiments, the dynamic environment features include lidar point cloud features and camera image features. Therefore, step S201 specifically includes:

[0113] S401: Based on IMU motion parameters, the lidar point cloud data is corrected through quaternion attitude calculation to obtain the corrected lidar point cloud data.

[0114] In this step, when the LiDAR scans during vehicle movement, changes in vehicle posture (such as acceleration or turning) can cause different points in the same frame of the point cloud to correspond to different vehicle poses at different times, resulting in "motion distortion" (manifested as point cloud stretching or twisting). For example, when the vehicle turns, the front and rear point clouds may shift in position due to the scanning time difference. Therefore, before extracting dynamic environmental features based on the LiDAR point cloud data, it is necessary to correct the LiDAR point cloud data.

[0115] Specifically, IMU motion parameters are used for correction via quaternion attitude calculation. The IMU motion parameters are six-axis motion parameters (three-axis acceleration and three-axis angular velocity). First, the attitude change between adjacent time steps is calculated based on the integral of the angular velocity, and the quaternions are updated. The quaternions are then converted into a selection matrix for point cloud correction. For each point in a single frame of the LiDAR point cloud, the IMU attitude is interpolated according to its scanning time, thereby eliminating point cloud distortion caused by vehicle motion and restoring the point cloud to a "static scan" state.

[0116] S402: Perform ground segmentation on the corrected lidar point cloud data to obtain lidar point cloud features.

[0117] In this step, the original point cloud contains multiple types of information such as ground, obstacles, and noise. In order to accurately calibrate, it is necessary to separate the ground point cloud to extract effective lane line features.

[0118] Specifically, ground segmentation can be achieved based on deep learning, and key point clouds can be extracted by using convolutional neural networks to obtain the point cloud features of LiDAR.

[0119] Optionally, to improve calibration accuracy, preprocessing can be performed after collecting the lidar point cloud data, such as removing excessively distant points that are more than a preset distance from the vehicle. The above-described ground segmentation is merely an example using deep learning; other implementation methods can be chosen in practical applications, and this application does not impose specific limitations.

[0120] S403: Lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain camera image features.

[0121] In this step, the camera image data needs to be quickly used to extract high-precision coordinates of traffic sign vertices for spatial matching with the lidar point cloud.

[0122] Specifically, a lightweight CNN is used to extract the vertex coordinates of traffic signs, thereby obtaining camera image features.

[0123] The intelligent driving sensor calibration method provided in this application, based on IMU motion parameters, corrects LiDAR point cloud data through quaternion attitude calculation to obtain corrected LiDAR point cloud data. Ground segmentation is then performed on the corrected LiDAR point cloud data to obtain LiDAR point cloud features. Lightweight CNNs are used to extract traffic sign vertex coordinates from camera image data to obtain camera image features. This method achieves high-precision extraction and coordination of multi-sensor features in dynamic environments, laying a reliable data foundation for subsequent calibration optimization.

[0124] Figure 5A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 4 ,like Figure 5 As shown, based on the above embodiments, step S202 specifically includes:

[0125] S501: Perform feature alignment based on the lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs.

[0126] In this step, in order to provide accurate data support for subsequent extrinsic parameter matrix calculations, it is necessary to establish a precise geometric correspondence between the lidar (3D space) and the camera (2D image).

[0127] Specifically, based on the LiDAR point cloud features, a cubic Bézier curve lane model is fitted. The camera image features are transformed from the image coordinate system to the vehicle coordinate system through inverse perspective mapping, thus obtaining the lane line edge point sequence after inverse perspective mapping. Then, spatial matching is performed between the two. Specifically, a set of 3D points is sampled from the Bézier curve model, and the lane line edge points of the lane line edge point sequence are matched with the LiDAR sampled points using nearest neighbor matching. For each LiDAR 3D point, the nearest 2D point in the lane line edge point sequence is found in terms of Euclidean distance. Points with a distance less than a preset distance are retained as valid matching pairs, thus obtaining 3D-2D corresponding point pairs.

[0128] S502: Based on 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

[0129] In this step, the initial extrinsic parameter matrix of the lidar to the camera is calculated using 3D-2D corresponding point pairs, providing high-precision initial values ​​for subsequent Lie group optimization.

[0130] Specifically, the EPnP algorithm is used to solve for the initial extrinsic parameters. This algorithm solves for the camera pose by minimizing the reprojection error, representing the LiDAR 3D points as a weighted sum of four virtual control points. Then, mismatches are eliminated using Mahalanobis distance; points with Mahalanobis distances greater than a preset distance are identified as mismatches and removed. Finally, RANSAC iterative optimization is performed, randomly sampling the minimum point set to calculate candidate extrinsic parameters, counting the number of interior points, iterating a preset number of times, and selecting the extrinsic parameter matrix with the most interior points as the initial extrinsic parameter matrix.

[0131] The intelligent driving sensor calibration method provided in this application aligns features based on LiDAR point cloud features and camera image features to obtain 3D-2D corresponding point pairs. Based on these 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted filtering is used to eliminate mismatched points, generating an initial extrinsic parameter matrix. This method improves feature alignment accuracy and generates a high-precision initial extrinsic parameter matrix, laying a convergent foundation for subsequent Lie group optimization.

[0132] Figure 6 A schematic flowchart of the intelligent driving sensor calibration method provided in this application embodiment. Figure 5 ,like Figure 6 As shown, based on the above embodiments, step S203 specifically includes:

[0133] S601: In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initial extrinsic parameter matrix by minimizing the pre-set joint calibration objective function.

[0134] S602: By combining the vehicle kinematics model with the wheel speed signal obtained during vehicle movement, the cumulative pose error is corrected using an extended Kalman filter.

[0135] Traditional Euclidean space optimization (such as gradient descent) can lead to numerical instability when dealing with rotation matrices due to Euler angle singularities (such as gimbal locks). Furthermore, directly optimizing the rotation matrix may result in non-orthogonal solutions, violating the mathematical properties of the rotation matrix. Therefore, by parameterizing the extrinsic parameter matrix in the SE(3) Lie group space (rigid body motion group) and avoiding singularities through Lie algebras, we can ensure that the optimization results always satisfy the rotation matrix.

[0136] Specifically, the extrinsic parameter matrix is ​​represented as a Lie algebra, and a joint calibration objective function is pre-defined and constructed based on the LiDAR-camera system. The joint calibration objective function can be specifically expressed as:

[0137]

[0138] Among them, P lidar i ∈R 3 , representing the coordinates of the i-th 3D point cloud in the lidar coordinate system, P camera i ∈R 2 , represents the corresponding 2D pixel coordinates in the camera image plane, R∈S0(3) represents the rotation matrix (special orthogonal matrix) of the laser radar to the camera, t∈R 3 Let π(·) represent the translation vector from the laser radar to the camera, and let π(·) represent the camera projection model.

[0139] The optimization process terminates by iteratively updating the Lie algebra until the maximum number of iterations is reached. The Lie algebra update is achieved through an exponential mapping transformation.

[0140] Pure sensor calibration (such as LiDAR-camera) ignores the vehicle's own motion characteristics (such as Ackerman steering geometry), leading to increased cumulative errors in dynamic scenarios. Furthermore, the calibration process does not integrate wheel speed signals, making it impossible to correct IMU integral drift; after long-term use, loose mounting brackets or temperature deformation can cause calibration parameters to gradually become invalid. Therefore, incorporating vehicle kinematic constraints can prevent purely data-driven optimization from getting trapped in local optima.

[0141] The vehicle kinematics model is built upon Ackerman steering geometry, which relates the front wheel steering angle to the yaw rate. Wheel speed and steering angle are then acquired via the CAN bus to predict vehicle pose changes. The predicted vehicle motion state (position, heading angle) is combined with the initial extrinsic parameter matrix to form the state vector of an extended Kalman filter, predicting the state and covariance at the next moment. Obstacle positions perceived by LiDAR and camera fusion, along with IMU measurements, are used to calculate the difference between the actual observed and predicted states. Kalman gain is then used to fuse the predicted and observed data, dynamically adjusting the translation and rotation components in the extrinsic parameter matrix to suppress accumulated errors.

[0142] The intelligent driving sensor calibration method provided in this application adopts the Gauss-Newton method in the SE(3) manifold space. By minimizing a pre-set joint calibration objective function, the initial extrinsic parameter matrix is ​​optimized. The cumulative pose error is corrected by combining the vehicle kinematics model with the wheel speed signal obtained during vehicle driving and using an extended Kalman filter. The above method achieves high-precision extrinsic parameter estimation through SE(3) manifold optimization, solves the numerical instability problem of traditional methods, and introduces vehicle kinematic constraints to suppress cumulative errors and improve long-term stability through an extended Kalman filter.

[0143] Figure 7 This is a schematic diagram of the structure of the calibration device for intelligent driving sensors provided in the embodiments of this application, as shown below. Figure 7 As shown, the calibration device 700 for intelligent driving sensors includes:

[0144] The extraction module 701 is used to extract dynamic environment features based on pre-acquired multi-dimensional data, including LiDAR point cloud data, camera image data and IMU motion parameters.

[0145] Module 702 is used to construct the initial extrinsic parameter matrix based on dynamic environment characteristics;

[0146] Update module 703 is used to iteratively update the initial extrinsic parameter matrix by combining the vehicle kinematics model and the Lie group optimization algorithm;

[0147] The first compensation module 704 is used to compensate the updated initial extrinsic parameter matrix based on the pre-acquired temperature parameters and vibration parameters to obtain the final extrinsic parameter matrix.

[0148] In one possible implementation, the calibration device 700 for intelligent driving sensors also includes:

[0149] The second compensation module 705 is used to compensate for timing deviations of different sensors through an IMU pre-integration algorithm;

[0150] The acquisition module 706 is used for spatiotemporal alignment based on a unified time reference, and to acquire multi-dimensional data with timestamp alignment. The unified time reference is the GNSS second pulse signal.

[0151] In one possible implementation, the extraction module 701 specifically includes:

[0152] Based on IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude calculation to obtain the corrected lidar point cloud data.

[0153] Ground segmentation is performed on the corrected lidar point cloud data to obtain lidar point cloud features;

[0154] A lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0155] In one possible implementation, module 702 specifically includes:

[0156] Feature alignment is performed based on the lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs;

[0157] Based on 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

[0158] In one possible implementation, updating module 703 specifically includes:

[0159] In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initial extrinsic parameter matrix by minimizing the pre-defined joint calibration objective function;

[0160] By combining the vehicle's kinematics model with the wheel speed signals acquired during vehicle movement, an extended Kalman filter is used to correct the cumulative pose error.

[0161] In one possible implementation, the first compensation module 704 specifically includes:

[0162] The translation vector is linearly adjusted based on temperature parameters;

[0163] The rotation matrix is ​​corrected based on vibration parameters.

[0164] In one possible implementation, the calibration device 700 for intelligent driving sensors also includes:

[0165] The verification module 707 is used to perform cross-validation based on millimeter-wave radar detection data and calibrated fusion sensing results;

[0166] The recalibration module 708 is used to trigger dynamic recalibration if the verification result does not meet the preset condition threshold.

[0167] The calibration device for intelligent driving sensors provided in this embodiment can execute the calibration method for intelligent driving sensors provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0168] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the electronic device 800 includes: a storage unit 801 and a processing unit 802;

[0169] Storage unit 801 stores computer-executed instructions;

[0170] The processing unit 802 executes the computer execution instructions stored in the storage unit 801, causing the processing unit 802 to execute the above-mentioned intelligent driving sensor calibration method.

[0171] Optionally, the electronic device further includes a communication component 803. The processing unit 802, the storage unit 801, and the communication component 803 are connected via a bus 804.

[0172] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processing unit can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0173] The storage unit may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0174] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0176] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0177] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0179] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A calibration method for intelligent driving sensors, characterized in that it includes: Based on pre-acquired multi-dimensional data, dynamic environment features are extracted. The multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters. Based on the dynamic environment characteristics, an initial extrinsic parameter matrix is ​​constructed; The initial extrinsic parameter matrix is ​​iteratively updated by combining the vehicle kinematics model and the Lie group optimization algorithm; The updated initial extrinsic parameter matrix is ​​compensated based on the pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix; The iterative update of the initial extrinsic parameter matrix, combining the vehicle kinematics model and the Lie group optimization algorithm, includes: In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initial extrinsic parameter matrix by minimizing the pre-set joint calibration objective function; the cumulative pose error is corrected by using the extended Kalman filter in combination with the wheel speed signal obtained during vehicle driving through the vehicle kinematic model; wherein, the vehicle kinematic model is based on Ackerman steering geometry, which is the relationship between the front wheel steering angle and the yaw rate, and the wheel speed and steering angle are obtained through the CAN bus to predict the vehicle pose change.

2. The method according to claim 1, characterized in that, The method further includes: The timing deviation of different sensors is compensated by the IMU pre-integration algorithm; Spatiotemporal alignment is performed based on a unified time reference to obtain the timestamp-aligned multi-dimensional data, wherein the unified time reference is a GNSS second pulse signal.

3. The method according to claim 1, characterized in that, Dynamic environment features include LiDAR point cloud features and camera image features. The step of extracting dynamic environment features based on pre-acquired multi-dimensional data includes: Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude calculation to obtain the corrected lidar point cloud data. The corrected lidar point cloud data is segmented on the ground to obtain the lidar point cloud features. Lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

4. The method according to claim 3, characterized in that, The construction of the initial extrinsic parameter matrix based on the dynamic environment characteristics includes: Based on the lidar point cloud features and the camera image features, feature alignment is performed to obtain 3D-2D corresponding point pairs; Based on the 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighted elimination of mismatched points is used to generate the initial extrinsic parameter matrix.

5. The method according to claim 1, characterized in that, The extrinsic parameter matrix includes a rotation matrix and a translation vector. The compensation of the updated initial extrinsic parameter matrix based on pre-acquired temperature and vibration parameters includes: The translation vector is linearly adjusted based on the temperature parameter; The rotation matrix is ​​corrected based on the vibration parameters.

6. The method according to claim 1, characterized in that, The method further includes: Cross-validation is performed based on millimeter-wave radar detection data and calibrated fusion sensing results; If the verification result does not meet the preset threshold, dynamic recalibration is triggered.

7. A calibration device for an intelligent driving sensor, characterized in that, include: The extraction module is used to extract dynamic environment features based on pre-acquired multi-dimensional data, including lidar point cloud data, camera image data, and IMU motion parameters. The construction module is used to construct an initial extrinsic parameter matrix based on the dynamic environment characteristics; The update module is used to iteratively update the initial extrinsic parameter matrix by combining the vehicle kinematics model and the Lie group optimization algorithm; The first compensation module is used to compensate the updated initial extrinsic parameter matrix based on the pre-acquired temperature and vibration parameters to obtain the final extrinsic parameter matrix. The update module specifically includes: In the SE(3) manifold space, the Gauss-Newton method is used to optimize the initial extrinsic parameter matrix by minimizing the pre-set joint calibration objective function; the cumulative pose error is corrected by using the extended Kalman filter in combination with the wheel speed signal obtained during vehicle driving through the vehicle kinematic model; wherein, the vehicle kinematic model is based on Ackerman steering geometry, which is the relationship between the front wheel steering angle and the yaw rate, and the wheel speed and steering angle are obtained through the CAN bus to predict the vehicle pose change.

8. An electronic device, characterized in that, include: Storage unit, processing unit; The storage unit stores computer-executed instructions; The processing unit executes the computer execution instructions stored in the storage unit, causing the processing unit to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images