Calibration method, device and equipment of intelligent driving sensor and storage medium

Abstract

The invention relates to a calibration method, device and equipment for an intelligent driving sensor and a storage medium, and the method comprises the steps: extracting dynamic environment features according to pre-obtained multi-dimensional data, constructing an initial external parameter matrix based on the dynamic environment features, carrying out the iterative updating of the initial external parameter matrix through combining a vehicle kinematics model and a Lie group optimization algorithm, and carrying out the calibration of the intelligent driving sensor. And compensating the updated initial external parameter matrix according to a temperature parameter and a vibration parameter which are acquired in advance to obtain a final external parameter matrix, performing cross validation based on millimeter wave radar detection data and a calibrated fusion sensing result, and triggering dynamic re-calibration if a validation result does not meet a preset condition threshold value. According to the method, the problems of low efficiency and poor adaptability of traditional calibration are solved, and the robustness and long-term stability of sensor calibration are remarkably improved through a dynamic compensation and online verification mechanism.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a calibration method, device, equipment and storage medium for an intelligent driving sensor. Background Art

[0002] The intelligent driving system is a multi-sensor fusion perception system. Vehicles equipped with this system need to calibrate each sensor after they roll off the production line to determine the position and posture of each sensor in the vehicle body coordinate system, avoiding perception and positioning deviations caused by installation position deviations.

[0003] Existing technologies usually use fixed sensor calibration methods based on calibration plates, which require dedicated calibration sites and high-precision calibration patterns.

[0004] However, the above method is only applicable to calibration after the factory is offline. It lacks flexibility and has great limitations, resulting in low calibration efficiency and insufficient accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a driving sensor calibration method, device, equipment and storage medium for solving the problem of reduced or insufficient accuracy of automobile intelligent driving sensors in dynamic environments.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A calibration method for an intelligent driving sensor, comprising:

[0008] Extracting dynamic environment features based on pre-acquired multi-dimensional data, wherein the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters;

[0009] Based on the dynamic environment characteristics, construct an initial external parameter matrix;

[0010] Combining the vehicle kinematic model and the Lie group optimization algorithm, iteratively updating the initial extrinsic parameter matrix;

[0011] The updated initial extrinsic parameter matrix is ​​compensated according to the pre-acquired temperature parameters and vibration parameters to obtain the final extrinsic parameter matrix.

[0012] In one possible implementation, the method further includes:

[0013] Compensate for the timing deviation of different sensors through IMU pre-integration algorithm;

[0014] Spatiotemporal alignment is performed based on a unified time reference to obtain the multi-dimensional data with time stamp alignment, wherein the unified time reference is a GNSS second pulse signal.

[0015] In one possible implementation, the dynamic environment features include lidar point cloud features and camera image features. Then, extracting the dynamic environment features based on the pre-acquired multi-dimensional data includes:

[0016] Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude settlement to obtain corrected lidar point cloud data;

[0017] Performing ground segmentation on the corrected lidar point cloud data to obtain the lidar point cloud features;

[0018] A lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0019] In one possible implementation, constructing an initial extrinsic parameter matrix based on the dynamic environment characteristics includes:

[0020] Perform feature alignment based on the laser radar point cloud features and the camera image features to obtain 3D-2D corresponding point pairs;

[0021] Based on the 3D-2D corresponding point pairs, the EPnP algorithm is used in combination with Mahalanobis distance weighting to eliminate mismatched points and generate the initial extrinsic parameter matrix.

[0022] In one possible implementation, combining the vehicle kinematic model and the Lie group optimization algorithm to iteratively update the initial extrinsic parameter matrix includes:

[0023] The Gauss-Newton method is used in the SE(3) manifold space to optimize the initialized extrinsic parameter matrix by minimizing a pre-set joint calibration objective function;

[0024] The vehicle kinematic model is combined with the wheel speed signal obtained during vehicle driving, and the accumulated posture error is corrected using an extended Kalman filter.

[0025] In one possible implementation, the extrinsic parameter matrix includes a rotation matrix and a translation vector, and compensating the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters includes:

[0026] linearly adjusting the translation vector based on the temperature parameter;

[0027] The rotation matrix is ​​modified based on the vibration parameters.

[0028] In one possible implementation, the method further includes:

[0029] Perform cross-validation based on millimeter-wave radar detection data and calibrated fusion perception results;

[0030] If the verification result does not meet the preset condition threshold, dynamic recalibration is triggered.

[0031] A calibration device for an intelligent driving sensor, comprising:

[0032] An extraction module is used to extract dynamic environment features based on pre-acquired multi-dimensional data, wherein the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters;

[0033] A construction module, configured to construct an initial external parameter matrix based on the dynamic environment characteristics;

[0034] An updating module, configured to iteratively update the initial extrinsic parameter matrix by combining a vehicle kinematic model and a Lie group optimization algorithm;

[0035] The first compensation module is used to compensate the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters to obtain a final extrinsic parameter matrix.

[0036] In one possible implementation, the device further includes:

[0037] The second compensation module is used to compensate for the timing deviation of different sensors through the IMU pre-integration algorithm;

[0038] The acquisition module is used to perform spatiotemporal alignment based on a unified time reference to acquire the multi-dimensional data aligned with timestamps, wherein the unified time reference is a GNSS second pulse signal.

[0039] In one possible implementation, the extraction module specifically includes:

[0040] Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude settlement to obtain corrected lidar point cloud data;

[0041] Performing ground segmentation on the corrected lidar point cloud data to obtain the lidar point cloud features;

[0042] A lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0043] In one possible implementation, the building blocks specifically include:

[0044] Perform feature alignment based on the laser radar point cloud features and the camera image features to obtain 3D-2D corresponding point pairs;

[0045] Based on the 3D-2D corresponding point pairs, the EPnP algorithm is used in combination with Mahalanobis distance weighting to eliminate mismatched points and generate the initial extrinsic parameter matrix.

[0046] In one possible implementation, the update module specifically includes:

[0047] The Gauss-Newton method is used in the SE(3) manifold space to optimize the initialized extrinsic parameter matrix by minimizing a pre-set joint calibration objective function;

[0048] The vehicle kinematic model is combined with the wheel speed signal obtained during vehicle driving, and the accumulated posture error is corrected using an extended Kalman filter.

[0049] In one possible implementation, the first compensation module specifically includes:

[0050] linearly adjusting the translation vector based on the temperature parameter;

[0051] The rotation matrix is ​​modified based on the vibration parameters.

[0052] In one possible implementation, the device further includes:

[0053] The verification module is used to perform cross-validation based on millimeter-wave radar detection data and calibrated fusion perception results;

[0054] The recalibration module is used to trigger dynamic recalibration if the verification result does not meet the preset condition threshold.

[0055] An electronic device comprising: a storage unit, a processing unit;

[0056] The storage unit stores computer-executable instructions;

[0057] The processing unit executes the computer-executable instructions stored in the storage unit, so that the processing unit performs the method described above.

[0058] A computer-readable storage medium stores computer-executable instructions, which are used to implement the method described above when executed by a processor.

[0059] A computer program product comprises a computer program, which implements the method described above when executed by a processor.

[0060] Beneficial effects of the present invention:

[0061] By synchronizing lidar, camera, and IMU data while the vehicle is in motion, the system extracts geometric and semantic features from the road environment (such as lane markings and traffic signs). The system then iteratively updates the sensor extrinsic parameter matrix using a Lie group optimization algorithm and a vehicle kinematic model (Ackerman steering model). Dynamic compensation for temperature and vibration parameters is also introduced. Finally, calibration accuracy is verified through independent observation of static obstacles by the millimeter-wave radar, triggering recalibration. This solution breaks through the limitations of traditional fixed calibration rooms, enabling high-precision sensor calibration in non-fixed scenarios. This improves calibration flexibility and addresses issues such as insufficient calibration accuracy and consistency for intelligent driving during pilot production and after-sales maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the calibration architecture for intelligent driving systems;

[0063] Figure 2 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 1 ;

[0064] Figure 3 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 2 ;

[0065] Figure 4 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 3 ;

[0066] Figure 5 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 4 ;

[0067] Figure 6 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 5 ;

[0068] Figure 7 A schematic diagram of the structure of a calibration device for an intelligent driving sensor provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0071] The intelligent driving system is a multi-sensor fusion perception system. Vehicles equipped with this system need to calibrate each sensor after they are produced, so as to determine the position and posture of each sensor in the vehicle body coordinate system and avoid perception and positioning errors caused by installation position deviation. However, traditional calibration equipment is mostly fixed structure, which lacks flexibility. In addition, temporary calibration is required in the following scenarios, which has problems such as low efficiency, insufficient calibration accuracy, and insufficient consistency: (1) During the trial production of new factories and new models, the calibration room is not completed and the vehicles off the production line can only be calibrated temporarily; (2) After-sales 4S stores do not have complete calibration room equipment, and only temporary calibration can be used after replacing sensors. (3) After the sensor is used for a long time, the accuracy decreases and it needs to be returned to the factory or 4S store for recalibration. The existing technology uses a fixed sensor calibration method based on a calibration plate, which requires reliance on a dedicated calibration site and high-precision calibration patterns. It is suitable for calibration after the factory rolls off the production line, but suffers from the aforementioned lack of flexibility and is not suitable for trial production and 4S store after-sales stages. Alternatively, workshop communication collaborative calibration is used, which requires the cooperation of multiple vehicles and has low calibration efficiency (taking ≥ 1 hour), and cannot solve the parameter drift problem caused by sensor aging. Alternatively, offline calibration is triggered by a diagnostic instrument, which is a calibration method commonly used in the current 4S store after-sales stage. However, this method lacks a dynamic environmental feedback mechanism, and the error of the calibration result is significantly increased under complex road conditions.

[0072] In response to the above problems, the present application provides a calibration method, device, equipment and storage medium for intelligent driving sensors, which realize high-precision sensor calibration and calibration in non-fixed scenarios, improve calibration flexibility, and solve the problems of insufficient accuracy and consistency of intelligent driving calibration during trial production and after-sales maintenance. Specifically, the existing fixed sensor calibration method based on calibration plate requires dependence on dedicated calibration site and high-precision calibration pattern, which is suitable for calibration after the factory rolls off the production line. This method is not flexible enough and cannot be applied to trial production and 4S shop after-sales stage; or adopts workshop communication collaborative calibration, which requires multiple vehicles to cooperate and has low calibration efficiency (taking ≥1 hour), but cannot solve the parameter drift problem caused by sensor aging; or triggers offline calibration by diagnostic instrument, which is the calibration method commonly used in the current 4S shop after-sales stage, but this method lacks dynamic environment feedback mechanism, and the error of calibration results increases significantly under complex road conditions. Taking these problems into consideration, the inventor studied whether it is possible to realize high-precision sensor calibration and calibration in non-fixed scenarios and improve calibration flexibility by integrating natural road feature extraction, Lie group optimization algorithm and online dynamic compensation mechanism. Based on this, this proposal is proposed.

[0073] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0074] Figure 1 The schematic diagram of the intelligent driving system calibration architecture is as follows: Figure 1 As shown in the figure, the hardware of the calibration system mainly includes: lidar, millimeter-wave radar, camera, inertial navigation module, time synchronization controller, computing unit, etc.; the software module mainly includes: dynamic calibration engine, visualization interface, etc.

[0075] LiDAR is one of the sensor units in the intelligent driving system. The point cloud data it scans can be used to perceive road conditions within the field of view, ensuring safe driving.

[0076] Millimeter-wave radar is one of the sensor units of the intelligent driving system. It detects targets by emitting and receiving high-frequency electromagnetic waves and uses the echoes to calculate the target's distance, speed, angle and other information.

[0077] The camera is one of the sensor units of the intelligent driving system. It processes the images captured by the camera to detect lane lines, traffic signs, etc., and assists intelligent driving in making decision-making and control.

[0078] The inertial navigation module (IMU) provides high-precision positioning and vehicle relative displacement information for smart driving vehicles. When used in conjunction with the positioning module (GNSS), it assists smart driving vehicles in driving within the expected path according to navigation.

[0079] The time synchronization controller implements microsecond-level time synchronization (time stamp resolution 1us) for various sensors such as lidar, millimeter-wave radar, camera, and IMU based on software modules. It also integrates a PPS signal decoding module to trigger the synchronous acquisition of lidar, camera, and IMU.

[0080] The computing unit (also called an electronic device) uses an embedded GPU and deploys a lightweight CNN model (accelerated by TensorRT) and an optimization algorithm library. It runs the environmental feature extraction network and optimization algorithm mentioned in this embodiment.

[0081] Dynamic calibration engine, performing rough calibration, optimization and online compensation algorithms, etc.:

[0082] Coarse calibration module: performs EPnP+RANSAC initial extrinsic parameter estimation and outputs the initial extrinsic parameter matrix and confidence score.

[0083] Optimization kernel: Implements Lie algebra parameterized nonlinear least squares solution and supports switching between LM algorithm and Dogleg algorithm.

[0084] Compensation logic: integrated temperature-vibration compensation look-up table (LUT) and EKF filter library.

[0085] A visual interface displays calibration error heatmaps (color-mapped calibration residual distribution) and 3D point cloud-image alignment status (OpenGL rendering). It also provides calibration report generation (PDF / CSV format) with key parameter statistics (such as reprojection error RMS and extrinsic matrix condition number).

[0086] Figure 2 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 1 ,like Figure 2 As shown, the method specifically includes:

[0087] S201: Extracting dynamic environment features based on pre-acquired multi-dimensional data.

[0088] In this step, in order to improve the sensor calibration accuracy, a data basis for extrinsic parameter estimation can be provided based on the multi-dimensional data corresponding to multiple sensors, where the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters.

[0089] Specifically, based on the IMU motion parameters, the lidar point cloud data is calibrated through quaternion attitude settlement to obtain the calibrated lidar point cloud data. The calibrated lidar point cloud data is then subjected to ground segmentation to obtain lidar point cloud features. A lightweight CNN is used to extract the vertex coordinates of traffic signs from the camera image data to obtain the camera image features.

[0090] S202: Construct an initial external parameter matrix based on dynamic environment characteristics.

[0091] In this step, after the dynamic environment features are extracted, the initial external parameter matrix is ​​constructed based on the dynamic environment features.

[0092] Specifically, feature alignment is performed based on the lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs. Based on the 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighting is used to eliminate mismatched points and generate the initial extrinsic parameter matrix.

[0093] S203: Combining the vehicle kinematic model and the Lie group optimization algorithm, iteratively update the initial extrinsic parameter matrix.

[0094] In this step, in order to improve the calibration accuracy, the vehicle kinematic model can be combined with the Lie group optimization algorithm to achieve iterative optimization and update of the initial extrinsic parameter matrix.

[0095] Specifically, the Gauss-Newton method is used in the SE(3) manifold space to optimize the initialization extrinsic parameter matrix by minimizing the pre-set joint calibration objective function. The vehicle kinematic model is combined with the wheel speed signal obtained during vehicle driving, and the extended Kalman filter is used to correct the accumulated pose error.

[0096] S204: Compensating the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters to obtain a final extrinsic parameter matrix.

[0097] In this step, in order to suppress the parameter drift caused by temperature and vibration and ensure the long-term temperature of the calibration results, after obtaining the updated initial extrinsic parameter matrix, it is also necessary to compensate the updated initial extrinsic parameter matrix based on the temperature parameters and vibration parameters to obtain the final extrinsic parameter matrix.

[0098] Specifically, the translation vector is linearly adjusted based on temperature parameters, and the rotation matrix is ​​corrected based on vibration parameters. Temperature compensation suppresses mounting offsets caused by thermal expansion and contraction. Correcting the rotation matrix based on vibration parameters involves performing FFT analysis on the IMU vibration spectrum and introducing a bandpass filter (20-50Hz) during pitch angle optimization to suppress parameter drift caused by high-frequency noise.

[0099] Optionally, the method may further include:

[0100] S205: Perform cross-validation based on the millimeter-wave radar detection data and the calibrated fusion perception results.

[0101] S206: If the verification result does not meet the preset condition threshold, dynamic recalibration is triggered.

[0102] To achieve online monitoring and adaptive maintenance of the calibration status, millimeter-wave radar is used to monitor static obstacles, such as streetlight poles, and then compared with the fused perception results after the final external parameter matrix calibration to calculate the lateral deviation and heading angle deviation. Conditional thresholds are pre-set, for example, the lateral deviation threshold is 5 cm and the heading angle deviation threshold is 0.5 degrees. If the calculated lateral deviation is greater than 5 and / or the calculated heading angle deviation is greater than 0.5, local or global recalibration is triggered.

[0103] The calibration method for intelligent driving sensors provided in the embodiment of the present application extracts dynamic environmental features based on pre-acquired multi-dimensional data, constructs an initial extrinsic parameter matrix based on the dynamic environmental features, iteratively updates the initial extrinsic parameter matrix in combination with the vehicle kinematic model and the Lie group optimization algorithm, compensates the updated initial extrinsic parameter matrix based on pre-acquired temperature parameters and vibration parameters, and obtains the final extrinsic parameter matrix. Cross-validation is performed based on millimeter-wave radar detection data and the fused perception results after calibration. If the verification result does not meet the preset condition threshold, dynamic recalibration is triggered. The above method solves the problems of low efficiency and poor adaptability of traditional calibration, and also significantly improves the robustness and long-term stability of sensor calibration through dynamic compensation and online verification mechanisms.

[0104] Figure 3 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 2 ,like Figure 3 As shown, based on the above embodiment, the method further includes:

[0105] S301: Compensate for the timing deviation of different sensors through the IMU pre-integration algorithm.

[0106] S302: Perform spatiotemporal alignment based on a unified time reference to obtain multi-dimensional data with aligned timestamps.

[0107] Due to differences in hardware sampling rates and transmission delays, multiple sensors will cause data timestamp misalignment, which will directly affect the calibration accuracy. In order to ensure calibration accuracy, it is necessary to solve the problem of multi-sensor data asynchrony. The timing deviation can be compensated first, and then the laser radar, camera, IMU and other sensors can be triggered to synchronize through GNSS second pulses to obtain multi-dimensional data with aligned timestamps.

[0108] Specifically, between adjacent lidar frames, the high-frequency angular velocity and acceleration of the IMU are integrated to calculate the relative pose change, thereby obtaining the vehicle motion increment between the two lidars. The camera image data is interpolated to the lidar timestamp through the IMU pre-integration result, so that the timestamps of all sensor data streams are aligned to the lidar frame time.

[0109] Using the GNSS pulse-per-second signal as the global time base, the system triggers the hardware timestamp alignment of each sensor. The LiDAR and camera trigger sampling on the rising edge of the GNSS pulse-per-second signal, and the IMU data is aligned with the GNSS pulse through its timestamp. This eliminates clock drift and ensures that multi-sensor data is strictly synchronized under the global time base.

[0110] Optionally, for scenarios without GNSS, you can switch between V2X roadside unit clock synchronization or IMU internal clock.

[0111] The intelligent driving sensor calibration method provided in this application embodiment uses an IMU pre-integration algorithm to compensate for timing deviations between different sensors, performs spatiotemporal alignment based on a unified time base, and obtains multi-dimensional data with aligned timestamps. This method can address the timing drift caused by differences in sensor sampling rates in traditional calibration, providing a highly consistent data foundation for dynamic calibration and real-time perception.

[0112] Figure 4 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 3 ,like Figure 4 As shown, based on the above embodiments, the dynamic environment features include laser radar point cloud features and camera image features, and step S201 specifically includes:

[0113] S401: Based on the IMU motion parameters, the lidar point cloud data is corrected through quaternion attitude settlement to obtain the corrected lidar point cloud data.

[0114] In this step, when the LiDAR scans a moving vehicle, different points in the same frame of the point cloud may correspond to different vehicle postures at different times due to changes in the vehicle's posture (such as acceleration and turning), resulting in "motion distortion" (manifested as stretching or twisting of the point cloud). For example, when a vehicle turns, the front and rear point clouds will offset due to the difference in scanning time. Therefore, before extracting dynamic environmental features based on the LiDAR point cloud data, the LiDAR point cloud data must be calibrated.

[0115] Specifically, calibration is performed using IMU motion parameters, which are six-axis motion parameters (three-axis acceleration and three-axis angular velocity). First, the attitude change between adjacent moments is calculated based on the angular velocity integral, and a quaternion update is performed. This quaternion is then converted into a selection matrix for point cloud calibration. For each point in a single-frame LiDAR point cloud, the IMU attitude is interpolated based on the scan time, eliminating point cloud distortion caused by vehicle motion and restoring the point cloud to a "static scan" state.

[0116] S402: Perform ground segmentation on the corrected LiDAR point cloud data to obtain LiDAR point cloud features.

[0117] In this step, the original point cloud contains multiple types of information such as the ground, obstacles, and noise. In order to accurately calibrate, it is necessary to separate the ground point cloud to extract effective lane line features.

[0118] Specifically, ground segmentation can be achieved based on deep learning segmentation, and convolutional neural networks can be used to segment and extract key point clouds to obtain lidar point cloud features.

[0119] Optionally, to improve calibration accuracy, preprocessing can be performed after collecting the LiDAR point cloud data, such as removing points that are more than a preset distance from the vehicle. The above ground segmentation is merely an example using deep learning. In actual applications, other implementation methods may be used, and this embodiment of the present application does not specifically limit this.

[0120] S403: Using a lightweight CNN to extract traffic sign vertex coordinates from the camera image data to obtain camera image features.

[0121] In this step, the camera image data needs to quickly extract high-precision traffic sign vertex coordinates for spatial matching with the lidar point cloud.

[0122] Specifically, the vertex coordinates of traffic signs are extracted through lightweight CNN to obtain camera image features.

[0123] The smart driving sensor calibration method provided in this application embodiment calibrates LiDAR point cloud data using quaternion attitude calculation based on IMU motion parameters to obtain calibrated LiDAR point cloud data. Ground segmentation is performed on the calibrated LiDAR point cloud data to obtain LiDAR point cloud features. A lightweight CNN is used to extract traffic sign vertex coordinates from camera image data to obtain camera image features. This method achieves high-precision extraction and coordination of multi-sensor features in dynamic environments, laying a reliable data foundation for subsequent calibration optimization.

[0124] Figure 5Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 4 ,like Figure 5 As shown, based on the above embodiments, step S202 specifically includes:

[0125] S501: Perform feature alignment based on the lidar point cloud features and the camera image features to obtain 3D-2D corresponding point pairs.

[0126] In this step, in order to provide accurate data support for the subsequent extrinsic parameter matrix calculation, it is necessary to establish an accurate geometric correspondence between the lidar (3D space) and the camera (2D image).

[0127] Specifically, a cubic Bezier curve lane model is fitted based on LiDAR point cloud features. The camera image features are then transformed from the image coordinate system to the vehicle body coordinate system through inverse perspective mapping, resulting in a sequence of lane edge points after inverse perspective mapping. The two are then spatially matched. Specifically, a set of 3D points is sampled from the Bezier curve model. Nearest-neighbor matching is performed between the lane edge points in the sequence and the LiDAR sampled points. For each LiDAR 3D point, the 2D point with the closest Euclidean distance is found in the sequence. Pairs with distances less than a preset distance are retained as valid matches, resulting in corresponding 3D-2D point pairs.

[0128] S502: Based on the 3D-2D corresponding point pairs, the EPnP algorithm is used in combination with Mahalanobis distance weighting to eliminate mismatched points and generate an initial extrinsic parameter matrix.

[0129] In this step, the initial extrinsic parameter matrix from the lidar to the camera is calculated using 3D-2D corresponding point pairs, providing high-precision initial values ​​for subsequent Lie group optimization.

[0130] Specifically, the EPnP algorithm is used to solve the initial extrinsic parameters. This algorithm solves the camera pose by minimizing the reprojection error and represents the LiDAR 3D points as a weighted sum of four virtual control points. False matches are then eliminated using the Mahalanobis distance. Points with a Mahalanobis distance greater than a preset distance are considered false matches and removed. RANSAC iterative optimization is then performed, randomly sampling a minimum point set to calculate candidate extrinsic parameters. The number of inliers is counted, and after a preset number of iterations, the extrinsic parameter matrix with the most inliers is selected as the initial extrinsic parameter matrix.

[0131] The intelligent driving sensor calibration method provided in this application embodiment performs feature alignment based on lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs. Based on these 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighting is used to eliminate mismatched points and generate an initial extrinsic parameter matrix. This method improves feature alignment accuracy and generates a high-precision initial extrinsic parameter matrix, laying the foundation for convergence of subsequent Lie group optimization.

[0132] Figure 6 Schematic diagram of the calibration method of the intelligent driving sensor provided in the embodiment of the present application Figure 5 ,like Figure 6 As shown, based on the above embodiments, step S203 specifically includes:

[0133] S601: Using the Gauss-Newton method in the SE(3) manifold space, the initialized extrinsic parameter matrix is ​​optimized by minimizing a pre-set joint calibration objective function.

[0134] S602: Using the vehicle kinematic model and the wheel speed signal obtained during vehicle driving, the accumulated posture error is corrected using an extended Kalman filter.

[0135] Traditional Euclidean space optimization (such as gradient descent) can lead to numerical instability when processing rotation matrices due to Euler angle singularities (such as gimbal lock). Furthermore, directly optimizing the rotation matrix can lead to non-orthogonal solutions, destroying the mathematical properties of the rotation matrix. By parameterizing the extrinsic matrix in the SE(3) Lie group space (rigid body motion group), singularities are avoided through Lie algebra, ensuring that the optimization result always satisfies the rotation matrix.

[0136] Specifically, the extrinsic parameter matrix is ​​expressed as Lie algebra, the joint calibration objective function is pre-set, and it is constructed based on the lidar-camera. The joint calibration objective function can be specifically expressed as:

[0137]

[0138] Among them, P lidar i ∈R 3 , represents the i-th 3D point cloud coordinate in the laser radar coordinate system, P camera i ∈R 2 , represents the corresponding 2D pixel coordinates in the camera image plane, R∈S0(3) represents the rotation matrix (special orthogonal matrix) from the laser radar to the camera, t∈R 3 represents the translation vector from the lidar to the camera, and π(·) represents the camera projection model.

[0139] The optimization is terminated by iteratively updating the Lie algebra until the maximum number of iterations is reached. The Lie algebra update is transformed through an exponential mapping.

[0140] Pure sensor calibration (e.g., lidar-camera) ignores the vehicle's inherent kinematic characteristics (e.g., Ackermann steering geometry), leading to increased cumulative error in dynamic scenarios. Furthermore, the calibration process fails to incorporate wheel speed signals, making it impossible to correct for IMU integral drift. Over time, loosening of mounting brackets or thermal deformation can cause calibration parameters to become ineffective. Therefore, integrating vehicle kinematic constraints can prevent purely data-driven optimization from falling into local optima.

[0141] The vehicle kinematic model is based on Ackermann steering geometry, which represents the relationship between the front wheel steering angle and the yaw rate. Wheel speed and steering angle are then acquired via the CAN bus to predict vehicle posture changes. The predicted vehicle motion state (position, heading angle) is combined with the initial extrinsic parameter matrix to form the state vector of the extended Kalman filter, which then predicts the state and covariance at the next moment. Obstacle positions perceived by the lidar-camera fusion and IMU measurements are used to calculate the difference between the actual observed and predicted states. The predicted and observed data are then fused using the Kalman gain, dynamically adjusting the translation and rotation components of the extrinsic parameter matrix to suppress cumulative error.

[0142] The calibration method for intelligent driving sensors provided in the embodiments of the present application uses the Gauss-Newton method within the SE(3) manifold space to optimize the initialization extrinsic parameter matrix by minimizing a pre-set joint calibration objective function. The vehicle kinematic model is combined with the wheel speed signal acquired during vehicle driving to correct the accumulated pose error using an extended Kalman filter. This method achieves high-precision extrinsic parameter estimation through SE(3) manifold optimization, addressing the numerical instability issues of traditional methods. Furthermore, the method introduces vehicle kinematic constraints and suppresses accumulated errors through an extended Kalman filter, thereby improving long-term stability.

[0143] Figure 7 This is a schematic diagram of the structure of the calibration device of the intelligent driving sensor provided in the embodiment of the present application, as shown in FIG. Figure 7 As shown, the calibration device 700 of the intelligent driving sensor includes:

[0144] Extraction module 701, for extracting dynamic environment features based on pre-acquired multi-dimensional data, where the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters;

[0145] A construction module 702 is used to construct an initial external parameter matrix based on dynamic environment characteristics;

[0146] An updating module 703 is used to iteratively update the initial extrinsic parameter matrix by combining the vehicle kinematic model and the Lie group optimization algorithm;

[0147] The first compensation module 704 is configured to compensate the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters to obtain a final extrinsic parameter matrix.

[0148] In one possible implementation, the calibration device 700 for the intelligent driving sensor further includes:

[0149] A second compensation module 705 is used to compensate for the timing deviation of different sensors through an IMU pre-integration algorithm;

[0150] The acquisition module 706 is configured to perform spatiotemporal alignment based on a unified time reference to acquire multi-dimensional data aligned with timestamps. The unified time reference is the GNSS pulse-per-second signal.

[0151] In one possible implementation, the extraction module 701 specifically includes:

[0152] Based on the IMU motion parameters, the lidar point cloud data is corrected through quaternion attitude settlement to obtain the corrected lidar point cloud data;

[0153] Perform ground segmentation on the corrected LiDAR point cloud data to obtain LiDAR point cloud features;

[0154] A lightweight CNN is used to extract the vertex coordinates of traffic signs from camera image data to obtain camera image features.

[0155] In one possible implementation, building module 702 specifically includes:

[0156] Perform feature alignment based on the lidar point cloud features and camera image features to obtain 3D-2D corresponding point pairs;

[0157] Based on 3D-2D corresponding point pairs, the EPnP algorithm combined with Mahalanobis distance weighting is used to eliminate mismatched points and generate the initial extrinsic parameter matrix.

[0158] In one possible implementation, the updating module 703 specifically includes:

[0159] The Gauss-Newton method is used in the SE(3) manifold space to optimize the initialization extrinsic parameter matrix by minimizing the pre-set joint calibration objective function;

[0160] The vehicle kinematic model is combined with the wheel speed signal obtained during vehicle driving, and the extended Kalman filter is used to correct the accumulated posture error.

[0161] In one possible implementation, the first compensation module 704 specifically includes:

[0162] linearly adjust the translation vector based on the temperature parameter;

[0163] Modify the rotation matrix based on the vibration parameters.

[0164] In one possible implementation, the calibration device 700 for the intelligent driving sensor further includes:

[0165] Verification module 707, used to perform cross-validation based on millimeter-wave radar detection data and calibrated fusion perception results;

[0166] The recalibration module 708 is configured to trigger dynamic recalibration if the verification result does not meet a preset condition threshold.

[0167] The calibration device of the intelligent driving sensor provided in this embodiment can execute the calibration method of the intelligent driving sensor provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0168] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes: a storage unit 801, a processing unit 802;

[0169] The storage unit 801 stores computer-executable instructions;

[0170] The processing unit 802 executes the computer-executable instructions stored in the storage unit 801 , so that the processing unit 802 performs the above-mentioned calibration method for the intelligent driving sensor.

[0171] Optionally, the electronic device further includes a communication component 803 , wherein the processing unit 802 , the storage unit 801 and the communication component 803 are connected via a bus 804 .

[0172] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processing unit may be a microprocessor, or the processor may be any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0173] The storage unit may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0174] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0175] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0176] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0177] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0179] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0180] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0183] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A calibration method for an intelligent driving sensor, characterized by comprising: Extracting dynamic environment features based on pre-acquired multi-dimensional data, wherein the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters; Based on the dynamic environment characteristics, construct an initial external parameter matrix; Combining the vehicle kinematic model and the Lie group optimization algorithm, iteratively updating the initial extrinsic parameter matrix; The updated initial extrinsic parameter matrix is ​​compensated according to the pre-acquired temperature parameters and vibration parameters to obtain the final extrinsic parameter matrix.

2. The method according to claim 1, characterized in that The method further comprises: Compensate for the timing deviation of different sensors through the IMU pre-integration algorithm; Spatiotemporal alignment is performed based on a unified time reference to obtain the multi-dimensional data with time stamp alignment, wherein the unified time reference is a GNSS second pulse signal.

3. The method according to claim 1, characterized in that The dynamic environment features include lidar point cloud features and camera image features. The extraction of dynamic environment features based on pre-acquired multi-dimensional data includes: Based on the IMU motion parameters, the lidar point cloud data is corrected by quaternion attitude settlement to obtain corrected lidar point cloud data; The corrected lidar point cloud data is subjected to ground segmentation to obtain the lidar point cloud features; and the camera image data is subjected to a lightweight CNN to extract the vertex coordinates of the traffic sign to obtain the camera image features.

4. The method according to claim 3, characterized in that The constructing of an initial external parameter matrix based on the dynamic environment characteristics includes: Perform feature alignment based on the laser radar point cloud features and the camera image features to obtain 3D-2D corresponding point pairs; Based on the 3D-2D corresponding point pairs, the EPnP algorithm is used in combination with Mahalanobis distance weighting to eliminate mismatched points and generate the initial extrinsic parameter matrix.

5. The method according to claim 1, wherein The iterative updating of the initial extrinsic parameter matrix by combining the vehicle kinematic model and the Lie group optimization algorithm includes: The Gauss-Newton method is used in the SE(3) manifold space to optimize the initialized extrinsic parameter matrix by minimizing a pre-set joint calibration objective function; The vehicle kinematic model is combined with the wheel speed signal obtained during vehicle driving, and the accumulated posture error is corrected using an extended Kalman filter.

6. The method according to claim 1, characterized in that The extrinsic parameter matrix includes a rotation matrix and a translation vector, and the compensating the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters includes: linearly adjusting the translation vector based on the temperature parameter; The rotation matrix is ​​modified based on the vibration parameters.

7. The method according to claim 1, characterized in that The method further comprises: Perform cross-validation based on millimeter-wave radar detection data and calibrated fusion perception results; If the verification result does not meet the preset condition threshold, dynamic recalibration is triggered.

8. A calibration device for an intelligent driving sensor, characterized in that: include: An extraction module is used to extract dynamic environment features based on pre-acquired multi-dimensional data, wherein the multi-dimensional data includes lidar point cloud data, camera image data, and IMU motion parameters; A construction module, configured to construct an initial external parameter matrix based on the dynamic environment characteristics; An updating module, configured to iteratively update the initial extrinsic parameter matrix by combining a vehicle kinematic model and a Lie group optimization algorithm; The first compensation module is used to compensate the updated initial extrinsic parameter matrix according to the pre-acquired temperature parameters and vibration parameters to obtain a final extrinsic parameter matrix.

9. An electronic device, characterized in that: include: Storage unit, processing unit; The storage unit stores computer-executable instructions; The processing unit executes the computer-executable instructions stored in the storage unit, so that the processing unit performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images