Adaptive optimization system and method for laser radar and multi-camera external parameter calibration

Abstract

The invention relates to a self-adaptive optimization system and method for laser radar and multi-camera external parameter calibration, which belongs to the technical field of high-precision calibration and comprises a system initialization module, a system control module, a data acquisition module, a data preprocessing module, an initial calibration module, a multi-frame optimization module, an online compensation module and a calibration precision monitoring module. And the modules are in data transmission connection. According to the adaptive optimization system and method for laser radar and multi-camera external parameter calibration, the external parameter matrix can be dynamically adjusted according to the actual environment, the calibration precision is ensured to be stable all the time, and therefore the operation requirement of an automatic driving vehicle in the complex dynamic environment is met; the external parameter error is iteratively corrected through the nonlinear optimization algorithm, the calibration robustness is improved, the calibration accuracy is remarkably improved, and the calibration error caused by factors such as sensor noise and data incompleteness can be effectively reduced through constraint and optimization of multiple frames of data.

Description

Technical Field

[0001] The present invention relates to the technical field of high-precision calibration, and particularly to an adaptive optimization system and method for external parameter calibration of a lidar and multiple cameras. Background Art

[0002] With the rapid development of technology and the remarkable breakthroughs in artificial intelligence technology, autonomous driving technology has gradually become one of the core driving forces for the innovation of the global automotive industry. This technology not only represents the future development direction of transportation but also is of great significance for enhancing road safety and reducing traffic accidents. An autonomous driving system requires extremely accurate environmental perception capabilities to be able to identify road conditions, obstacles, pedestrians, other vehicles, and various traffic participants in real time and make rapid and accurate responses accordingly to ensure the safety and smoothness of the driving process. To this end, modern autonomous vehicles are usually equipped with a series of advanced sensors, including but not limited to lidar (LiDAR), high-definition cameras, millimeter-wave radars, etc. These sensors each perform their own functions and work together to provide the vehicle with three-dimensional spatial information and high-quality image information of the surrounding environment, thereby constructing a detailed and dynamically updated environmental model.

[0003] However, existing methods for external parameter calibration of sensors have several limitations: First, most traditional calibration methods are suitable for operation under static conditions or in specific environments. This limitation makes it difficult for them to adapt to the challenges faced by autonomous vehicles in complex and changing actual operating environments and unable to provide continuously stable, accurate, and reliable environmental perception data. Second, due to inaccurate external parameter calibration between different sensors, the consistency of multi-source sensor data in the spatial coordinate system and time axis is severely affected. This not only weakens the effect of data fusion but also may lead to distorted information after fusion, unable to accurately reflect the actual environmental situation. More seriously, the calibration error will be transmitted to the subsequent multi-sensor fusion system, directly affecting the positioning accuracy and target detection performance of the system, having a negative impact on the effectiveness of the entire autonomous driving system, and even threatening driving safety. Finally, existing methods generally lack the dynamic adaptation ability to the continuously changing external environment during the vehicle operation process and cannot achieve automatic adjustment and optimization based on real-time data. This results in a disconnection between the calibration result and the actual operating state of the vehicle, reducing the stability and reliability of the system.

[0004] Therefore, developing an adaptive optimization calibration method that can adapt to complex dynamic environments is of great significance for improving the performance and safety of autonomous driving systems. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the present invention provides an adaptive optimization system and method for calibrating the external parameters of a lidar and multiple cameras, which has the advantages of dynamically adjusting the external parameter matrix according to the actual environment, and solves the problem that traditional calibration technologies are difficult to adapt to dynamic environments.

[0006] To achieve the above object, the present invention provides the following technical solution: An adaptive optimization system for calibrating the external parameters of a lidar and multiple cameras, comprising: a system initialization module, a system control module, a data acquisition module, a data preprocessing module, an initial calibration module, a multi-frame optimization module, an online compensation module, and a calibration accuracy monitoring module, and data transmission connections are established between the modules;

[0007] Among them,

[0008] System initialization module: Start the system, load the configuration file, initialize the hardware device interface and preprocessing algorithm parameters, and prepare for the subsequent calibration process;

[0009] System control module: Control the execution of the entire calibration process, monitor the system operation status in real time, perform fault diagnosis, and provide a user interaction interface;

[0010] Data acquisition module: Collect lidar point cloud data, camera image data, and RTK and IMU data, and provide raw data for subsequent data processing;

[0011] Data preprocessing module: Preprocess the collected raw data, including point cloud denoising, data conversion, image processing, positioning data verification, and filtering;

[0012] Initial calibration module: Extract feature points from the preprocessed data, perform rough matching, calculate the initial external parameter matrix, and establish the spatial transformation relationship between the lidar and multiple cameras;

[0013] Multi-frame optimization module: Based on the motion consistency constraint of continuous frames, iteratively correct the external parameter error through a non-linear optimization algorithm;

[0014] Online compensation module: Real-time monitor the environmental light and vibration interference, and perform real-time compensation on the online collected data based on the established model;

[0015] Calibration accuracy monitoring module: Calculate and evaluate the accuracy of the current calibration result, and issue an alarm and prompt message when the calibration accuracy is lower than the set threshold.

[0016] Furthermore, the system initialization module includes a system startup configuration unit, a hardware initialization unit, and a preprocessing algorithm unit, specifically as follows:

[0017] System startup configuration unit: Responsible for the startup configuration of the system, including but not limited to loading necessary configuration files and setting initial parameters;

[0018] Hardware Initialization Unit: Used for the initialization of hardware devices;

[0019] Preprocessing Algorithm Unit: Sets parameters and algorithms for data preprocessing to prepare for subsequent data processing stages.

[0020] Furthermore, the system control module includes a calibration process control unit, a system status monitoring unit, a fault diagnosis unit, and a user interaction unit, specifically as follows:

[0021] Calibration Process Control Unit: Controls the execution of the entire calibration process to ensure that each unit operates in accordance with the predetermined sequence and logic;

[0022] System Status Monitoring Unit: Monitors the operating status of the system in real time, including but not limited to hardware interface status and data processing progress;

[0023] Fault Diagnosis Unit: Diagnoses the occurring faults, automatically recovers, and provides fault handling suggestions;

[0024] User Interaction Unit: Provides an interactive interface for convenient user operation.

[0025] Furthermore, the data acquisition module includes a lidar unit, a camera device unit, an RTK unit, and an IMU unit, specifically as follows:

[0026] Lidar Unit: Obtains three-dimensional point cloud data of the surrounding environment and provides accurate distance information;

[0027] Camera Device Unit: Captures image information in the environment to provide basic data for visual processing;

[0028] RTK Unit: Provides position information with centimeter-level accuracy;

[0029] IMU Unit: Monitors the acceleration and angular velocity motion states of the vehicle to assist in positioning and attitude estimation.

[0030] Furthermore, the data preprocessing module includes a point cloud denoising unit, a point cloud data conversion unit, an image data processing unit, a positioning data verification unit, and a positioning data filtering unit, specifically as follows:

[0031] Point Cloud Denoising Unit: Removes noise points from the lidar point cloud data;

[0032] Point Cloud Data Conversion Unit: Converts the denoised point cloud data into the format required by the system;

[0033] Image Data Processing Unit: Performs preprocessing on the image data collected by the camera;

[0034] Positioning data verification unit: Verifies the positioning data collected by the RTK unit and the IMU unit;

[0035] Positioning data filtering unit: Filters the verified positioning data.

[0036] Furthermore, the initial calibration module includes a feature point extraction unit, a feature point rough matching unit, and an external parameter matrix calculation unit, specifically as follows:

[0037] Feature point extraction unit: Extracts feature points from lidar point cloud data and camera image data;

[0038] Feature point rough matching unit: Performs a preliminary match on the feature points extracted from the point cloud and the image;

[0039] External parameter matrix calculation unit: Calculates the initial external parameter matrix between the lidar and the camera based on the results of the feature point rough matching.

[0040] Furthermore, the multi-frame optimization module includes a frame data caching unit, a constraint calculation unit, a non-linear optimization unit, an error iterative correction unit, and an evaluation and feedback unit, specifically as follows:

[0041] Frame data caching unit: Continuously collects and caches lidar point cloud and multi-camera image data to form a continuous frame sequence;

[0042] Constraint calculation unit: Performs motion consistency constraint calculations on the data between consecutive frames;

[0043] Non-linear optimization unit: Optimizes the initial external parameter matrix using a non-linear optimization algorithm;

[0044] Error iterative correction unit: Iteratively corrects the optimized external parameter matrix;

[0045] Evaluation and feedback unit: Evaluates the optimized external parameter matrix and makes feedback adjustments according to the evaluation results.

[0046] Furthermore, the online compensation module includes an environmental light monitoring unit, a vibration interference detection unit, a light change model unit, a vibration interference model unit, and a real-time external parameter drift compensation unit, specifically as follows:

[0047] Environmental light monitoring unit: Monitors the changes in environmental light conditions in real time;

[0048] Vibration interference detection unit: Detects the vibration interference generated during vehicle driving;

[0049] Light change model unit: Establishes a light change model based on the results of environmental light monitoring;

[0050] Vibration interference model unit: Establish a vibration interference model based on vibration interference detection data;

[0051] Real-time extrinsic parameter drift compensation unit: Combine the results of the light change model and the vibration interference model, and calculate and compensate the drift of the extrinsic parameter matrix in real time.

[0052] Furthermore, the calibration accuracy monitoring module includes a calibration accuracy calculation unit, an accuracy threshold setting unit, an anomaly detection unit, and an alarm and prompt unit, specifically as follows:

[0053] Calibration accuracy calculation unit: Calculate and evaluate the accuracy of the current calibration result;

[0054] Accuracy threshold setting unit: Set the threshold of the calibration accuracy according to the requirements of the application scenario;

[0055] Anomaly detection unit: Detect abnormal situations that occur during the calibration process;

[0056] Alarm and prompt unit: Send alarm and prompt messages when the calibration accuracy is lower than the set threshold.

[0057] This application also proposes an adaptive optimization method for the extrinsic calibration of lidar and multi-cameras, including the following steps:

[0058] S1 Preparation and initialization: Ensure that the lidar, multi-cameras, and high-precision RTK / IMU devices are installed and calibrated, and are in normal working condition. The system initialization module loads the configuration file to complete the hardware initialization and preprocessing algorithm settings;

[0059] S2 Initial calibration: Coarsely match the calibration point cloud in the vehicle body coordinate system with the image feature points, and calculate the initial extrinsic parameter matrix through feature point extraction and matching;

[0060] S3 Multi-frame optimization: Based on the motion consistency constraint of consecutive frames, iteratively correct the extrinsic parameter error through the Levenberg-Marquardt algorithm to improve the calibration accuracy;

[0061] S4 Environmental light and vibration interference modeling: Analyze the changes of image feature points under different light conditions to establish a light change model; at the same time, analyze the vibration situation during vehicle driving to establish a vibration interference model;

[0062] S5 Online real-time compensation: According to the light change model and the vibration interference model, calculate the compensation value of the extrinsic parameter in real time, adjust the extrinsic parameter matrix, and compensate for the extrinsic parameter drift caused by light change and vibration interference;

[0063] S6 Evaluation and feedback adjustment: According to the performance evaluation results, perform feedback adjustment on the initial calibration, multi-frame optimization, and online compensation steps to continuously optimize the extrinsic calibration method.

[0064] Compared with the prior art, the technical solution of the present application has the following beneficial effects:

[0065] 1. The adaptive optimization system and method for the extrinsic parameter calibration of the lidar and multi-cameras can dynamically adjust the extrinsic parameter matrix according to the actual environment, ensuring that the calibration accuracy is always stable, so as to meet the operation requirements of autonomous driving vehicles in complex dynamic environments. Secondly, based on the motion consistency constraint of consecutive frames, the extrinsic parameter error is iteratively corrected through a non-linear optimization algorithm, which not only improves the robustness of the calibration, but also significantly enhances the accuracy of the calibration. Through the constraint and optimization of multi-frame data, the calibration error caused by factors such as sensor noise and incomplete data can be effectively reduced, ensuring the reliability of the calibration results. The environmental light change and vibration interference models are introduced, which can monitor and compensate the influence of these environmental factors on the extrinsic parameter calibration in real time. By establishing the light change model and vibration interference model, the compensation value of the extrinsic parameter can be calculated in real time, the extrinsic parameter matrix can be adjusted, and the extrinsic parameter drift caused by light change and vibration interference can be compensated, ensuring the continuous effectiveness of the calibration results.

[0066] 2. The adaptive optimization system and method for the extrinsic parameter calibration of the lidar and multi-cameras can significantly improve the positioning and detection accuracy of the multi-sensor fusion perception system through an online compensation mechanism. The online compensation mechanism can not only adjust the extrinsic parameter matrix in real time, but also dynamically optimize the calibration parameters according to environmental changes, ensuring the accuracy and consistency of sensor data, so that the autonomous driving vehicle can always maintain high-efficiency environmental perception ability in complex dynamic environments, thereby improving the overall performance and safety of the system. At the same time, by introducing a calibration accuracy monitoring module and triggering an alarm when the calibration accuracy is lower than the preset threshold to prompt the user for further inspection and adjustment, this mechanism not only improves the stability of the system, but also ensures the reliability of the calibration results, providing a strong guarantee for the safe operation of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the system module flow structure of the present invention;

[0068] Figure 2 It is a schematic diagram of the flow of the adaptive optimization method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0070] Please refer to Figure 1-2, the adaptive optimization system for the extrinsic calibration of lidar and multi-cameras in this embodiment includes: a system initialization module, a system control module, a data acquisition module, a data preprocessing module, an initial calibration module, a multi-frame optimization module, an online compensation module, and a calibration accuracy monitoring module, and the data is transmitted and connected between the modules;

[0071] Among them,

[0072] System initialization module: Start the system, load the configuration file, initialize the hardware device interface and preprocessing algorithm parameters, and prepare for the subsequent calibration process;

[0073] System control module: Control the execution of the entire calibration process, monitor the system operation status in real time, perform fault diagnosis, and provide a user interaction interface;

[0074] Data acquisition module: Collect lidar point cloud data, camera image data, and RTK and IMU data, and provide the raw data for subsequent data processing;

[0075] Data preprocessing module: Preprocess the collected raw data, including point cloud denoising, data conversion, image processing, positioning data verification, and filtering;

[0076] Initial calibration module: Extract feature points from the preprocessed data, perform rough matching, calculate the initial extrinsic parameter matrix, and establish the spatial transformation relationship between the lidar and the multi-cameras;

[0077] Multi-frame optimization module: Based on the motion consistency constraint of consecutive frames, iteratively correct the extrinsic parameter error through a non-linear optimization algorithm;

[0078] Online compensation module: Monitor the ambient light and vibration interference in real time, and perform real-time compensation on the online collected data based on the established model.

[0079] Calibration accuracy monitoring module: Calculate and evaluate the accuracy of the current calibration result, and issue an alarm and prompt message when the calibration accuracy is lower than the set threshold.

[0080] It should be noted that the lidar and multi-camera extrinsic parameter calibration adaptive optimization system of this embodiment includes modules such as system initialization, control, data acquisition, preprocessing, initial calibration, multi-frame optimization, and online compensation. The data of each module is connected. The system initialization module is responsible for starting, loading configurations, and initializing hardware and algorithm parameters; the system control module controls the calibration process, monitors the status, diagnoses faults, and provides an interactive interface; the data acquisition module collects lidar, camera, RTK, and IMU data; the data preprocessing module denoises, converts, and validates the data; the initial calibration module extracts feature points, performs rough matching, and calculates the initial extrinsic parameter matrix; the multi-frame optimization module optimizes the extrinsic parameters using motion consistency constraints; the online compensation module monitors environmental changes in real time and compensates the data to ensure calibration accuracy and stability.

[0081] In this embodiment, the system initialization module includes a system startup configuration unit, a hardware initialization unit, and a preprocessing algorithm unit, as follows:

[0082] System startup configuration unit: Responsible for the startup configuration of the system, including but not limited to loading necessary configuration files and setting initial parameters;

[0083] Hardware initialization unit: Used for the initialization of hardware devices;

[0084] Preprocessing algorithm unit: Sets the parameters and algorithms for data preprocessing to prepare for the subsequent data processing stage.

[0085] It should be noted that the system initialization module in this embodiment, through the close cooperation of the system startup configuration unit, the hardware initialization unit, and the preprocessing algorithm unit, ensures that the system can start efficiently and accurately and be ready. The system startup configuration unit is responsible for loading necessary configuration files and setting initial parameters, laying the foundation for the smooth operation of the system; the hardware initialization unit ensures that all sensors and other hardware devices are correctly initialized, guaranteeing the reliability of data acquisition; the preprocessing algorithm unit sets the parameters and algorithms for data preprocessing, optimizing the subsequent data processing process. This systematic initialization mechanism not only improves the stability and robustness of the system but also provides a solid guarantee for subsequent precise data processing and extrinsic parameter calibration, thus enhancing the overall performance and response speed of the autonomous driving system.

[0086] In this embodiment, the system control module includes a calibration process control unit, a system status monitoring unit, a fault diagnosis unit, and a user interaction unit, as follows:

[0087] Calibration process control unit: Controls the execution of the entire calibration process to ensure that each unit operates in a predetermined order and logic;

[0088] System status monitoring unit: Monitors the operating status of the system in real time, including but not limited to the status of hardware interfaces and the progress of data processing;

[0089] Fault diagnosis unit: Diagnose the occurring faults, automatically recover, and provide suggestions for fault handling;

[0090] User interaction unit: Provide an interactive interface for convenient user operation.

[0091] It should be noted that the system control module in this embodiment consists of a calibration process control unit, a system status monitoring unit, a fault diagnosis unit, and a user interaction unit, which realizes the comprehensive management and optimization of the calibration process. The calibration process control unit ensures that each step is efficiently executed according to the predetermined sequence and logic; the system status monitoring unit monitors the status of the hardware interface and the progress of data processing in real time to ensure the stable operation of the system; the fault diagnosis unit can quickly diagnose and attempt to automatically recover the fault, and at the same time provide effective handling suggestions; the user interaction unit provides a convenient operation interface for users to configure, monitor, and view the results. Through this structured and intelligent design, the system not only improves the convenience and flexibility of operation, but also enhances the overall reliability and user experience, ensuring that the autonomous driving system can maintain the best performance under various conditions.

[0092] In this embodiment, the data acquisition module includes a lidar unit, a camera device unit, an RTK unit, and an IMU unit, which are specifically as follows:

[0093] Lidar unit: Obtain the three-dimensional point cloud data of the surrounding environment and provide accurate distance information;

[0094] Camera device unit: Capture the image information in the environment and provide basic data for visual processing;

[0095] RTK unit: Provide position information with centimeter-level accuracy;

[0096] IMU unit: Monitor the acceleration and angular velocity motion states of the vehicle and assist in positioning and attitude estimation.

[0097] It should be noted that the data acquisition module in this embodiment realizes the comprehensive and accurate data acquisition of the vehicle's surrounding environment through the collaborative work of the lidar unit, the camera device unit, the RTK unit, and the IMU unit. The lidar unit obtains the three-dimensional point cloud data and provides high-precision distance information; the camera device unit captures the environmental images to lay the foundation for visual processing; the RTK unit provides position information with centimeter-level accuracy to ensure precise positioning; the IMU unit monitors the acceleration and angular velocity of the vehicle to assist in positioning and attitude estimation. This method of multi-sensor fusion not only improves the richness and accuracy of the data, but also enhances the perception ability and robustness of the system in complex dynamic environments, providing high-quality input for subsequent external parameter calibration and multi-sensor data fusion, thus significantly improving the overall performance and reliability of the autonomous driving system.

[0098] In this embodiment, the data preprocessing module includes a point cloud denoising unit, a point cloud data conversion unit, an image data processing unit, a positioning data verification unit, and a positioning data filtering unit, which are specifically as follows:

[0099] Point cloud denoising unit: Removes the noise points in the lidar point cloud data;

[0100] Point cloud data conversion unit: Converts the denoised point cloud data into the format required by the system;

[0101] Image data processing unit: Performs preprocessing on the image data collected by the camera;

[0102] Positioning data verification unit: Verifies the positioning data collected by the RTK unit and the IMU unit;

[0103] Positioning data filtering unit: Performs filtering processing on the verified positioning data.

[0104] It should be noted that the data preprocessing module in this embodiment effectively improves the quality and usability of the original data through the collaborative work of the point cloud denoising unit, the point cloud data conversion unit, the image data processing unit, the positioning data verification unit, and the positioning data filtering unit. The point cloud denoising unit removes the noise in the lidar data to ensure pure data; the point cloud data conversion unit converts the denoised point cloud into the format required by the system for subsequent processing; the image data processing unit preprocesses the camera images to optimize visual information; the positioning data verification unit ensures the accuracy of the positioning data provided by the RTK and IMU; the positioning data filtering unit further filters out the interference in the positioning data to ensure the stability and reliability of the finally input data to the system. This comprehensive processing flow significantly enhances the accuracy and consistency of the data, providing a solid foundation for subsequent extrinsic parameter calibration and multi-sensor fusion.

[0105] In this embodiment, the initial calibration module includes a feature point extraction unit, a feature point rough matching unit, and an extrinsic parameter matrix calculation unit, which are specifically as follows:

[0106] Feature point extraction unit: Extracts feature points from the lidar point cloud data and the camera image data;

[0107] Feature point rough matching unit: Performs preliminary matching on the feature points extracted from the point cloud and the image;

[0108] Extrinsic parameter matrix calculation unit: Calculates the initial extrinsic parameter matrix between the lidar and the camera based on the results of the feature point rough matching.

[0109] It should be noted that in this embodiment, the initial calibration module realizes efficient and accurate preliminary extrinsic parameter calibration through the close cooperation of the feature point extraction unit, the feature point rough matching unit, and the extrinsic parameter matrix calculation unit. The feature point extraction unit accurately extracts feature points from the lidar point cloud data and the camera images, laying a foundation for subsequent processing; the feature point rough matching unit preliminarily matches these feature points to establish a corresponding relationship between the lidar and the camera; the extrinsic parameter matrix calculation unit calculates the initial extrinsic parameter matrix between the two based on the matching results, establishing an accurate spatial transformation relationship. This process not only improves the accuracy and reliability of calibration but also ensures that the system can quickly respond and adapt to different environmental conditions, providing a solid foundation for subsequent multi-frame optimization and online compensation.

[0110] In this embodiment, the multi-frame optimization module includes a frame data caching unit, a constraint calculation unit, a non-linear optimization unit, an error iterative correction unit, and an evaluation and feedback unit, which are specifically as follows:

[0111] Frame data caching unit: Continuously collect and cache lidar point cloud and multi-camera image data to form a continuous frame sequence;

[0112] Constraint calculation unit: Perform motion consistency constraint calculations on the data between consecutive frames;

[0113] Non-linear optimization unit: Optimize the initial extrinsic parameter matrix using a non-linear optimization algorithm;

[0114] Error iterative correction unit: Iteratively correct the optimized extrinsic parameter matrix;

[0115] Evaluation and feedback unit: Evaluate the optimized extrinsic parameter matrix and perform feedback adjustment according to the evaluation results.

[0116] It should be noted that in this embodiment, the multi-frame optimization module realizes high-precision optimization of the extrinsic parameter matrix through the collaborative work of the frame data caching unit, the constraint calculation unit, the non-linear optimization unit, the error iterative correction unit, and the evaluation and feedback unit. The frame data caching unit continuously collects and caches lidar point cloud and multi-camera image data to form a continuous frame sequence; the constraint calculation unit calculates the motion consistency constraints based on the inter-frame data to ensure the consistency and stability of the calibration results; the non-linear optimization unit uses an advanced algorithm to finely optimize the initial extrinsic parameter matrix; the error iterative correction unit further iteratively corrects the optimized extrinsic parameter matrix to gradually reduce errors; the evaluation and feedback unit comprehensively evaluates the calibration results and performs necessary feedback adjustment according to the evaluation results to ensure the high precision and robustness of the system. This comprehensive process significantly improves the accuracy and reliability of extrinsic parameter calibration, enabling the system to operate stably and efficiently in complex dynamic environments.

[0117] In this embodiment, the online compensation module includes an environmental light monitoring unit, a vibration interference detection unit, a light change model unit, a vibration interference model unit, and a real-time extrinsic parameter drift compensation unit, as follows:

[0118] Environmental light monitoring unit: Continuously monitor the changes in environmental light conditions;

[0119] Vibration interference detection unit: Detect the vibration interference generated during vehicle driving;

[0120] Light change model unit: Establish a light change model based on the results of environmental light monitoring;

[0121] Vibration interference model unit: Establish a vibration interference model according to the vibration interference detection data;

[0122] Real-time extrinsic parameter drift compensation unit: Combine the results of the light change model and the vibration interference model to calculate and compensate the drift of the extrinsic parameter matrix in real time.

[0123] It should be noted that the online compensation module in this embodiment realizes the real-time and accurate compensation of the drift of the extrinsic parameter matrix through the close cooperation of the environmental light monitoring unit, the vibration interference detection unit, the light change model unit, the vibration interference model unit, and the real-time extrinsic parameter drift compensation unit. The environmental light monitoring unit continuously monitors the changes in light conditions, and the vibration interference detection unit captures the vibration interference during vehicle driving; the light change model unit and the vibration interference model unit respectively establish corresponding mathematical models based on the monitoring data to describe the impact of environmental changes on sensor data; the real-time extrinsic parameter drift compensation unit combines the results of these two models to dynamically adjust and compensate the drift of the extrinsic parameter matrix, ensuring that the calibration results always remain accurate and stable in the complex and changeable actual operating environment. This adaptive compensation mechanism significantly improves the robustness and reliability of the system and effectively guarantees the high-precision perception ability of autonomous driving vehicles in various environments.

[0124] In this embodiment, the calibration accuracy monitoring module includes a calibration accuracy calculation unit, an accuracy threshold setting unit, an anomaly detection unit, and an alarm and prompt unit, as follows:

[0125] Calibration accuracy calculation unit: Calculate and evaluate the accuracy of the current calibration result;

[0126] Accuracy threshold setting unit: Set the threshold of the calibration accuracy according to the requirements of the application scenario;

[0127] Anomaly detection unit: Detect the abnormal situations that occur during the calibration process;

[0128] Alarm and prompt unit: Send alarm and prompt information when the calibration accuracy is lower than the set threshold.

[0129] It should be noted that the calibration accuracy monitoring module in this embodiment ensures the high accuracy and reliability of the extrinsic parameter calibration process through the collaborative work of the calibration accuracy calculation unit, the accuracy threshold setting unit, the anomaly detection unit, and the alarm and prompt unit. The calibration accuracy calculation unit is responsible for calculating and evaluating the accuracy of the current calibration result; the accuracy threshold setting unit sets a reasonable accuracy standard according to the requirements of the specific application scenario; the anomaly detection unit monitors in real time the possible deviations or anomalies that may occur during the calibration process; and the alarm and prompt unit issues an alarm and prompt message in a timely manner when the calibration accuracy is lower than the preset threshold, reminding the user to make necessary adjustments. This systematic process not only improves the accuracy and consistency of the calibration result, but also enhances the stability and security of the system, ensuring that the autonomous driving vehicle can maintain the best performance under various conditions.

[0130] This application also proposes an adaptive optimization method for the extrinsic parameter calibration of lidar and multi-cameras, including the following steps:

[0131] S1 Preparation and initialization: Ensure that the lidar, multi-cameras, and high-precision RTK / IMU devices are installed and calibrated, and are in a normal working state. The system initialization module loads the configuration file to complete the hardware initialization and preprocessing algorithm settings;

[0132] S2 Initial calibration: Coarsely match the calibration point cloud in the vehicle body coordinate system with the image feature points, and calculate the initial extrinsic parameter matrix through feature point extraction and matching;

[0133] S3 Multi-frame optimization: Based on the motion consistency constraint of consecutive frames, iteratively correct the extrinsic parameter error through the Levenberg-Marquardt algorithm to improve the calibration accuracy;

[0134] S4 Modeling of environmental light and vibration interference: Analyze the changes in image feature points under different lighting conditions to establish a lighting change model; at the same time, analyze the vibration conditions during vehicle driving to establish a vibration interference model;

[0135] S5 Online real-time compensation: According to the lighting change model and the vibration interference model, calculate the compensation value of the extrinsic parameter in real time, adjust the extrinsic parameter matrix, and compensate for the extrinsic parameter drift caused by lighting changes and vibration interference;

[0136] S6 Evaluation and feedback adjustment: According to the performance evaluation results, perform feedback adjustment on the initial calibration, multi-frame optimization, and online compensation steps to continuously optimize the extrinsic parameter calibration method.

[0137] It should be noted that the adaptive optimization method for laser radar and multi-camera extrinsic calibration proposed in this application realizes high-precision and dynamically adaptive extrinsic calibration through six steps. First, in the preparation and initialization stage (S1), the system ensures that all hardware devices are installed and calibrated, and loads the configuration file and sets the preprocessing algorithm; then in the initial calibration stage (S2), the calibration point cloud in the vehicle coordinate system is roughly matched with the image feature points to calculate the initial extrinsic parameter matrix; then, in the multi-frame optimization stage (S3), based on the motion consistency constraint of continuous frames, the Levenberg-Marquardt algorithm is used to iteratively correct the extrinsic parameter error to obtain the optimal calibration result. High calibration accuracy; then, in the environmental illumination and vibration interference modeling stage (S4), an illumination change model and a vibration interference model are established respectively to describe the data changes under different conditions; followed by an online real-time compensation stage (S5) to adjust the external parameter matrix in real time according to the above model to compensate for the drift caused by illumination change and vibration interference; finally, in the evaluation and feedback adjustment stage (S6), feedback adjustment is performed on each step based on the performance evaluation results to continuously optimize the entire calibration process. This method not only significantly improves the accuracy and robustness of the calibration, but also enhances the system's adaptability in complex dynamic environments, providing more reliable environmental perception support for autonomous driving vehicles.

[0138] The advantages of the above embodiment are as follows:

[0139] Through a series of innovative technologies, efficient fusion of lidar point cloud data and multi-camera image data is achieved, and combined with high-precision RTK / I MU positioning information, millimeter-level extrinsic parameter calibration accuracy is achieved. Compared with traditional methods, this application adopts an adaptive optimization strategy, which can dynamically adjust the extrinsic parameter matrix according to real-time environmental conditions to ensure the stability of the calibration results in complex and changeable autonomous driving environments. This flexibility and accuracy are difficult to achieve with traditional static or specific calibration methods.

[0140] Specifically, the initial extrinsic parameter matrix is iteratively corrected using a nonlinear optimization algorithm based on motion consistency constraints between consecutive frames, thereby significantly improving the robustness and accuracy of the calibration. This method not only takes into account the calibration requirements under static conditions, but also effectively responds to various challenges encountered during vehicle driving, such as interference factors such as lighting changes and road vibration. In particular, a mathematical model specifically designed for ambient lighting changes and vibration interference is introduced, which enables the system to monitor and compensate for the extrinsic parameter drift caused by these factors in real time during operation, thereby ensuring the continued effectiveness of the calibration results.

[0141] In addition, a complete set of online compensation mechanisms is designed. This mechanism can not only detect changes in the external environment but also respond quickly. By adjusting the external parameter matrix, it can offset adverse effects, thereby greatly improving the positioning and detection accuracy of the multi-sensor fusion perception system. This improvement significantly enhances the environmental perception ability of autonomous vehicles, enabling them to maintain a high level of safety and reliability even under strong light changes or poor road conditions.

[0142] In summary, this application is not just a simple upgrade of the calibration method but a comprehensive technological innovation. It solves multiple deficiencies in the prior art, such as the inability to adapt to dynamic environments and the lack of effective means to handle external interferences. Through the above series of measures, this application significantly improves the overall performance of the autonomous driving system, laying a solid foundation for the development of future intelligent transportation systems. This not only helps improve driving safety and reduce traffic accidents but also provides strong technical support for the popularization and application of autonomous driving technology. Therefore, this application is of great significance for promoting the progress of the autonomous driving field.

[0143] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0144] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention.

Claims

1. An adaptive optimization system for the extrinsic calibration of lidar and multi-cameras, characterized in that, Including: A system initialization module, a system control module, a data acquisition module, a data preprocessing module, an initial calibration module, a multi-frame optimization module, an online compensation module, and a calibration accuracy monitoring module, with data transmission connections between each module; Among them, System initialization module: Start the system, load the configuration file, initialize the hardware device interface and preprocessing algorithm parameters, and prepare for the subsequent calibration process; System control module: Control the execution of the entire calibration process, monitor the system operation status in real time, perform fault diagnosis, and provide a user interaction interface; Data acquisition module: Collect lidar point cloud data, camera image data, and RTK and IMU data, and provide raw data for subsequent data processing; Data preprocessing module: Preprocess the collected raw data, including point cloud denoising, data conversion, image processing, positioning data verification, and filtering; Initial calibration module: Extract feature points from the preprocessed data, perform rough matching, calculate the initial external parameter matrix, and establish the spatial transformation relationship between the lidar and multiple cameras; Multi-frame optimization module: Based on the motion consistency constraint of continuous frames, iteratively correct the external parameter error through a nonlinear optimization algorithm; Online compensation module: Real-time monitor the environmental light and vibration interference, and perform real-time compensation on the online collected data based on the established model Calibration accuracy monitoring module: Calculate and evaluate the accuracy of the current calibration result, and issue an alarm and prompt message when the calibration accuracy is lower than the set threshold.

2. The adaptive optimization system for calibrating the extrinsic parameters of a lidar and multiple cameras according to claim 1, wherein: The system initialization module includes a system startup configuration unit, a hardware initialization unit, and a preprocessing algorithm unit, specifically as follows: System startup configuration unit: Responsible for the startup configuration of the system, including but not limited to loading necessary configuration files and setting initial parameters; Hardware initialization unit: Used for the initialization of hardware devices; Preprocessing algorithm unit: Set the parameters and algorithms for data preprocessing, and prepare for the subsequent data processing stage.

3. The adaptive optimization system for the extrinsic parameter calibration of a lidar and multiple cameras according to claim 1, wherein: The system control module includes a calibration process control unit, a system status monitoring unit, a fault diagnosis unit, and a user interaction unit, specifically as follows: Calibration process control unit: Control the execution of the entire calibration process, and ensure that each unit operates according to the predetermined order and logic; System status monitoring unit: Real-time monitor the operation status of the system, including but not limited to the hardware interface status and data processing progress; Fault diagnosis unit: Diagnose the occurring faults, automatically recover, and provide fault handling suggestions; User interaction unit: Provide an interactive interface that is convenient for users to operate.

4. The adaptive optimization system for calibration of the extrinsic parameters between a lidar and multiple cameras according to claim 1, wherein: The data acquisition module includes a lidar unit, a camera device unit, an RTK unit, and an IMU unit, specifically as follows: Lidar unit: Obtain the three-dimensional point cloud data of the surrounding environment and provide accurate distance information; Camera device unit: Capture the image information in the environment and provide basic data for visual processing; RTK unit: Provide position information with centimeter-level accuracy; IMU unit: Monitor the acceleration and angular velocity motion states of the vehicle, and assist in positioning and attitude estimation.

5. The adaptive optimization system for the extrinsic parameter calibration of a lidar and multiple cameras according to claim 1, wherein: The data preprocessing module includes a point cloud denoising unit, a point cloud data conversion unit, an image data processing unit, a positioning data verification unit, and a positioning data filtering unit, specifically as follows: Point cloud denoising unit: Remove the noise points in the lidar point cloud data; Point cloud data conversion unit: Convert the denoised point cloud data into the format required by the system; Image data processing unit: Preprocess the image data collected by the camera; Positioning data verification unit: Verify the positioning data collected by the RTK unit and the IMU unit; Positioning data filtering unit: Filter the verified positioning data.

6. The adaptive optimization system for the extrinsic parameter calibration of a lidar and multiple cameras according to claim 1, characterized in that: The initial calibration module includes a feature point extraction unit, a feature point rough matching unit, and an external parameter matrix calculation unit, specifically as follows: Feature point extraction unit: Extract feature points from the lidar point cloud data and the camera image data; Feature point rough matching unit: Perform a preliminary match on the feature points extracted from the point cloud and the image; External parameter matrix calculation unit: Calculate the initial external parameter matrix between the lidar and the camera based on the result of the feature point rough matching; 7. The adaptive optimization system for the extrinsic parameter calibration of a lidar and multiple cameras according to claim 1, wherein: The multi-frame optimization module includes a frame data caching unit, a constraint calculation unit, a non-linear optimization unit, an error iterative correction unit, and an evaluation and feedback unit, specifically as follows: Frame data caching unit: Continuously collect and cache the lidar point cloud and multi-camera image data to form a continuous frame sequence; Constraint calculation unit: Perform a motion consistency constraint calculation on the data between consecutive frames; Non-linear optimization unit: Optimize the initial external parameter matrix using a non-linear optimization algorithm; Error iterative correction unit: Iteratively correct the optimized external parameter matrix; Evaluation and feedback unit: Evaluate the optimized external parameter matrix and perform feedback adjustment according to the evaluation result.

8. The adaptive optimization system for the extrinsic parameter calibration of a lidar and multiple cameras according to claim 1, wherein: The online compensation module includes an environmental light monitoring unit, a vibration interference detection unit, a light change model unit, a vibration interference model unit, and a real-time external parameter drift compensation unit, specifically as follows: Environmental light monitoring unit: Real-time monitor the change of the environmental light condition; Vibration interference detection unit: Detect the vibration interference generated during the vehicle driving; Light change model unit: Establish a light change model based on the result of the environmental light monitoring; Vibration interference model unit: Establish a vibration interference model according to the vibration interference detection data; Real-time external parameter drift compensation unit: Combine the results of the light change model and the vibration interference model to calculate and compensate the drift of the external parameter matrix in real time.

9. The adaptive optimization system for calibrating the extrinsic parameters of a lidar and multiple cameras according to claim 1, characterized in that: The calibration accuracy monitoring module includes a calibration accuracy calculation unit, a precision threshold setting unit, an anomaly detection unit, and an alarm and prompt unit, specifically as follows: Calibration accuracy calculation unit: Calculate and evaluate the accuracy of the current calibration result; Precision threshold setting unit: Set the threshold of the calibration accuracy according to the requirements of the application scenario; Anomaly detection unit: Detect the abnormal situation occurring during the calibration process; Alarm and prompt unit: Send an alarm and prompt message when the calibration accuracy is lower than the set threshold.

10. An adaptive optimization method for the extrinsic parameter calibration of lidar and multi-cameras, including the adaptive optimization system for the extrinsic parameter calibration of lidar and multi-cameras according to any one of claims 1 to 9, characterized in that Including the following steps: S1 Preparation and initialization: Ensure that the lidar, multi-camera, and high-precision RTK / IMU devices are installed and calibrated, and are in a normal working state. The system initialization module loads the configuration file to complete the hardware initialization and the preprocessing algorithm setting; S2 Initial calibration: Use the calibrated point cloud in the vehicle body coordinate system to perform a rough match with the image feature points, and calculate the initial external parameter matrix through feature point extraction and matching; S3 Multi-frame Optimization: Based on the motion consistency constraint of consecutive frames, the external parameter error is iteratively corrected by the Levenberg-Marquardt algorithm to improve the calibration accuracy; S4 Modeling of Ambient Light and Vibration Interference: Analyze the changes of image feature points under different lighting conditions to establish a lighting change model; at the same time, analyze the vibration conditions during vehicle driving to establish a vibration interference model; S5 Online Real-time Compensation: According to the lighting change model and the vibration interference model, calculate the compensation value of the external parameters in real time, adjust the external parameter matrix, and compensate for the external parameter drift caused by lighting changes and vibration interference; S6 Evaluation and Feedback Adjustment: According to the performance evaluation results, perform feedback adjustment on the initial calibration, multi-frame optimization, and online compensation steps to continuously optimize the external parameter calibration method.

Images