External parameter calibration method and device and electronic equipment

Abstract

The invention provides an external parameter calibration method and device and electronic equipment, and the method can obtain image data collected by a camera and point cloud data collected by a laser radar, and calculates a predicted external parameter calibration parameter between the camera and the laser radar based on the image data and the point cloud data. And calculating an external parameter error value based on the predicted external parameter calibration parameter and the current external parameter calibration parameter. And when the external parameter error value satisfies a correction condition, extracting a visual semantic feature of the target object in the image data and a point cloud semantic feature of the target object in the point cloud data, and calculating a matching error value of the visual semantic feature and the point cloud semantic feature. And constructing a graph optimization framework based on the matching error value and the external parameter error value, solving a target external parameter calibration parameter corresponding to the minimum matching error value based on the graph optimization framework, and updating the current external parameter calibration parameter into the target external parameter calibration parameter. According to the method, the external parameter calibration parameter change between the camera and the laser radar can be automatically monitored, and the external parameter calibration parameter can be automatically corrected.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an external parameter calibration method, device and electronic equipment. Background Art

[0002] With the development of autonomous driving technology, autonomous driving technology has increasingly higher requirements for the perception of the vehicle's surrounding environment. The intelligent driving data acquisition system is a key component for realizing advanced driving assistance systems (ADAS). The intelligent driving data acquisition system integrates multiple sensors, such as LiDAR, cameras, millimeter-wave radar (Radar), ultrasonic sensors, global positioning system (GPS), inertial measurement unit (IMU), etc., and collects environmental data during vehicle driving through multiple sensors to provide support for subsequent model training and simulation speed measurement.

[0003] In order to achieve effective multi-sensor data fusion, the relative position and posture relationship between each sensor must be determined. This process is called sensor calibration. The sensor calibration process is divided into internal parameter calibration and external parameter calibration. Internal parameter calibration is to determine the internal parameters of the sensor, such as the focal length and distortion coefficient of the camera. External parameter calibration is to determine the relative position and coordinate transformation relationship between sensors. In the actual application of the intelligent driving data acquisition system, environmental data from different road sections will be collected outdoors for a long time. The relative position changes of sensors caused by the diverse and complex environment or the vehicle's own movement will cause changes in the sensor's external parameters. If these changes cannot be detected and corrected in a timely and accurate manner, the accuracy of data acquisition and the effectiveness of subsequent model training and simulation testing will be reduced.

[0004] The sensor's external parameter calibration method can be manually calibrated using specific calibration facilities (such as a calibration plate) in a specific environment. However, this method is time-consuming and inefficient, and it is difficult to adapt to the changing outdoor environmental conditions. It is difficult to respond to changes in the sensor's external parameters in real time, and it is impossible to achieve real-time, automatic calibration and correction. Summary of the Invention

[0005] The present application provides an external parameter calibration method, device and electronic equipment to solve the problem of being unable to automatically detect changes in sensor external parameters and make real-time adjustments.

[0006] In a first aspect, the present application provides an external parameter calibration method, comprising:

[0007] Obtaining image data collected by a camera and point cloud data collected by a lidar, wherein the timestamp of the image data is the same as the timestamp of the point cloud data;

[0008] Calculating predicted extrinsic calibration parameters between the camera and the lidar based on the image data and the point cloud data;

[0009] Get the current external calibration parameters;

[0010] Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter;

[0011] When the extrinsic parameter error value satisfies a correction condition, extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data;

[0012] Calculating a matching error value between the visual semantic feature and the point cloud semantic feature;

[0013] Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, the graph optimization framework including nodes and edges, the nodes being used to represent target extrinsic parameter calibration parameters, and the edges being used to represent constraints formed based on the extrinsic parameter error value and the matching error value;

[0014] Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework;

[0015] The current external parameter calibration parameters are updated to the target external parameter calibration parameters.

[0016] In a second aspect, the present application provides an external parameter calibration device, comprising:

[0017] A data acquisition module is used to acquire image data collected by the camera and point cloud data collected by the lidar, where the timestamp of the image data is the same as the timestamp of the point cloud data;

[0018] a parameter monitoring module, configured to calculate predicted extrinsic calibration parameters between the camera and the lidar based on the image data and the point cloud data;

[0019] Get the current external calibration parameters;

[0020] Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter;

[0021] a parameter correction module, configured to extract visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data when the extrinsic parameter error value satisfies a correction condition;

[0022] Calculating a matching error value between the visual semantic feature and the point cloud semantic feature;

[0023] Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, the graph optimization framework including nodes and edges, the nodes being used to represent target extrinsic parameter calibration parameters, and the edges being used to represent constraints formed based on the extrinsic parameter error value and the matching error value;

[0024] Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework;

[0025] The parameter updating module is used to update the current extrinsic calibration parameters to the target extrinsic calibration parameters.

[0026] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a bus. The processor and the memory communicate with each other via the bus; the memory stores computer program instructions executable by the processor, and the processor is configured to:

[0027] Obtaining image data captured by a camera and point cloud data captured by a lidar, wherein the timestamp of the image data is the same as the timestamp of the point cloud data;

[0028] Calculating predicted extrinsic calibration parameters between the camera and the lidar based on the image data and the point cloud data;

[0029] Get the current external calibration parameters;

[0030] Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter;

[0031] When the extrinsic parameter error value satisfies a correction condition, extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data;

[0032] Calculating a matching error value between the visual semantic feature and the point cloud semantic feature;

[0033] Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, the graph optimization framework including nodes and edges, the nodes being used to represent target extrinsic parameter calibration parameters, and the edges being used to represent constraints formed based on the extrinsic parameter error value and the matching error value;

[0034] Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework;

[0035] The current external parameter calibration parameters are updated to the target external parameter calibration parameters.

[0036] It can be seen from the above technical solutions that the present application provides an extrinsic parameter calibration method, device and electronic device, which can obtain image data collected by a camera and point cloud data collected by a laser radar, calculate the predicted extrinsic parameter calibration parameters between the camera and the laser radar based on the image data and the point cloud data, and calculate the extrinsic parameter error value based on the predicted extrinsic parameter calibration parameters and the current extrinsic parameter calibration parameters. When the extrinsic parameter error value meets the correction condition, the visual semantic features of the target object in the image data and the point cloud semantic features of the target object in the point cloud data are extracted, and the matching error value between the visual semantic features and the point cloud semantic features is calculated. A graph optimization framework is constructed based on the matching error value and the extrinsic parameter error value, and the target extrinsic parameter calibration parameters corresponding to the minimum matching error value are solved based on the graph optimization framework, and the current extrinsic parameter calibration parameters are updated to the target extrinsic parameter calibration parameters. The method can automatically monitor the changes in the extrinsic parameter calibration parameters between the camera and the laser radar, and automatically correct the extrinsic parameter calibration parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of the structure of the external parameter calibration device provided in an embodiment of the present application;

[0039] Figure 2 A flow chart of the external parameter calibration method provided in the embodiment of the present application;

[0040] Figure 3 A schematic diagram of the monitoring process of the external calibration parameters provided in the embodiment of the present application;

[0041] Figure 4 A schematic diagram of the calibration process for the external calibration parameters provided in the embodiment of the present application;

[0042] Figure 5 A schematic diagram of the calibration result verification process provided in the embodiment of the present application;

[0043] Figure 6 A schematic diagram of a real image collected according to an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the position offset of a pole when the external parameters between the camera and the laser radar provided in an embodiment of the present application change;

[0045] Figure 8A schematic diagram of the alignment and coincidence of the pole positions after iterative optimization of the extrinsic calibration parameters provided in an embodiment of the present application;

[0046] Figure 9 This is a structural block diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all embodiments consistent with the present application. They are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the claims.

[0048] With the development of autonomous driving technology, autonomous driving technology has increasingly higher requirements for the perception of the vehicle's surrounding environment. The intelligent driving data acquisition system is a key component for realizing advanced driving assistance systems (ADAS). The intelligent driving data acquisition system integrates multiple sensors, such as LiDAR, cameras, millimeter-wave radar (Radar), ultrasonic sensors, global positioning system (GPS), inertial measurement unit (IMU), etc., and collects environmental data during vehicle driving through multiple sensors to provide support for subsequent model training and simulation speed measurement.

[0049] To achieve effective multi-sensor data fusion, the relative positions and poses of each sensor must be determined. This process is called sensor calibration. Sensor calibration is divided into intrinsic calibration and extrinsic calibration. Intrinsic calibration determines the internal parameters of the sensor, such as the camera's focal length and distortion coefficient. Extrinsic calibration determines the relative positions and coordinate transformations between sensors.

[0050] Cameras and lidar, as sensors in intelligent driving data collection systems, are crucial for the accuracy and stability of their external parameters. In practical applications, these systems collect environmental data from various road sections outdoors for extended periods of time. The diverse and complex environment or the vehicle's own motion can cause changes in the relative position of sensors, leading to variations in sensor external parameters. If these changes are not detected and corrected promptly and accurately, they will reduce data collection accuracy and the effectiveness of subsequent model training and simulation testing.

[0051] The calibration method of sensor external parameters can be performed manually using specific calibration facilities (such as calibration plates) in a specific environment. However, this method is time-consuming and inefficient, and it is difficult to adapt to the changing outdoor environmental conditions. It is difficult to respond to changes in sensor external parameters in real time, and it is impossible to achieve real-time, automatic calibration and correction.

[0052] Based on the above implementations, it can be seen that the calibration method for sensor external parameters lacks automation and is unable to automatically detect changes in sensor external parameters and make real-time adjustments. The real-time performance is insufficient and cannot adapt to the dynamic changes in external parameters that may occur during vehicle use. The manual calibration process is cumbersome and requires specific environmental conditions or calibration facilities, which is time-consuming and limits the flexibility and universality of the calibration process. The maintenance cost is high. Once the calibration is completed, subsequent maintenance and recalibration require professional operation, which increases maintenance costs and time.

[0053] Therefore, some embodiments of the present application provide an external parameter calibration method that can automatically monitor the changes in external parameter calibration parameters between sensors and automatically correct the external parameter calibration parameters. Figure 1 As shown, it is a structural schematic diagram of the external parameter calibration device provided in an embodiment of the present application. The external parameter calibration device includes a data acquisition module, a parameter monitoring module, a parameter correction module and a parameter update module. The data acquisition module is used to collect sensor data. The parameter monitoring module is used to automatically monitor whether the external parameter calibration parameters between sensors have changed. The parameter correction module is used to automatically correct the external parameter calibration parameters between sensors when the external parameter calibration parameters change. The parameter update module is used to verify the corrected external parameter calibration parameters and, after the verification passes, update the current external parameter calibration parameters.

[0054] For ease of understanding, the extrinsic parameter calibration method is applied to the above-mentioned intelligent driving data acquisition system. The sensor calibration device is integrated into the intelligent driving data acquisition system. By acquiring the environmental data collected by the camera and the lidar during the vehicle's driving process, the extrinsic parameter calibration parameters between the camera and the lidar are automatically monitored and corrected. Figure 2 As shown, Figure 2 A flow chart of an external parameter calibration method provided in an embodiment of the present application, wherein the external parameter calibration method comprises the following steps:

[0055] S201: Obtain image data collected by the camera and point cloud data collected by the lidar.

[0056] The timestamps of the image data and the point cloud data are the same. After the data acquisition module starts, it initializes the various sensor modules, including the camera, lidar, and IMU, to ensure proper functioning and begin data collection. To improve data processing accuracy, the data acquisition module performs preprocessing on the acquired image and point cloud data, including time synchronization and data quality checks, before transmitting it to the parameter monitoring module.

[0057] For example, a hardware timestamp synchronizer or a global synchronization network (Precision Time Protocol, PTP) can be used to synchronize the timestamps of image and point cloud data. If there is a discrepancy between the timestamps of the image and point cloud data, the data can be aligned using timestamp interpolation. After the timestamp synchronization process, the quality of the image and point cloud data can be checked to ensure that there are no outliers or missing frames.

[0058] S202: Calculate the predicted extrinsic calibration parameters between the camera and the lidar based on the image data and the point cloud data.

[0059] It should be noted that in this embodiment, the predicted extrinsic parameter calibration parameters, the current extrinsic parameter calibration parameters, and the target extrinsic parameter calibration parameters are all extrinsic parameter calibration parameters between the camera and the laser radar, and all include a rotation matrix and a translation vector. The rotation matrix is ​​used to characterize the rotation relationship between the coordinate system of the camera and the coordinate system of the laser radar, and the translation vector is used to characterize the position offset between the coordinate system of the camera and the coordinate system of the laser radar. For the sake of ease of understanding, the extrinsic parameter calibration parameters calculated based on the image data and the point cloud data are referred to as predicted extrinsic parameter calibration parameters, and the predicted extrinsic parameter calibration parameters include the predicted rotation matrix and the predicted translation vector; the extrinsic parameter calibration parameters currently actually used by the system are referred to as current extrinsic parameter calibration parameters, and the current extrinsic parameter calibration parameters include the current rotation matrix and the current translation vector; the extrinsic parameter calibration parameters obtained by automatic correction are referred to as target extrinsic parameter calibration parameters, and the target extrinsic parameter calibration parameters include the target rotation matrix and the target translation vector.

[0060] In some embodiments, as Figure 3 The figure shows a flow chart of monitoring extrinsic calibration parameters provided in an embodiment of the application. A first extrinsic parameter of the camera can be calculated based on the image data of two adjacent frames, and a second extrinsic parameter of the lidar can be calculated based on the point cloud data of two adjacent frames. The extrinsic calibration parameters are then predicted based on the first and second extrinsic parameters.

[0061] The first extrinsic parameter includes a first rotation matrix and a first translation vector. The first rotation matrix is ​​used to represent the rotation relationship between the camera's coordinate system and the reference coordinate system, and the first translation vector is used to represent the position offset between the camera's coordinate system and the reference coordinate system. The second extrinsic parameter includes a second rotation matrix and a second translation vector. The second rotation matrix is ​​used to represent the rotation relationship between the laser radar's coordinate system and the reference coordinate system, and the second translation vector is used to represent the position offset between the laser radar's coordinate system and the reference coordinate system.

[0062] It should be noted that the reference coordinate system can be adjusted according to actual conditions. For example, in the field of autonomous driving in the embodiment of the present application, the reference coordinate system can be the vehicle coordinate system, and in other fields, it can also be the world coordinate system.

[0063] In some embodiments, as Figure 3 As shown, the visual odometry can be used to estimate the relative motion data of the camera based on two frames of image data, and the position and posture of the camera in three-dimensional space can be calculated based on the relative motion data to obtain the first extrinsic parameter of the camera.

[0064] LiDAR odometry can be used to estimate the relative motion data of the LiDAR based on two frames of point cloud data, and the position of the LiDAR in three-dimensional space can be calculated based on the relative motion data to obtain the second external parameter of the LiDAR.

[0065] For example, you can define the coordinate systems of the camera and lidar, as well as the transformation relationship between the camera and lidar. cam Indicates that the laser radar coordinate system is lidar Indicates that the external calibration parameters between the camera and the lidar are expressed as: Among them, R calib is the rotation matrix, t calib is the translation vector.

[0066] During vehicle driving, a visual odometry (such as ORB-SLAM) is used to estimate the relative motion data of the camera based on two frames of image data, thereby calculating the first external parameters of the camera, including the first external rotation matrix R cam and the first translation vector t cam Similarly, a lidar odometer (such as FAST-LIO2) is used to estimate the relative motion data of the lidar based on two frames of point cloud data, thereby calculating the second external parameters of the lidar, including the second external rotation matrix R lidar and the second translation vector t lidar .

[0067] Since the image data and point cloud data may be collected at different timestamps by the camera and lidar, the continuous pose difference calculated by the IMU and wheel speed can be used to synchronize the image data and point cloud data to the same time point through interpolation. The extrinsic calibration parameters can be calculated and predicted based on the first and second extrinsic parameters according to the following formula:

[0068] R cam R calib =R calib R lidar ;

[0069] R cam t calib +st cam =R calib t lidar +t calib ;

[0070] Among them, R calib is the predicted rotation matrix, t calib To predict the translation vector, R cam is the first rotation matrix, t cam is the first translation vector, R lidar is the second rotation matrix, t lidar is the second translation vector, and s is the preset scaling factor used to address the scale inconsistency problem that may exist in the visual odometry.

[0071] S203: Obtain current external calibration parameters.

[0072] The current extrinsic calibration parameters are the extrinsic calibration parameters between the currently used camera and lidar.

[0073] S204: Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter.

[0074] In some embodiments, as Figure 3 As shown, the error function ε can be defined i To quantify the difference between the current extrinsic calibration parameters and the predicted extrinsic calibration parameters. That is, the extrinsic error value can be calculated based on the predicted extrinsic calibration parameters and the current extrinsic calibration parameters according to the following formula:

[0075]

[0076] ε i，trans =t calib -t calib_init ;

[0077] Among them, the external parameter error value includes the rotation error value ε i，rot and translation error ε i，trans , ε i，rot is the rotation error value, which is used to characterize the change of the rotation matrix, ε i，trans is the translation error value, which is used to characterize the change of the translation vector, R calib is the predicted rotation matrix, t calib To predict the translation vector, R calib_init is the current rotation matrix, t calib_init is the current translation vector, log represents the logarithmic mapping of the matrix, which is used to map the rotation matrix to the rotation vector.

[0078] S205: Determine whether the external parameter error value meets the correction condition.

[0079] The correction condition refers to the change of the external calibration parameters between the camera and the lidar.

[0080] In some embodiments, as Figure 3As shown, you can set an extrinsic parameter error threshold and compare the extrinsic parameter error value with the set extrinsic parameter error threshold to determine whether the extrinsic parameter calibration parameters between the camera and the lidar have changed. You can compare the extrinsic parameter error value with the extrinsic parameter error threshold. If the extrinsic parameter error value is greater than the extrinsic parameter error threshold, it is marked as meeting the calibration conditions, and subsequent extrinsic parameter calibration operations are performed. If the extrinsic parameter error value is less than or equal to the extrinsic parameter error threshold, it is marked as not meeting the calibration conditions, and subsequent extrinsic parameter calibration operations are not performed.

[0081] In some embodiments, the extrinsic error threshold includes a rotation error threshold T rot and translation error threshold T trans , we can compare the rotation error value ε i，rot and the rotation error threshold T rot , and the translation error value ε i，trans and the translation error threshold T trans , if ε i，rot <T rot And ε i，trans <T trans , marking the external parameter error value does not meet the correction conditions, and the subsequent external parameter calibration parameter correction operation is not performed. i，rot ≥T rot or ε i，trans ≥T trans , indicating that the external parameter calibration parameters have deviated, marking the external parameter error value meets the correction conditions, and continuing to perform subsequent external parameter calibration parameter correction operations.

[0082] In some embodiments, multiple predicted extrinsic parameter calibration parameters calculated based on image data of two adjacent frames can be obtained, and an average of the multiple predicted extrinsic parameter calibration parameters can be calculated. If the average value is greater than an extrinsic parameter error threshold, the extrinsic parameter error value is marked as meeting the correction condition. If the average value is less than or equal to the extrinsic parameter error threshold, the extrinsic parameter error value is marked as not meeting the correction condition.

[0083] S206: When the extrinsic parameter error value satisfies the correction condition, extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data.

[0084] like Figure 4 The figure shows a schematic diagram of the calibration process for extrinsic calibration parameters provided in an embodiment of the application. In the field of autonomous driving, the target object can be a lane line, a pole, a building, etc. To extract the visual semantic features of the target object, the image data can be input into a feature extraction model. The feature extraction model is a neural network model trained based on a dataset of image samples labeled with the visual semantic features of the target object. The feature extraction model identifies and extracts the visual semantic features of the target object in the image data.

[0085] To extract the visual semantic features of lane markings and poles, image data can be preprocessed, including resizing, normalizing brightness and contrast, and removing noise, to improve the model's recognition capabilities. The preprocessed image data is then fed into a trained feature extraction model, which then identifies and extracts the visual semantic features of lane markings and poles from the image data. Post-processing can also be performed on the output of the feature extraction model. For example, non-maximum suppression (NMS) can be used to remove overlapping prediction boxes, and morphological operations (such as dilation, erosion, opening, and closing) can be used to improve the continuity and clarity of lane markings.

[0086] To extract semantic features from the point cloud of the target object, the point cloud data can be preprocessed, including denoising, filtering, and downsampling, to reduce computational complexity and improve the accuracy of subsequent processing. To extract semantic features from lane lines, ground points can be segmented from the point cloud data based on their height information (i.e., Z-axis coordinates). Taking advantage of the higher reflectivity of lane line points compared to other road point clouds, intensity rendering is used to project the ground points onto a top-down view, converting them into a two-dimensional image defined by the X and Y axes. Binarization is then performed on the two-dimensional image to separate the lane lines from the background. Clustering (such as DBSCAN clustering or K-means clustering) is then performed on the binarized 2D image to group points belonging to the same lane line together to obtain clustering results. Finally, a curve fitting (such as polynomial fitting or least squares method) is performed on the clustering results to the lane line shape to obtain the point cloud semantic features of the lane lines.

[0087] For the point cloud semantic feature extraction of the pole, non-ground points can be segmented in the point cloud data based on the Z-axis coordinates of the point cloud data. Based on the geometric characteristics of the pole (such as the vertical or approximately vertical characteristics of the pole-shaped object) and through a clustering algorithm or a region growing algorithm, the pole can be identified at the non-ground point and the feature points of the pole can be extracted. The extracted pole feature points can also be post-processed (such as removing noise points, smoothing edges, correcting geometric shapes, etc.) to ensure that the extracted feature points conform to the actual geometric shape and position, thereby obtaining the point cloud semantic features of the pole.

[0088] S207: Calculate the matching error value between the visual semantic features and the point cloud semantic features.

[0089] In some embodiments, as Figure 4As shown, in order to improve the subsequent correction effect, key frame screening can be performed, and image data and point cloud data from which corresponding semantic features are detected at the same time and whose feature quality meets the requirements are selected as key frame data. The visual semantic features of the target object in the key frame data are associated with the point cloud semantic features, and the matching error value is calculated. That is, key image data can be extracted from the image data, and key point cloud data can be extracted from the point cloud data. The matching error value is calculated based on the visual semantic features corresponding to the key image data and the point cloud semantic features corresponding to the key point cloud data. Among them, the key image data is the image data from which the visual semantic features of the target object are extracted and the visual semantic features meet the feature quality requirements, the key point cloud data is the point cloud data from which the point cloud semantic features of the target object are extracted and the point cloud semantic features meet the feature quality requirements, and the timestamp of the key image data is the same as the timestamp of the key point cloud data.

[0090] In some embodiments, when calculating the matching error between the visual semantic feature and the point cloud semantic feature, the feature points in the point cloud semantic feature can be projected from the laser radar coordinate system to the camera coordinate system based on the predicted extrinsic calibration parameters to obtain second projected feature points. A matching error value is then calculated based on the pixel coordinates of the feature points in the visual semantic feature and the pixel coordinates of the second projected feature points. The matching error value is the difference between the pixel coordinates of the feature points in the visual semantic feature and the pixel coordinates of the second projected feature points.

[0091] For example, the error function e can be defined as corr To quantify the difference between the observation z (feature points of visual semantic features and feature points of point cloud semantic features) and the state variable x (external calibration parameter to be corrected), in automatic calibration, the error function is quantified based on the correspondence between feature points. Let P 3D is a feature point in the point cloud semantic feature extracted from the 3D point cloud data, P cam is the camera projection matrix, T calib Calibrate the external parameters for the camera-to-lidar prediction. This can be done based on the camera projection matrix P cam and the predicted external parameter calibration parameter T calib , the feature point P 3D Project it onto the 2D image plane of the camera to get the feature point P 3D Feature point P on the 2D image plane 2D , P 2D The 2D pixel coordinates can be expressed as:

[0092] P 2D =P cam T calib P 3D ;

[0093] set up is the 2D pixel coordinate of the feature point in the visual semantic features extracted by the feature extraction model, which can be calculated based on the error function e corr The difference between the 2D pixel coordinates predicted by the quantization model and the 2D pixel coordinates obtained by projection is calculated as follows:

[0094]

[0095] Among them, e corr is the matching error value, is the pixel coordinate of the feature point in the visual semantic feature, P 2D It is the pixel coordinate of the corresponding feature point in the point cloud semantic feature projected into the camera coordinate system.

[0096] S208: Constructing a graph optimization framework based on the matching error value and the external parameter error value.

[0097] Graph optimization is a mathematical optimization method that can abstract the variables and constraints in practical problems into nodes and edges in graph theory, thereby converting the solution of the problem into a process of operating on this graph structure. In graph optimization, nodes represent variables to be optimized, and edges represent the constraints between variables. In this embodiment, the graph optimization framework includes nodes and edges. The nodes are used to represent the state variables to be optimized, that is, the external parameter calibration parameters between the camera and the lidar. The edges represent the relationship between the nodes and are used to represent the constraints based on the external parameter error value and the matching error value.

[0098] S209: Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework.

[0099] The graph optimization framework includes a cost function, which is used to calculate the optimal solution of state variables while satisfying all constraints. The goal of graph optimization is to find the state vector that minimizes the cost function.

[0100] The cost function of the graph optimization framework is:

[0101] J=J mot +αJ corr ;

[0102] J corr =∑ j ρ(||e j，corr || 2 );

[0103] J mot =∑ j ρ(||ε i，rot || 2 )+∑ j ρ(||ε i，trans || 2);

[0104] Among them, J is the cost function, which is the weighted sum of all error terms, J mot is the motion error value, J corr is the matching error value, e j，corr is the matching error value corresponding to the jth feature point between the visual semantic feature and the point cloud semantic feature, ε i，rot is the rotation error value, ε i，trans is the translation error value, α is the weight parameter, and ρ is the kernel function, such as the Huber function.

[0105] Based on the cost function, the target extrinsic calibration parameters corresponding to the minimum matching error value can be iteratively solved by the iterative optimization algorithm according to the following formula:

[0106]

[0107] Where x is the target extrinsic calibration parameter, and J is the cost function. The iterative optimization algorithm can be a nonlinear least squares method, such as the Levenberg-Marquardt algorithm. In each iteration, the state vector x is updated based on the current error estimate to minimize the difference between the model's predicted 2D pixel coordinates and the projected 2D pixel coordinates. This iterative solution achieves accurate calibration between the camera and lidar.

[0108] In this embodiment, when a deviation in the extrinsic calibration parameters is detected, the correction of the extrinsic calibration parameters will be automatically triggered. Feature extraction is performed on the image data and point cloud data respectively, including extraction of semantic features such as lane lines and poles, and whether the frame data is saved as a key frame is determined by judging whether the semantic features meet the calibration requirements. After saving a certain number of key frame data, the semantic features extracted from the image data in the key frame and the features extracted from the point cloud data are correlated and matched, and the matching error value is calculated based on the matching degree. Combined with the matching error values ​​of multiple key frames and the constraints formed by the relative motion between the key frames, the extrinsic calibration parameters between the camera and the lidar are solved based on the graph optimization framework.

[0109] S210: Update the current extrinsic calibration parameters to the target extrinsic calibration parameters.

[0110] In some embodiments, after the target external parameter calibration parameters are calculated, the target external parameter calibration parameters can be verified. Figure 5The figure shows a flow chart of calibration result verification provided by an embodiment of the present application. New image data collected by the camera and new point cloud data collected by the lidar can be obtained. Based on the target extrinsic calibration parameters, the feature points in the new point cloud data are projected from the laser radar coordinate system to the camera coordinate system to obtain the target projection feature points. The target matching error value is then calculated based on the pixel coordinates of the feature points in the new image data and the pixel coordinates of the target projection feature points. The target matching error value is obtained by adding the pixel coordinates of the feature points in the new image data and the pixel coordinates of the target projection feature points and then taking the average.

[0111] For the target matching error value and the matching error value threshold, if the target matching error value is less than the matching error value threshold, it means that the verification is passed, and the current extrinsic parameter calibration parameters are updated to the target extrinsic parameter calibration parameters. If the target matching error value is greater than or equal to the matching error value threshold, it means that the verification fails, and the operation steps for extrinsic parameter calibration are repeated.

[0112] For example, we can obtain the image data collected by multiple frames of cameras and the point cloud data collected by lidar, and then use the camera projection matrix P cam and target external calibration parameter T calib , project the 3D pixel coordinates of the feature points in the point cloud data onto the 2D image plane of the camera to obtain the 2D pixel coordinates of the feature points on the 2D image plane. According to the above error function e corr The calculation method is to calculate the error value between the 2D pixel coordinates of the feature points in the image data and the 2D pixel coordinates obtained by projection, and then add the error values ​​of the pixel coordinates of all feature points and take the average value to obtain the target matching error value e avg，corr , compared with the target matching error value e avg，corr and matching error threshold T corr , if e avg，corr <T corr , indicating that the verification is passed, if e avg，corr ≥T corr , indicating that the verification failed and the external parameter calibration needs to be performed again.

[0113] In this embodiment, after the extrinsic calibration parameters are corrected, the calibration results can be checked to see if they meet the preset accuracy requirements. Using the automatically corrected extrinsic calibration parameters, the newly acquired LiDAR point cloud data is projected onto the image data to calculate the positional difference. The error is then compared with a set threshold to determine if the calibration results meet the accuracy requirements. If the calibration results do not meet the accuracy requirements, data that meets the requirements can be collected and recalibrated.

[0114] If the calibration results meet the requirements, they can be recorded and stored for subsequent data collection and processing. The system will then continuously monitor the sensor's performance and external calibration parameters to ensure data acquisition accuracy and system stability. If the external calibration parameters change significantly, the automatic calibration process will be retriggered.

[0115] In some embodiments, a user interface can be provided to display calibration status, accuracy indicators, and possible error messages, allowing the user to monitor system status and interact with the system. If the system detects a significant change in calibration parameters, it will automatically correct the process. Upon completion, a prompt message can be displayed in the user interface to prompt the user to pay attention and confirm. After the user confirms, the current system's extrinsic calibration parameters are updated to the target extrinsic calibration parameters obtained by the automatic correction, and the result is logged.

[0116] In this embodiment, the extrinsic calibration method can monitor changes in extrinsic calibration parameters between sensors in real time during the acquisition process, without requiring manual intervention. When changes are detected, data can be automatically collected to calibrate the extrinsic calibration parameters, eliminating the need for manual calibration based on a specific calibration environment. This reduces maintenance costs and improves acquisition efficiency.

[0117] like Figures 6 to 8 As shown, Figure 6 is the original image collected. Figure 7 To demonstrate that the change in external parameters between the camera and the lidar causes the laser point cloud of the pole to be projected into the image and the position of the 2D pixel points of the pole in the image to deviate, Figure 8 This demonstrates that the laser point cloud of the pole is aligned with the position of the pole in the image after iterative optimization of the external calibration parameters.

[0118] Based on the above external parameter calibration method, some embodiments of the present application also provide an external parameter calibration device, such as Figure 1 , which is a flow chart of an external parameter calibration device provided in an embodiment of the present application, the external parameter calibration device includes a data acquisition module, a parameter monitoring module, a parameter correction module and a parameter updating module.

[0119] The data acquisition module is used to obtain image data collected by the camera and point cloud data collected by the lidar. The timestamp of the image data is the same as the timestamp of the point cloud data.

[0120] The parameter monitoring module is used to calculate the predicted extrinsic calibration parameters between the camera and the lidar based on image data and point cloud data.

[0121] Get the current external calibration parameters.

[0122] The extrinsic parameter error value is calculated based on the predicted extrinsic parameter calibration parameters and the current extrinsic parameter calibration parameters.

[0123] The parameter correction module is used to extract the visual semantic features of the target object in the image data and the point cloud semantic features of the target object in the point cloud data when the external parameter error value meets the correction condition.

[0124] Calculate the matching error between the visual semantic features and the point cloud semantic features.

[0125] A graph optimization framework is constructed based on the matching error value and the extrinsic parameter error value. The graph optimization framework includes nodes and edges. The nodes are used to represent the target extrinsic parameter calibration parameters, and the edges are used to represent the constraints based on the extrinsic parameter error value and the matching error value.

[0126] Based on the graph optimization framework, the target extrinsic calibration parameters corresponding to the minimum matching error value are solved.

[0127] The parameter updating module is used to update the current external parameter calibration parameters to the target external parameter calibration parameters.

[0128] Based on the above external parameter calibration method, some embodiments of the present application further provide an electronic device, such as Figure 9 As shown, Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 900 includes: at least one processor 901, at least one communication interface 902, at least one memory 903 and at least one bus 904. Among them, the bus 904 is used to realize direct connection and communication between these components, the communication interface 302 is used to communicate signaling or data with other node devices, and the memory 903 stores computer program instructions that can be executed by the processor 901. When the electronic device 900 is running, the processor 901 communicates with the memory 903 through the bus 904. The processor 901 calls the computer program stored in the memory 903 and executes the computer program to implement the external parameter calibration method provided in the embodiment of the present application.

[0129] Based on the above-mentioned external parameter calibration method, some embodiments of the present application also provide a computer program product, including a computer program stored on a computer-readable storage medium, the computer program including computer program instructions. When the computer program instructions are executed by a computer, the computer can execute the external parameter calibration method in the above-mentioned embodiment.

[0130] Based on the above-mentioned external parameter calibration method, some embodiments of the present application also provide a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the external parameter calibration method described in the above-mentioned method embodiment.

[0131] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A method for calibrating an external parameter, characterized in that: include: Acquire image data collected by a camera and point cloud data collected by a laser radar, wherein a timestamp of the image data is the same as a timestamp of the point cloud data; Calculate the predicted extrinsic calibration parameters between the camera and the laser radar based on the image data and the point cloud data; Get the current external calibration parameters; Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter; When the extrinsic parameter error value satisfies a correction condition, extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data; Calculating a matching error value between the visual semantic feature and the point cloud semantic feature; Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, wherein the graph optimization framework includes nodes and edges, wherein the nodes are used to represent target extrinsic parameter calibration parameters, and the edges are used to represent constraints formed based on the extrinsic parameter error value and the matching error value; Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework; The current external parameter calibration parameters are updated to the target external parameter calibration parameters.

2. The external parameter calibration method according to claim 1, characterized in that: The step of calculating the predicted extrinsic calibration parameters between the camera and the laser radar based on the image data and the point cloud data comprises: Calculating a first extrinsic parameter of the camera based on image data of two adjacent frames, the first extrinsic parameter comprising a first rotation matrix and a first translation vector, the first rotation matrix being used to characterize a rotation relationship between a coordinate system of the camera and a reference coordinate system, and the first translation vector being used to characterize a position offset between the coordinate system of the camera and the reference coordinate system; Calculating a second extrinsic parameter of the laser radar based on point cloud data of two adjacent frames; the second extrinsic parameter includes a second rotation matrix and a second translation vector, the second rotation matrix is ​​used to characterize the rotation relationship between the coordinate system of the laser radar and the reference coordinate system, and the second translation vector is used to characterize the position offset between the coordinate system of the laser radar and the reference coordinate system; A predicted extrinsic parameter calibration parameter is calculated based on the first extrinsic parameter and the second extrinsic parameter.

3. The external parameter calibration method according to claim 2, characterized in that: The predicted extrinsic calibration parameters include a predicted rotation matrix and a predicted translation vector, wherein the predicted rotation matrix is ​​used to characterize the rotation relationship between the coordinate system of the camera and the coordinate system of the laser radar, and the predicted translation vector is used to characterize the position offset between the coordinate system of the camera and the coordinate system of the laser radar; the predicted extrinsic calibration parameters are calculated according to the first extrinsic parameter and the second extrinsic parameter according to the following formula: R cam R calib =R calib R lidar ; R cam t calib +st cam =R calib t lidar +t calib ; Among them, R calib is the predicted rotation matrix, t calib To predict the translation vector, R cam is the first rotation matrix, t cam is the first translation vector, R lidar is the second rotation matrix, t lidar is the second translation vector, and s is the preset scaling factor.

4. The external parameter calibration method according to claim 3, characterized in that: The current extrinsic calibration parameters include a current rotation matrix and a current translation vector; the extrinsic error value is calculated based on the predicted extrinsic calibration parameters and the current extrinsic calibration parameters according to the following formula: ε i，trans =t calib -t calib_init ; Among them, the external parameter error value includes the rotation error value and the translation error value, ε i，rot is the rotation error value, ε i,trans is the translation error value, R calib is the predicted rotation matrix, t calib To predict the translation vector, R calib_init is the current rotation matrix, t calib_init is the current translation vector, log represents the logarithmic mapping of the matrix, which is used to map the rotation matrix to the rotation vector.

5. The external parameter calibration method according to claim 4, characterized in that: The method further comprises: Acquire a plurality of predicted extrinsic calibration parameters calculated based on image data of two adjacent frames; Calculating an average value for a plurality of the predicted external parameter calibration parameters; If the average value is greater than the external parameter error threshold, mark the external parameter error value as meeting the correction condition; If the average value is less than or equal to the external parameter error threshold, it is marked that the external parameter error value does not meet the correction condition.

6. The external parameter calibration method according to claim 1, characterized in that: The step of extracting visual semantic features of the target object in the image data comprises: Inputting the image data into a feature extraction model, wherein the feature extraction model is a neural network model trained based on an image sample data set labeled with visual semantic features of a target object; The visual semantic features of the target object in the image data are extracted by the feature extraction model.

7. The external parameter calibration method according to claim 1, characterized in that: The step of extracting point cloud semantic features of the target object in the point cloud data comprises: Segmenting ground points in the point cloud data; Converting the ground points into a two-dimensional image; Performing binarization processing on the two-dimensional image; Performing clustering processing on the two-dimensional image after binarization processing to obtain a clustering result; The clustering result curve is fitted to the shape of the target object to obtain the point cloud semantic features of the target object.

8. The external parameter calibration method according to claim 1, characterized in that: The step of extracting point cloud semantic features of the target object in the point cloud data comprises: Segmenting non-ground points in the point cloud data; Feature points of the target object are extracted at non-ground points based on the geometric characteristics of the target object to obtain point cloud semantic features of the target object.

9. The external parameter calibration method according to claim 1, characterized in that: The step of calculating the matching error value between the visual semantic feature and the point cloud semantic feature comprises: Extracting key image data from the image data, and extracting key point cloud data from the point cloud data, wherein the key image data is image data from which visual semantic features of a target object are extracted and the visual semantic features meet feature quality requirements, and the key point cloud data is point cloud data from which point cloud semantic features of a target object are extracted and the point cloud semantic features meet feature quality requirements, and the timestamp of the key image data is the same as the timestamp of the key point cloud data; Calculate the matching error value between the visual semantic features corresponding to the key image data and the point cloud semantic features corresponding to the key point cloud data.

10. The external parameter calibration method according to claim 1, characterized in that: The step of calculating the matching error value between the visual semantic feature and the point cloud semantic feature comprises: Based on the predicted extrinsic calibration parameters, projecting the feature points in the point cloud semantic features from the coordinate system of the laser radar to the coordinate system of the camera to obtain second projected feature points; A matching error value is calculated according to the pixel coordinates of the feature point in the visual semantic feature and the pixel coordinates of the second projection feature point, wherein the matching error value is the difference between the pixel coordinates of the feature point in the visual semantic feature and the pixel coordinates of the second projection feature point.

11. The external parameter calibration method according to claim 4, characterized in that: The cost function of the graph optimization framework is: J=J mot +αJ corr ; J corr =∑ j p(||e j，corr || 2 ); J mot =∑ j p(||e i，rot || 2 )+∑ j p(||e i，trans || 2 ); Among them, J is the cost function, J mot is the motion error value, J corr is the matching error value, e j,corr is the matching error value corresponding to the jth feature point between the visual semantic feature and the point cloud semantic feature, ε i,rot is the rotation error value, ε i,trans is the translation error value, α is the weight parameter, and ρ is the kernel function; The target extrinsic parameter calibration parameter corresponding to the minimum matching error value is iteratively solved based on the cost function.

12. The external parameter calibration method according to claim 1, characterized in that: The method further comprises: Obtain new image data collected by the camera and new point cloud data collected by the lidar; Based on the target extrinsic calibration parameters, the feature points in the new point cloud data are projected from the coordinate system of the laser radar to the coordinate system of the camera to obtain target projection feature points; Calculate a target matching error value according to the pixel coordinates of the feature points in the new image data and the pixel coordinates of the target projection feature points, wherein the target matching error value is obtained by adding the pixel coordinates of the feature points in the new image data and the pixel coordinates of the target projection feature points and then averaging them; If the target matching error value is less than the matching error value threshold, the current extrinsic parameter calibration parameter is updated to the target extrinsic parameter calibration parameter.

13. An external parameter calibration device, characterized in that: include: A data acquisition module, used to acquire image data acquired by a camera and point cloud data acquired by a laser radar, wherein the timestamp of the image data is the same as the timestamp of the point cloud data; A parameter monitoring module, used for calculating the predicted extrinsic calibration parameters between the camera and the laser radar based on the image data and the point cloud data; Get the current external calibration parameters; Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter; A parameter correction module, used for extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data when the external parameter error value meets the correction condition; Calculating a matching error value between the visual semantic feature and the point cloud semantic feature; Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, wherein the graph optimization framework includes nodes and edges, wherein the nodes are used to represent target extrinsic parameter calibration parameters, and the edges are used to represent constraints formed based on the extrinsic parameter error value and the matching error value; Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework; A parameter updating module is used to update the current external parameter calibration parameters to the target external parameter calibration parameters.

14. An electronic device, characterized in that: include: processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions executable by the processor, the processor being configured to: Acquire image data collected by a camera and point cloud data collected by a laser radar, wherein a timestamp of the image data is the same as a timestamp of the point cloud data; Calculate the predicted extrinsic calibration parameters between the camera and the laser radar based on the image data and the point cloud data; Get the current external calibration parameters; Calculating an extrinsic parameter error value based on the predicted extrinsic parameter calibration parameter and the current extrinsic parameter calibration parameter; When the extrinsic parameter error value satisfies a correction condition, extracting visual semantic features of the target object in the image data and point cloud semantic features of the target object in the point cloud data; Calculating a matching error value between the visual semantic feature and the point cloud semantic feature; Constructing a graph optimization framework based on the matching error value and the extrinsic parameter error value, wherein the graph optimization framework includes nodes and edges, wherein the nodes are used to represent target extrinsic parameter calibration parameters, and the edges are used to represent constraints formed based on the extrinsic parameter error value and the matching error value; Solve the target extrinsic calibration parameters corresponding to the minimum matching error value based on the graph optimization framework; The current external parameter calibration parameters are updated to the target external parameter calibration parameters.

Images