Multi-mode sensor external parameter automatic calibration method and system, electronic equipment and storage medium

Abstract

The invention relates to the field of intelligent perception and data fusion, and discloses a multi-modal sensor external parameter automatic calibration method and system, electronic equipment and a storage medium, and the method comprises the following steps: S1, collecting observation data; s2, constructing an observation map; s3, establishing a feature matching relationship; s4, constructing a matrix equation set; s5, solving the matrix equation set to obtain external parameters; s6, error evaluation is carried out, and whether the steps S2 to S5 are executed again is determined; the system comprises a data acquisition module, a feature map construction module, a matching analysis module, an external parameter optimization module and a drift detection module, and the electronic equipment and the storage medium are both used for the method. By fusing the data of the laser radar, the camera and the inertial measurement unit, efficient data synchronization and integration are realized, the time-space consistency of the data among different sensors can be ensured, and the problem that the data among the different sensors are not matched or have large deviation is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent perception and data fusion, and in particular to a multi-modal sensor extrinsic parameter automatic calibration method and system, an electronic device and a storage medium. BACKGROUND

[0002] In the fields of autonomous driving, robotics, etc., multi-modal sensor fusion technology has become an important means to achieve high-precision positioning, mapping, and environmental perception. In existing technologies, common multi-modal sensors include lidar, cameras, and inertial measurement units (IMU). Lidar, with its high-precision distance measurement capability, is widely used for mapping and positioning, but its adaptability in complex environments is relatively poor; cameras provide rich image information and can be used to identify object and scene details, but their performance may be limited in low light or adverse weather conditions; IMU can provide object attitude and motion information, which is particularly important in high-speed motion and dynamic environments. Therefore, fusing data from multiple sensors has become an effective way to improve system perception capability and accuracy. In order to achieve data fusion and registration between different sensors, many studies have proposed feature extraction and matching, extrinsic parameter estimation, etc. to ensure that data of different modalities can be accurately aligned in time and space to support subsequent processing and analysis.

[0003] However, existing technologies still have some deficiencies in multi-modal data processing, feature matching, and extrinsic parameter estimation. First, in terms of data synchronization, although some algorithms have attempted to synchronize sensor times through static methods, when there is a large deviation in timestamps of different sensors, it may lead to inaccurate data fusion, thereby affecting the overall accuracy of the system. Second, existing feature point matching algorithms are prone to false matching when dealing with complex environments or low-quality data, especially when there are many types of sensors and frequent environmental changes, the reliability of matching cannot be guaranteed. Third, traditional extrinsic parameter estimation algorithms mostly use fixed-weight least squares optimization, which lacks sufficient robustness to abnormal point interference, resulting in unstable estimation results. Finally, existing calibration methods often rely on manual operation and can only be performed at the initial installation of the system. Once the extrinsic parameters drift, the system accuracy will be greatly reduced, making it difficult to meet the needs of long-term and dynamic operation. The above problems make existing technologies have certain limitations in practical applications, which cannot fully meet the needs of complex environments, affecting the long-term stability and reliability of the system. SUMMARY

[0004] The present application relates to the technical field of intelligent perception and data fusion, and in particular to a multi-modal sensor extrinsic parameter automatic calibration method and system, an electronic device and a storage medium.

[0005] To achieve the above object, the present application is realized by the following technical scheme: a multi-modal sensor external parameter automatic calibration method, comprising the following steps: S1, collecting observation data of a plurality of heterogeneous sensors; S2, extracting feature points based on the observation data, and constructing an observation graph of each sensor; S3, establishing a cross-modal feature matching relationship according to the geometric and structural characteristics between the observation graphs; S4, constructing a matrix equation set for external parameter estimation based on the feature matching relationship; S5, solving the matrix equation set by using a variable weight robust least squares optimization algorithm to obtain the external parameter; S6, error evaluation of the current estimated external parameter based on a set drift error evaluation function, and determining whether to re-execute steps S2 to S5 for external parameter update according to the evaluation result.

[0006] Preferably, in the S1 step, the observation data includes: three-dimensional point cloud data obtained by a laser radar; image data obtained by a camera; sensor acceleration and angular velocity data obtained by an inertial measurement unit.

[0007] Preferably, in the S2 step, the extraction of feature points includes: extracting corner points, edge points and semantic target centers from images by image processing algorithms; extracting edge points and curvature extreme points from laser radar data by point cloud processing algorithms; extracting key position trajectory points from inertial measurement unit data; The construction of the observation graph includes the following steps: extracted feature points are used as graph nodes, and edges are established according to the spatial distance or time sequence between nodes; Then the graph structure of the laser radar, the camera and the inertial measurement unit is constructed; The weight value of the edge is assigned according to the similarity measurement between nodes.

[0008] Preferably, in the S3 step, the cross-modal feature matching relationship is matched based on geometric consistency, and the geometric consistency is judged by calculating the distance difference of feature points in space between different observation graphs.

[0009] Preferably, in the S4 step, the construction of the matrix equation set includes the following steps: The cross-modal matching point pairs are represented as a three-dimensional coordinate set; a joint transformation model of rotation matrix and translation vector is constructed; applying a transformation model to each set of matched points and taking the transformation error thereof as a residual term; combining the residual terms to establish a linear matrix equation in the form of a least squares objective function; which is expressed in the following form: Ra i + T = b i + ∈ i ; wherein R is a rotation matrix; T is a translation vector; a i , b i are matched pairs of feature points; ∈ i is a disturbance term.

[0010] Preferably, in the S5 step, the objective function used by the variable-weight robust least squares optimization algorithm is: wherein R is a rotation matrix representing the rotational transformation from the source sensor coordinate system to the target sensor coordinate system; T is a translation vector representing the translational relationship between the two coordinate systems; n is the number of pairs of feature points participating in the optimization; a l is the lth feature point in the source sensor coordinate system; b l is the lth feature point matched with a l in the target sensor coordinate system; w i is the matching confidence; ||R|| σ is the spectral norm of the rotation matrix; and λ is a regularization coefficient.

[0011] Preferably, in the S6 step, the determination of whether to re-execute steps S2 to S5 includes the following steps: calculating the average residual corresponding to the current estimated external parameter; comparing the average residual with a preset threshold; when the residual exceeds the threshold, re-executing the steps S2 to S5 to update the external parameter result; when the residual does not exceed the threshold, maintaining the current external parameter unchanged and continuing to run the sensor data processing flow; the drift error evaluation function is: wherein E drift is the drift error evaluation function; n is the number of pairs of feature points participating in the error calculation; a l is the lth feature point in the source sensor coordinate system; b l is the lth feature point matched with a iThe matched first feature point; R is a rotation matrix, indicating the rotation relationship from the source coordinate system to the target coordinate system; T is a translation vector, indicating the translation relationship from the source coordinate system to the target coordinate system; when the error exceeds the set threshold, automatic re-calibration is triggered.

[0012] The multi-modal sensor external parameter automatic calibration system comprises: A data acquisition module is configured to acquire observation data of a plurality of heterogeneous sensors. A feature map construction module is configured to extract feature points from the observation data and construct a modal graph. A matching analysis module is configured to establish a feature matching relationship between different modalities. An external parameter optimization module is configured to solve external parameters based on a variable weight robust least squares method. A drift detection module is configured to determine whether to re-perform calibration based on error evaluation results.

[0013] An electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-modal sensor external parameter automatic calibration method.

[0014] A storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a multi-modal sensor external parameter automatic calibration method In summary, the present application has at least one of the following beneficial technical effects: 1. The present application realizes efficient data synchronization and integration by fusing lidar, camera and inertial measurement unit data through multi-modal data acquisition and preprocessing technology, and ensures the spatio-temporal consistency of data between different sensors. Compared with the single modal data processing scheme in the prior art, the present application effectively solves the problem of large data mismatch or deviation between different sensors, and improves the accuracy and stability of multi-source data fusion.

[0015] 2. The present application converts data of different modalities into a unified graph structure for processing through a cross-modal graph construction method based on feature point extraction. By accurately extracting and matching feature points, the differences between different modalities are successfully eliminated, ensuring the reliability of the graph structure. Compared with the scheme of modeling only relying on single image or point cloud data in the prior art, the present application effectively solves the problem of graph matching error accumulation, and enhances the robustness of cross-modal matching.

[0016] 3. This invention improves the accuracy of extrinsic parameter estimation by introducing a variable-weight robust least squares optimization algorithm. By dynamically adjusting the weights, the impact of outlier matching points on the overall optimization result can be effectively reduced, greatly improving the accuracy and robustness of the calculation. Compared with existing fixed-weight or unadaptive optimization algorithms, this invention can more flexibly handle noisy data, improve the stability of extrinsic parameter estimation, and avoid erroneous estimations caused by mismatched points.

[0017] 4. This invention employs a dynamic calibration mechanism for extrinsic parameter estimation errors. When extrinsic parameter drift occurs, a recalibration operation is triggered promptly, ensuring the long-term accuracy and stability of the system. Compared to existing technologies that rely on a fixed calibration process, this invention effectively avoids the impact of extrinsic parameter drift, providing an intelligent and automatic online calibration method that significantly improves the system's adaptability and real-time performance. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is the system module architecture of the present invention; Figure 3 This is a schematic diagram of the electronic device of the present invention; Figure 4 This is a schematic diagram of the storage medium of the present invention. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1 - Appendix Figure 4 The present invention will be further described in detail below.

[0020] This invention provides a method, system, electronic device, and storage medium for automatic calibration of extrinsic parameters of multimodal sensors, such as... Figure 1 As shown, the automatic calibration method for the extrinsic parameters of a multimodal sensor may include the following steps: S1 collects observation data from multiple heterogeneous sensors; Specifically, in this embodiment, the three-dimensional point cloud data acquired by the lidar can be represented as the following set: Among them, P L A set of three-dimensional point clouds obtained from a single frame of LiDAR data; Let be the three-dimensional spatial coordinates of the i-th point in the point cloud; These represent the X, Y, and Z coordinates of the point in the lidar body coordinate system, respectively; N L This indicates the number of points contained in the current point cloud frame.

[0021] Image data captured by a camera can be represented as a two-dimensional grayscale image or a color image, denoted as: I C = {I(u, v)}, (u, v) ∈ Ω; where I(u, v) represents the gray value or color information at the pixel position in the image, (u, v) represents the pixel position coordinates on the image plane; Ω is the image definition domain.

[0022] The data collected by the inertial measurement unit includes three-axis acceleration and three-axis angular velocity, represented as: a(t) = [a x (t), a y (t), a z (t)] T , ω(t) = [ω x (t), ω y (t), ω z (t)] T ; where a(t) and ω(t) are linear acceleration and angular velocity vectors at time t; a x (t), a y (t), a z (t) are acceleration components along X, Y, Z axis directions; ω x (t), ω y (t), ω z (t) are angular velocity components around X, Y, Z axis directions.

[0023] The collection process is coordinated by a multi-sensor synchronization module, which includes a time synchronizer and a data buffer. The time synchronizer receives clock signals from each sensor and aligns the timestamps of the collected data.

[0024] The synchronization is achieved by: for each image frame find the point cloud frame and IMU data sequence closest to its timestamp to form a unified observation frame: where F t is a frame of multi-modal observation data constructed at time t; is the image data frame collected at time t; is the point cloud data frame corresponding to time t; a(τ), ω(τ) are acceleration and angular velocity data; τ is the sampling time point of IMU observation data; δ is the half length of the IMU sliding window, used to extract data within a certain time range.

[0025] The data acquisition module includes an image acquisition unit, a point cloud acquisition unit, and an inertial measurement acquisition unit. The image acquisition unit is connected to a camera, the point cloud acquisition unit is connected to a laser radar, and the inertial measurement acquisition unit is connected to an IMU module. All of them are physically connected to a time synchronizer to realize data consistency.

[0026] In logic, the multi-modal observation frames output by the data acquisition module will be used as the input of subsequent feature extraction and graph construction, ensuring the correspondence of the data of each mode in space and time.

[0027] The acquisition frequency and resolution of multi-modal observation data are configured according to the actual application scenario. For example, the laser radar generates a point cloud frame at a sampling frequency of 10 Hz, the camera acquires an image frame at a sampling frequency of 30 Hz, and the IMU provides inertial measurement values at a sampling frequency of 200 Hz.

[0028] The acquisition process can realize the unified packaging and structured organization of data from multi-modal sensors, laying a data foundation for subsequent feature point extraction, graph structure construction, matching and optimization calculation, and has cross-modal data consistency and scalability.

[0029] S2, extracting feature points based on observation data, and constructing an observation graph for each sensor; Specifically, in this embodiment, the feature point extraction operation is performed on image, point cloud and inertial measurement unit data respectively to construct the observation graph of the corresponding mode.

[0030] First, an image feature point extraction model is established, and an image gradient and corner detection operator is used to obtain a key point set in the image.

[0031] The image feature point set is represented as: wherein, represents the two-dimensional pixel coordinates of the i-th image feature point, N I is the number of feature points, (u i ,v i ) is the pixel coordinate.

[0032] Further, an image semantic segmentation algorithm is used to extract semantic center points with stable structure meaning in the image.

[0033] The semantic center point set is represented as: wherein, is the j-th semantic target center, M I is the number of semantic points, is the image coordinate thereof.

[0034] Subsequently, edge detection and curvature analysis are performed on the point cloud data to obtain significant feature points in the lidar data.

[0035] The point cloud feature point set is defined as follows: wherein, represents the three-dimensional coordinates of the kth point cloud feature point, N L is the number of lidar feature points, is the spatial position of the point in the lidar coordinate system.

[0036] To enhance the description ability of the point cloud feature points, the local curvature value of each point is further calculated: wherein λ1, λ2, λ3 are the eigenvalues of the neighborhood covariance matrix of the point, κ k is the curvature index of the point , which is used to screen edge or prominent points.

[0037] The inertial measurement unit feature points are extracted according to the key position trajectory points recovered from the acceleration and angular velocity sequences.

[0038] A discrete time point t m is set, and the pose sequence T m is recovered in the IMU integration process, which is in the form of: wherein R m is the rotation matrix corresponding to the mth frame of IMU data, t m is the translation vector, which constitutes the transformation matrix.

[0039] The above feature points are used to construct the observation graph structure under each modality.

[0040] The graph structure is defined as an undirected weighted graph G=(V, E, W), wherein V is the node set composed of feature points, E is the connection edge, and W is the weight matrix of the edge.

[0041] The image observation graph construction process is: G I =(V I ,E I ,W I ); wherein the edge E I is connected to the adjacent nodes based on the Euclidean distance in the pixel space, and the weight is defined as a similarity function: wherein ||f i -f j|| is the distance between nodes i and j in the image space, and σ is a scale factor that controls the decay of similarity.

[0042] In the point cloud observation graph GL=(VL, EL, WL), the nodes VL correspond to the feature points of the point cloud, the edges EL are connected according to the spatial proximity, and the weights are defined according to the three-dimensional Euclidean distance.

[0043] In terms of structure, the feature extraction module includes an image feature extraction unit, a point cloud feature extraction unit, and an inertial feature extraction unit. The image feature extraction unit is connected to the output end of the camera, the point cloud feature extraction unit is connected to the output end of the laser radar, and the inertial feature extraction unit is connected to the IMU module. All of them are in communication connection with the observation graph construction module.

[0044] S3, according to the geometric and structural features between the observation graphs, the cross-modal feature matching relationship is established; Specifically, in this embodiment, the cross-modal feature matching is based on the structural relationship between the previously constructed image observation graph, point cloud observation graph and IMU observation graph.

[0045] First, the initial matching candidate set of the feature nodes between the observation graphs is established, and the nearest neighbor search strategy is used for coarse matching.

[0046] Let the set of image feature nodes be V I , the set of point cloud feature nodes be V L , and the set of IMU feature nodes be V IMU .

[0047] From any two modal graphs, the initial candidate matching pair set is represented as: wherein, and are the feature nodes of modal a and modal b respectively, dist(.,.) is the spatial or projected distance between nodes, and ∈ is the matching tolerance threshold.

[0048] Subsequently, the initial matching pairs are screened through geometric consistency constraints to judge the degree of spatial structure preservation between the matching pairs.

[0049] Let any two matching pairs be The structural preservation error is defined as: wherein, represents the Euclidean distance between nodes in modal a, is the distance of the corresponding nodes in modal b, and e ijkl represents the geometric preservation error between the pair.

[0050] The geometric consistency threshold is set, and when e ijklIf the distance is less than the threshold, the two matching pairs are considered consistent, and the combination is retained.

[0051] Further calculate the structural consistency score between all matching pairs, and construct a consistency graph G M M , where V M is the candidate matching pair, and the edge E M represents the geometric consistency between the matching pairs. M

[0052] In the consistency graph, search for the maximum weight group as the optimal matching set. This problem can be converted into maximum consistent subset extraction.

[0053] The final cross-modal matching point pair set is represented as: where, is the final feature point matching set between modal a and modal b, and N M is the number of retained matching point pairs.

[0054] In the above feature matching process, the image feature points need to be projected in three dimensions according to the camera intrinsic parameters, unified into the same coordinate system for distance calculation, and the three-dimensional projection relationship is: where p C is a three-dimensional point in the camera coordinate system, K is the camera intrinsic matrix, d is the pixel point corresponding depth, and D -1 represents the distortion correction matrix.

[0055] Structurally, the system includes a feature matching module, which includes an initial matching part, a consistency calculation part, and a maximum group extraction part.

[0056] The initial matching part is connected to the output end of the feature map construction module, the consistency calculation part performs structural consistency judgment, and the maximum group extraction part outputs the final matching result.

[0057] Logically, the matching point pairs output by the feature matching module will be directly input to the subsequent extrinsic parameter estimation step, ensuring the geometric correspondence of the input data.

[0058] The geometric consistency matching strategy can realize accurate structural alignment of cross-modal data and ensure that the matching point set has high confidence, providing a reliable foundation for matrix equation construction.

[0059] S4, based on the feature matching relationship, construct the matrix equation set for extrinsic parameter estimation; Specifically, in this embodiment, the extrinsic parameter estimation is used to optimize the geometric relationship between the cross-modal feature points to accurately determine the relative position and attitude between different sensors. ​​

[0060] Firstly, the extrinsic parameters between modal a and modal b are estimated by minimizing the re-projection error using the matched point pairs obtained in the previous step.

[0061] Set T ab as the extrinsic transformation matrix between modal a and modal b, which is represented as: where R ab is the rotation matrix, t ab is the translation vector, representing the rotation and translation transformation between modal a and modal b.

[0062] According to the matched point pairs, the extrinsic parameters are optimized by least squares method: where, is the coordinate of the i-th matched point in modal a, is the coordinate of the j-th matched point in modal b. The objective function is minimized to optimize the extrinsic parameters T ab .

[0063] Secondly, to improve the estimation accuracy, a robust estimation strategy is introduced to handle the influence of mismatched pairs. The RANSAC algorithm is used to remove inconsistent matched points to enhance the robustness of extrinsic parameter estimation.

[0064] Set the maximum number of iterations, and in each iteration, randomly select p min matched point pairs for extrinsic parameter estimation and update the current best extrinsic parameters.

[0065] Inconsistent matched points are removed by the following judgment condition: where ∈ is the re-projection error threshold. The matched point pairs that meet the condition will be used for final extrinsic parameter estimation.

[0066] To further improve the accuracy of cross-modal data registration, the global information of the observation graph is combined, and a global optimization method is used to jointly optimize all extrinsic parameter estimates. The global optimization objective is to minimize the total re-projection error between all modalities: where {T ab} represents the set of extrinsic transformation matrices between all modalities. Through joint optimization method, the global consistency between multiple modalities is ensured, further improving the accuracy of extrinsic parameter estimation.

[0067] Structurally, the system includes an extrinsic parameter estimation module, which includes an extrinsic parameter optimization unit, a robust estimation unit, and a global optimization unit.

[0068] The external parameter optimization unit calculates the external parameter according to the matching point pairs, the robust estimation unit introduces the RANSAC algorithm to remove the false matching points, the global optimization unit jointly optimizes all the external parameters, and outputs the final external parameter result.

[0069] In terms of physical structure, the external parameter estimation module is connected with the aforementioned feature matching module, uses the result output by the matching point set to perform external parameter calculation, and further provides accurate inter-modal relationships for multi-sensor data fusion and spatial positioning.

[0070] The external parameter estimation process can realize high-precision registration of different sensor data, optimize the relative attitude between modes, and provide accurate spatial conversion parameters for subsequent data fusion and positioning.

[0071] In summary, the cross-modal matching point pairs are represented as a set of three-dimensional coordinates; then a joint transformation model of the rotation matrix and the translation vector is constructed; then the transformation model is applied to each group of matching points, and the transformation error thereof is taken as a residual term; finally, the linear matrix equation set in the form of the least squares objective function is established by combining the residual terms; which is expressed in the following form: Ra i +T=b i +∈ i ; Wherein, R is a rotation matrix; T is a translation vector; a i , b i are matching feature point pairs; ∈ i is a disturbance term.

[0072] S5, a variable weight robust least squares optimization algorithm is used to solve the matrix equation set to obtain the external parameter; Specifically, in the embodiment, a variable weight robust least squares optimization algorithm is used to solve the matrix equation set, thereby obtaining the external parameter.

[0073] First, based on the aforementioned matching point pairs, the external parameter estimation problem can be converted into a least squares problem, the goal of which is to minimize the error of all matching point pairs by optimizing the transformation matrix T ab between modes. Let the external parameter matrix T ab be as shown in the S4 step.

[0074] The goal is to optimize the following error function by variable weight least squares (WLS): Wherein, and are the coordinates of the feature points under mode a and mode b respectively, w ij is the weight of each matching point pair, reflecting the reliability and signal-to-noise ratio of the matching. By minimizing the error function E, the optimal external parameter T ab can be obtained.

[0075] The optimization problem can be solved by an iterative re-weighted robust least squares method. The specific steps are as follows: Initialize the external parameters: set the initial external parameter estimate.

[0076] Weight update: calculate the weight according to the error of each matched point pair, and the larger the error, the smaller the weight.

[0077] The weight calculation formula is as follows: where σ is the standard deviation that controls the error decay, and w ij is the weight of the matched point pair.

[0078] Minimize the error function: minimize the objective function by the re-weighted least squares method.

[0079] Update the external parameters: update the external parameter matrix according to the optimization result.

[0080] Iteration: repeat the above process until the error function converges.

[0081] Finally, the optimized external parameters are obtained, which are the optimal external parameter transformation between modal a and modal b.

[0082] The objective function used by the re-weighted robust least squares optimization algorithm is: where R is the rotation matrix, representing the rotation transformation from the source sensor coordinate system to the target sensor coordinate system; T is the translation vector, representing the translation relationship between the two coordinate systems; n is the number of feature point pairs participating in optimization; a l is the lth feature point in the source sensor coordinate system; b l is the lth feature point matched with a l in the target sensor coordinate system; w i is the matching confidence; ||R|| σ is the spectral norm of the rotation matrix; λ is the regularization coefficient.

[0083] S6, error evaluation of the current estimated external parameters based on the set drift error evaluation function, and decision whether to re-execute steps S2 to S5 for external parameter update according to the evaluation result; Specifically, in the present embodiment, error evaluation of the current estimated external parameters based on the set drift error evaluation function is performed, and whether to re-execute steps S2 to S5 for external parameter update is decided according to the evaluation result.

[0084] The drift error evaluation function is used to evaluate the accuracy of the current external parameter estimate, ensuring that the external parameter update does not drift. The drift error evaluation function E drift is set, and its calculation formula is: wherein E drift is the drift error evaluation function; n is the number of feature point pairs participating in error calculation; a l is the lth feature point in the source sensor coordinate system; b l is the lth feature point matched with a i in the target sensor coordinate system; R is a rotation matrix representing the rotation relationship from the source coordinate system to the target coordinate system; T is a translation vector representing the translation relationship from the source coordinate system to the target coordinate system; when the error exceeds the set threshold, automatic re-calibration is triggered.

[0085] If E drift is greater than the set threshold ∈ drift , it is considered that the current external parameter estimation has drift and needs to re-execute steps S2 to S5 for external parameter update. The judgment condition is: ε drift > ∈ drift ; If the error is below the threshold ∈ drift , it is considered that the current external parameter estimation is accurate enough and does not need to be updated.

[0086] This process can effectively detect and control the external parameter drift, ensure that the external parameter always maintains accuracy in the multi-modal data fusion process, and avoid the influence of model drift on the final result.

[0087] In terms of system structure, the system includes a drift error evaluation module, which calculates the drift error according to the current external parameter estimation and reference data, and judges whether it needs to re-execute external parameter update. The external parameter update module executes each optimization step in steps S2 to S5, and outputs the updated external parameter.

[0088] In terms of physical connection relationship, the drift error evaluation module is connected with the external parameter estimation module to form a feedback mechanism, which adjusts the external parameter estimation through error evaluation to ensure the accuracy and stability of the external parameter.

[0089] Through the above technical method, real-time external parameter update and error control can be realized to prevent the external parameter drift from affecting the accuracy and stability of the entire system.

[0090] The multi-modal sensor external parameter automatic calibration system described below can be correspondingly referred to the multi-modal sensor external parameter automatic calibration method described above.

[0091] Please refer to the accompanying Figure 2 , the present application also provides a multi-modal sensor external parameter automatic calibration system, comprising: A data acquisition module for acquiring observation data of a plurality of heterogeneous sensors; A feature map construction module is configured to extract feature points from the observation data and construct a modal map; A matching analysis module is configured to establish a feature matching relationship between different modalities; An external parameter optimization module is configured to solve the external parameters based on a variable weight robust least square method; A drift detection module is configured to determine whether to re-perform calibration based on the error evaluation result The system of the embodiment can be used to execute the above-mentioned method embodiments, and the principles and technical effects are similar, which will not be described here.

[0092] Please refer to the accompanying Figure 3 - the accompanying Figure 4 The application further provides a computer device, comprising a processor and a memory, the memory storing a computer program executable by the processor, and the computer program is executed by the processor to perform the above method.

[0093] The application further provides a storage medium, which stores a computer program, and the computer program is executed by the processor to perform the above method.

[0094] Although the embodiments of the application have been shown and described, it is to be understood that the application is not limited to these embodiments. It will be obvious to a person skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the application, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. A method for automatic calibration of extrinsic parameters of a multi-modal sensor, characterized in that, The method comprises the following steps: S1, collecting observation data of a plurality of heterogeneous sensors; S2, extracting feature points based on the observation data, and constructing an observation graph of each sensor; S3, establishing a cross-modal feature matching relationship according to geometric and structural features between the observation graphs; S4, constructing a matrix equation set for external parameter estimation based on the feature matching relationship; S5, solving the matrix equation set by using a variable weight robust least squares optimization algorithm to obtain the external parameters; S6, performing error evaluation on the current estimated external parameters based on a set drift error evaluation function, and determining whether to re-execute steps S2 to S5 for external parameter updating according to the evaluation result.

2. The multi-modal sensor extrinsic parameter automatic calibration method of claim 1, wherein, In the S1 step, the observation data includes: three-dimensional point cloud data obtained by a laser radar; image data obtained by a camera; sensor acceleration and angular velocity data obtained by an inertial measurement unit.

3. The method of claim 1, wherein, In the S2 step, the extraction of feature points includes: extracting corner points, edge points and semantic target centers from images by image processing algorithms; extracting edge points and curvature extreme points from laser radar data by point cloud processing algorithms; extracting key position trajectory points from inertial measurement unit data; The construction of the observation graph includes the following steps: extracted feature points are taken as graph nodes, and edges are established according to the spatial distance or time sequence between the nodes; then the graph structures of the laser radar, the camera and the inertial measurement unit are constructed; and then the weight values of the edges are assigned according to the similarity measurement between the nodes.

4. The method of claim 1, wherein, In the S3 step, the cross-modal feature matching relationship is matched based on geometric consistency, and the geometric consistency is judged by calculating the distance difference of feature points in space between different observation graphs.

5. The method of claim 1, wherein, In the S4 step, the construction of the matrix equation set includes the following steps: the cross-modal matching point pairs are represented as a three-dimensional coordinate set; a joint transformation model of a rotation matrix and a translation vector is constructed; the transformation model is applied to each group of matching points, and the transformation error thereof is taken as a residual term; a linear matrix equation set in the form of a least squares objective function is established by combining the residual terms; which is expressed in the following form: Ra i + T = b i + ∈ i ; where R is a rotation matrix; T is a translation vector; a i , b i are matched feature pairs, respectively; ∈ i is a perturbation term.

6. The method of claim 1, wherein, In the S5 step, the objective function used by the variable weight robust least squares optimization algorithm is: wherein R is a rotation matrix representing the rotation transformation from the source sensor coordinate system to the target sensor coordinate system; T is a translation vector representing the translation relationship between the two coordinate systems; n is the number of feature point pairs participating in optimization; a l is the ith feature point in the source sensor coordinate system; b l is the ith feature point matched with a l in the target sensor coordinate system; w i is the matching confidence; ||R|| σ is the spectral norm of the rotation matrix; and λ is a regularization coefficient.

7. The method of claim 1, wherein, In the S6 step, the determination of whether to re-execute steps S2 to S5 includes the following steps: the average value of the residual error corresponding to the current estimated external parameters is calculated; the average residual error is compared with a preset threshold value; when the residual error exceeds the threshold value, the steps S2 to S5 are re-executed to update the external parameter results; when the residual error does not exceed the threshold value, the current external parameters are maintained unchanged and the sensor data processing process continues to run; the drift error evaluation function is: wherein E drift is the drift error evaluation function; n is the number of feature point pairs participating in error calculation; a l is the lth feature point in the source sensor coordinate system; b l is the lth feature point matched with a i in the target sensor coordinate system; R is a rotation matrix representing the rotation relationship from the source coordinate system to the target coordinate system; T is a translation vector representing the translation relationship from the source coordinate system to the target coordinate system; and the recalibration is automatically triggered when the error exceeds a set threshold.

8. The system for automatic calibration of multi-modal sensor extrinsic parameters according to any one of claims 1-7, characterized in that, including: a data acquisition module for acquiring observation data of a plurality of heterogeneous sensors; a feature graph construction module for extracting feature points from the observation data and constructing a modal graph; a matching analysis module for establishing a feature matching relationship between different modalities; an external parameter optimization module for solving external parameters based on a variable weight robust least squares method; a drift detection module for determining whether to re-execute calibration based on the error evaluation result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method of any one of claims 1-7 is implemented.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, which when executed by the processor, implements the method as claimed in any of claims 1-7.

Images