Coarse-to-fine multi-mode multi-sensor external parameter off-line calibration method

Abstract

The invention discloses a coarse-to-fine multi-mode multi-sensor external parameter off-line calibration method. The method is mainly completed through two steps of coarse calibration based on a calibration plate and fine calibration based on off-line SLAM. Firstly, a checkerboard calibration plate is adopted to extract two-dimensional feature points in a visible light camera, a PNP algorithm is utilized to calculate external parameters of the two-dimensional feature points, meanwhile, plane parameters of the calibration plate are extracted from laser radar point cloud through region growth and an RANSAC algorithm, and finally, initial external parameters between a laser radar and the camera are optimized. Secondly, in the optimization process, in combination with multiple laser radars and a vision-inertia SLAM system, the robustness of feature points is enhanced through SuperPoint feature extraction, and support is provided for the depth of a monocular camera by using local laser point cloud; and finally obtaining a high-precision multi-sensor extrinsic parameter matrix through feature fusion and global optimization of an error minimization objective function. The process not only improves the registration precision between the sensors, but also provides a solid foundation for subsequent SLAM and automatic navigation tasks.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle / unmanned vehicle navigation, and in particular relates to a coarse-to-fine multi-modal multi-sensor external parameter offline calibration method. Background Art

[0002] In the field of autonomous driving and robotics, the external parameter calibration of multi-sensor systems is crucial to achieve accurate environmental perception and positioning. Without an accurate calibration process, the data collected by different sensors cannot be effectively integrated in a unified coordinate system, which will limit the performance of unmanned systems in environmental perception, decision-making planning, and tracking control. In addition, the accuracy of calibration plays a decisive role in the effectiveness of multi-sensor data fusion systems. It directly limits the upper limit of system performance and ultimately determines the accuracy of perception and positioning of unmanned systems. However, the current multi-sensor external parameter calibration still faces some challenges. First, the target calibration algorithms are overly dependent on calibration plates or carefully designed calibration sites, and have high requirements for the calibration environment; second, the non-target calibration algorithms basically only consider the calibration of a lidar and a visible light camera, or the calibration of multiple single-modal sensors, which is difficult to expand to multi-modal sensors; at the same time, inertial measurement units are rarely added to the multi-modal multi-sensor external parameter calibration process.

[0003] In response to the above problems, the present invention aims to combine the advantages of different calibration methods such as static and dynamic, with target and without target, to achieve unified calibration of multiple sensor extrinsics of different modalities, and lay the foundation for related research on multi-modal multi-sensor fusion positioning. Summary of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention aims to provide a coarse-to-fine multi-modal multi-sensor external parameter offline calibration method. First, an external parameter calibration method based on a calibration plate (static with target) is adopted to obtain the external parameters of each visible light camera and lidar respectively as the initial value of the subsequent joint calibration; then, an external parameter calibration method of point cloud matching (static without target) is adopted to estimate the external parameters between multiple lidars; finally, an external parameter calibration method of offline SLAM (dynamic without target) is adopted to jointly optimize the external parameters between each sensor, so as to further improve the accuracy and consistency of the calibration.

[0005] In order to solve the technical problem, the technical solution of the present invention is:

[0006] A method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine, the method comprising:

[0007] S1: Based on the rough calibration of the calibration plate, the corner features of the calibration plate are extracted from the camera image and its PnP parameters are calculated. The plane parameters are extracted and optimized using the LiDAR point cloud. Finally, the preliminary external parameter matrix between the LiDAR and the camera is obtained through coordinate system matching and least squares optimization.

[0008] S2: Based on offline SLAM precision calibration, external parameter calibration is performed by extracting and matching overlapping point clouds of multiple laser radars, using SuperPoint in visual-inertial SLAM to extract 2D feature points and using local laser point clouds to estimate depth to generate sparse feature maps, matching feature points in multiple frames of data, performing joint optimization of laser-visual features and constructing an error minimization objective function. Finally, a high-precision external parameter matrix is ​​obtained through multiple iterations to improve the accuracy of multi-sensor fusion and SLAM systems.

[0009] Further, the step S1 specifically includes:

[0010] S101: extracting corner features of the calibration plate from the camera image, and using the PnP algorithm to calculate the plane equation, center position and normal vector of the calibration plate in the camera coordinate system;

[0011] S102: extract multiple planes from the laser radar point cloud, select the plane where the calibration plate is located, and use the RANSAC method to optimize its plane parameters to obtain the center position and normal vector of the calibration plate in the laser radar coordinate system;

[0012] S103: Calculate a preliminary extrinsic parameter matrix by matching the calibration plate planes in the two coordinate systems;

[0013] S104: Based on least squares optimization, minimize the distance from the lidar point cloud to the plane detected by the camera to optimize a more accurate external parameter matrix.

[0014] Further, the step S2 specifically includes:

[0015] S201: calibrating the external parameters of multiple laser radars, extracting the overlapping point cloud areas of the repeated scanning or non-repeated scanning laser radars, performing point cloud matching, and calculating the transformation matrix between the laser radars;

[0016] S202: In the visual-inertial SLAM system, the SuperPoint method is used to extract 2D feature points in the camera image, and the depth of some feature points is estimated in combination with the local laser point cloud data to generate a sparse visual feature map;

[0017] S203: Matching the visual feature points in the multi-frame data to ensure the correlation between the feature points in different frames to improve the accuracy of visual SLAM;

[0018] S204: Perform joint optimization of external parameters for laser-visual feature fusion. First, calculate the bidirectional reprojection error of the visual feature points and the matching error between the 3D visual feature points and the lidar point cloud map. Then, use SuperGlue to perform cross-frame 2D feature matching to enhance the matching accuracy of the visual feature points. On this basis, construct an error minimization objective function to optimize the external parameter matrix between the lidar, camera, and IMU.

[0019] S205: After multiple iterations of optimization, a high-precision extrinsic parameter matrix is ​​obtained, including camera-lidar extrinsic parameters, lidar-IMU extrinsic parameters, inter-lidar extrinsic parameters, and a global extrinsic parameter matrix after joint optimization; that is, the extrinsic parameters are used for multi-sensor fusion to improve the accuracy of the SLAM system and automatic navigation tasks.

[0020] Further, the extracting corner point features of the calibration plate from the camera image includes:

[0021] Visual plane feature extraction uses a checkerboard pattern as a calibration plate; first, the gradient of the image is calculated to capture the edge information of the image, which is achieved through edge detection operators such as the Sobel operator or the Canny operator. Secondly, the gradient information is used to construct a structure tensor or Hessian matrix through the outer product of the gradient. Then, the response function is calculated through the Harris corner detector to measure the possibility of each pixel in the image becoming a corner point, and the response function is non-maximum suppressed, that is, only the point with the maximum response value is retained in the local area, and the response values ​​of other points will be suppressed; finally, the appropriate corner points are screened through thresholding.

[0022] Further, the use of the PnP algorithm to calculate the plane equation, center position and normal vector of the calibration plate in the camera coordinate system includes:

[0023] A series of 3D feature points on the calibration plate And the corresponding two-dimensional feature points have Where K is the camera's intrinsic parameter matrix, T ex =[R|t] is the camera's extrinsic matrix, including the rotation matrix R and the translation vector t. The PnP algorithm solves the camera's extrinsic matrix based on the corresponding points to obtain the center position and normal vector of the calibration plate plane in the camera coordinate system.

[0024] Furthermore, the RANSAC method optimizes its plane parameters, including:

[0025] Randomly select three points in the plane point cloud and calculate the plane parameters according to the plane equation Ax+By+Cz+D=0;

[0026] Use the remaining points to calculate the distance to the plane equation. If the distance is less than the set distance threshold, the point is an interior point; otherwise, it is an exterior point. Count the number of interior points under the plane parameters.

[0027] Continue to execute the above two steps. If the number of inner points of the current plane parameter is greater than the set number threshold, update the plane parameter and save the plane parameter with the largest number of inner points.

[0028] Repeat the above three steps and iterate continuously until the iteration threshold is reached, and the plane parameters with the largest number of inliers are found. Finally, the model parameters are estimated again using the inliers to obtain the final plane parameters.

[0029] Furthermore, in step S104, the following least squares extrinsic parameter optimization problem is established:

[0030]

[0031] Among them, R and t represent the rotation matrix and translation vector of the external parameters between the lidar and the camera, respectively. represents the number of lidar and camera data pairs, Indicates the number of points in the calibration plate point cloud detected by the lidar in the i-th pair of data, Indicates that the laser radar in the i-th pair of data detects the j-th point in the calibration plate point cloud, Indicates the position of the center point of the calibration plate detected by the image in the i-th pair of data, Indicates the normal vector of the calibration plate detected by the image.

[0032] Furthermore, in step S204, the external parameter joint optimization of laser-visual feature fusion includes the following sub-steps:

[0033] For including Lidar, The multi-sensor external parameter joint optimization problem of a visible light camera and an inertial measurement unit is defined as:

[0034]

[0035]

[0036]

[0037] Among them, x i It represents the state of IMU in the world coordinate system at time i, including position and attitude. T represents the external parameters between multiple sensors, including the external parameters between each camera and lidar. External parameters between LiDARs External reference between LiDAR and IMU

[0038] During the offline optimization process, the external parameters between the two laser radars are fixed, and for the 3D visual feature points, the optimization objective function is established as:

[0039]

[0040] in represents the external parameter between the i-th camera and the lidar, represents the external parameter between the lidar and IMU, ρ(·) is the kernel function, η 1 and η 2 They represent the normalization factors of the visual bidirectional reprojection error and the laser-vision matching error, and denote the pixel coordinates of the lth feature point on the jth and kth cameras, respectively, π(·) is the camera projection function, and denote the normalized coordinates of the lth feature point on the jth and kth cameras, respectively, and Σ z represents the covariance matrix of camera observations, represents the coordinates of the lth feature point in the world coordinate system, Q l represents the nearest point of the lth feature point on the laser point cloud map, n l Indicates Q l Normal vectors of nearby point clouds, Σ Q represents the covariance matrix of the point, N P Indicates the number of visual feature points;

[0041] For the matching problem of the same feature between different cameras, that is, the visual bidirectional reprojection error term in the above optimization objective function, SuperGlue is used to match 2D features and solve the optimization objective function to obtain the optimized external parameters.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] Combining the advantages of different calibration methods such as static and dynamic, with target and without target, unified calibration of multiple sensor extrinsics of different modalities is achieved, laying the foundation for related research on multi-modal multi-sensor fusion positioning.

[0044] Firstly, the extrinsic parameter calibration method based on calibration plate (static with target) is used to obtain the extrinsic parameters of each visible light camera and lidar respectively, which are used as the initial values ​​for the subsequent joint calibration. Then, the extrinsic parameter calibration method based on point cloud matching (static without target) is used to estimate the extrinsic parameters between multiple lidars. Finally, the extrinsic parameter calibration method of offline SLAM (dynamic without target) is used to jointly optimize the extrinsic parameters between sensors to further improve the accuracy and consistency of the calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 , chessboard diagram;

[0046] Figure 2 ,PnP algorithm diagram;

[0047] Figure 3 , depth recovery map of visual feature points based on local map of laser point cloud;

[0048] Figure 4 , calibration plate plane feature extraction data flow table;

[0049] Figure 5 , laser radar-camera external parameter rough calibration data flow table;

[0050] Figure 6 , multi-lidar external parameter calibration data flow table;

[0051] Figure 7 , Visual-Inertial SLAM sparse feature map generation data flow table;

[0052] Figure 8 , the external parameters of laser-vision feature fusion jointly optimize the data flow table. DETAILED DESCRIPTION

[0053] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0054] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0055] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0056] Embodiment 1:

[0057] S1: Detect the plane parameters of the calibration plate from the lidar point cloud and the visible light camera image respectively. Use extrinsic parameters to associate the plane parameters obtained by the two sensors. Optimize the extrinsic parameter matrix between the lidar and the camera through multiple measurements to provide the initial value of the extrinsic parameter for the target-free extrinsic parameter fine calibration based on offline SLAM. The lidar-camera extrinsic parameter rough calibration based on the calibration plate includes the following sub-steps:

[0058] S1.1. Detect the plane parameters of the calibration plate from the visible light camera image and the lidar point cloud respectively. Extract the plane features of the calibration plate, such as Figure 4 As shown, it includes the following sub-steps:

[0059] S1.1.1, Visual plane feature extraction, using a checkerboard pattern as a calibration plate (see Appendix Figure 1 ). First, the gradient of the image is calculated to capture the edge information of the image. The gradient is a measure of the change in image brightness, which is usually implemented through edge detection operators such as the Sobel operator or the Canny operator. Secondly, the gradient information is used to construct the structure tensor (or Hessian matrix) through the outer product of the gradient. The structure tensor is a second-order derivative matrix used to describe the texture characteristics of the local area of ​​the image. Then, the response function is calculated through the Harris corner detector to measure the possibility of each pixel in the image becoming a corner point. In order to improve the accuracy of corner detection, it is necessary to perform non-maximum suppression on the response function, that is, only the points with the maximum response value are retained in the local area, and the response values ​​of other points will be suppressed. Finally, the appropriate corner points are screened through thresholding.

[0060] For a series of 3D feature points on the calibration plate And the corresponding two-dimensional feature points have Where K is the camera's intrinsic parameter matrix, T ex = [R|t] is the camera's external parameter matrix, including the rotation matrix R and the translation vector t. The PnP algorithm solves the camera's external parameter matrix based on these corresponding points, and then obtains the center position and normal vector of the calibration plate plane in the camera coordinate system, as shown in the attached figure. Figure 2 shown.

[0061] S1.1.2. Laser point cloud plane feature extraction

[0062] In the calibration scenario, due to the existence of multiple planes, it is impossible to use a single plane fitting method to obtain the unknown number of plane parameters. Therefore, the plane segmentation method based on region growing is used to obtain the parameters of multiple planes, and then the plane where the calibration plate is located is selected as the target plane. Finally, the RANSAC method is used to optimize the plane parameters to obtain the center position and normal vector of the calibration plate.

[0063] The plane segmentation method based on region growing first calculates the curvature and normal vector of each point in the point cloud. Since the point with small curvature indicates that the point is on a certain plane, the segmentation starts from the point with the smallest curvature and expands outward, and the point is added to the candidate point set. Then, for each point in the set, the neighboring points of the point are searched, and the angle difference between the neighboring point and the normal vector of the current point is determined. If the angle is less than the threshold, the neighboring point is added to the current plane. At the same time, if the curvature of the neighboring point is less than the threshold, the neighboring point is added to the candidate point set. After all the neighboring points of the current point are traversed, the current point is deleted from the candidate point set. When the candidate point set is empty, it means that the plane segmentation is completed. The algorithm flow is as follows:

[0064]

[0065] After obtaining a list of all planes in the environment, select the plane closest to the LiDAR as the calibration plane, and use the RANSAC algorithm to fit the plane model to obtain the plane parameters. The steps of the RANSAC plane fitting algorithm are as follows:

[0066] a) Randomly select three points in the plane point cloud and calculate the plane parameters according to the plane equation Ax+By+Cz+D=0;

[0067] b) Use the remaining points to calculate the distance to the plane equation. If the distance is less than the set distance threshold, the point is an interior point; otherwise, it is an exterior point, and the number of interior points under the plane parameters is counted;

[0068] c) Continue to execute the above two steps. If the number of inner points of the current plane parameter is greater than the set number threshold, update the plane parameter and save the plane parameter with the largest number of inner points;

[0069] d) Repeat the above three steps and iterate continuously until the iteration threshold is reached, and the plane parameters with the largest number of inliers are found. Finally, the inliers are used to estimate the model parameters again to obtain the final plane parameters.

[0070] S1.2, LiDAR-Camera extrinsic parameters rough calibration, such as Figure 5 As shown;

[0071] In the previous section, the camera and lidar respectively extracted the plane parameters of the calibration plate plane in their respective sensor coordinate systems. These two plane parameters can be associated through the external parameters between the lidar and the camera. Since there is a certain error in the way the center position of the calibration plate is calculated by the calibration plate point cloud, the external parameters between the lidar and the camera are optimized by minimizing the distance from each point in the calibration plate point cloud to the calibration plate plane detected by the camera. Considering the possible measurement errors of the lidar point cloud, it is necessary to exclude points far from the plane and only use points close to the plane for external parameter optimization. Based on multiple measurements, the following least squares external parameter optimization problem is established:

[0072]

[0073] Among them, R and t represent the rotation matrix and translation vector of the external parameters between the lidar and the camera, respectively. represents the number of lidar and camera data pairs, Indicates the number of points in the calibration plate point cloud detected by the lidar in the i-th pair of data, Indicates that the laser radar in the i-th pair of data detects the j-th point in the calibration plate point cloud, Indicates the position of the center point of the calibration plate detected by the image in the i-th pair of data, Indicates the normal vector of the calibration plate detected by the image.

[0074] S2: This section uses FAST-LIO2 and VINS-Mono as the laser-inertial SLAM system and visual-inertial SLAM system frameworks, respectively, and uses Super-Point to replace the front-end feature point extraction part of VINS-Mono. In this process, first, dense laser point cloud maps and sparse visual feature maps are constructed respectively, and some visual 2D features are initialized using local laser point clouds. Secondly, SuperGlue is used to match 2D feature points between different cameras, and then the 3D visual features are optimized according to the odometer and external parameter information. Finally, the 3D visual features are aligned with the lidar point cloud map. After multiple iterations, the optimization results of multi-sensor external parameters are obtained. Targetless multi-sensor external parameter precision calibration based on offline SLAM includes the following sub-steps:

[0075] S2.1. In the calibration of the external parameters of multiple laser radars, different laser radars can be divided into two categories according to the scanning method: repeated scanning and non-repeated scanning. For the laser radar with repeated scanning, the accumulated point cloud obtained by leaving the sensor still for a period of time is used as the data to be calibrated. For the laser radar with non-repeated scanning, the single-frame point cloud information is used as the data to be calibrated. Then, the overlapping areas in space are selected and the point clouds in the area are used for matching. The final transformation matrix is ​​the external parameter between each laser radar, such as Figure 6 shown.

[0076] S2.2, sparse visual feature map generation for visual-inertial SLAM, such as Figure 7 As shown;

[0077] Since offline SLAM is used for optimization, the accuracy of offline visual-inertial SLAM can be improved by increasing the number of features in the visual front end. Based on VINS-MONO, the front-end feature point extraction method is replaced with SuperPoint. Compared with the traditional feature point extraction method, SuperPoint can effectively deal with problems such as image rotation, scaling, brightness changes, etc., and has stronger robustness.

[0078] Compared with binocular cameras and RGB-D cameras, monocular cameras have weaker performance in recovering the depth of feature points. Therefore, in the same way as LVI-SAM, a local laser point cloud map is introduced to provide depth information for some two-dimensional features of the monocular camera. When the visual inertial system is initialized, the estimated odometer is used to construct a local map of the laser point cloud within a period of time and project it onto the unit sphere centered on the camera. At the same time, polar coordinates and a two-dimensional KD tree are used to represent the local map. For a visual feature point, the three closest map points are searched on the unit sphere. If the depths of these three map points are close, the depth of the current visual feature is the average of the depths of these three map points, as shown in the attached figure. Figure 3 shown.

[0079] S2.3, joint optimization of external parameters for laser-visual feature fusion, e.g. Figure 8 As shown, it includes the following sub-steps:

[0080] For including Lidar, The multi-sensor external parameter joint optimization problem of a visible light camera and an inertial measurement unit is defined as:

[0081]

[0082]

[0083]

[0084] Among them, x i It represents the state of IMU in the world coordinate system at time i, including position and attitude. T represents the external parameters between multiple sensors, including the external parameters between each camera and lidar. External parameters between LiDARs External reference between LiDAR and IMU

[0085] In the offline optimization process, the external parameters between the two laser radars are fixed. For the 3D visual feature points, they are not only observed by different cameras at different times, but also need to be matched with the laser point cloud map. Therefore, the optimization objective function is established as:

[0086]

[0087] in represents the external parameter between the i-th camera and the lidar, represents the external parameter between the lidar and IMU, ρ(·) is the kernel function, η 1 and η 2 They represent the normalization factors of the visual bidirectional reprojection error and the laser-vision matching error, and denote the pixel coordinates of the lth feature point on the jth and kth cameras, respectively, π(·) is the camera projection function, and denote the normalized coordinates of the lth feature point on the jth and kth cameras, respectively, and Σ z represents the covariance matrix of camera observations, represents the coordinates of the lth feature point in the world coordinate system, Q l represents the nearest point of the lth feature point on the laser point cloud map, n l Indicates Q l Normal vectors of nearby point clouds, Σ Q represents the covariance matrix of the point, N P Represents the number of visual feature points.

[0088] For the matching problem of the same feature between different cameras, that is, the visual bidirectional reprojection error term in the aforementioned optimization objective function, SuperGlue is used to match these 2D features. By solving the optimization objective function, the optimized external parameters can be obtained.

[0089] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0094] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A coarse-to-fine multi-modal multi-sensor extrinsic parameter offline calibration method, characterized in that: The method comprises: S1: Based on the rough calibration of the calibration plate, the corner features of the calibration plate are extracted from the camera image and its PnP parameters are calculated. The plane parameters are extracted and optimized using the LiDAR point cloud. Finally, the preliminary external parameter matrix between the LiDAR and the camera is obtained through coordinate system matching and least squares optimization. S2: Based on offline SLAM precision calibration, external parameter calibration is performed by extracting and matching overlapping point clouds of multiple laser radars, using SuperPoint in visual-inertial SLAM to extract 2D feature points and using local laser point clouds to estimate depth to generate sparse feature maps, matching feature points in multiple frames of data, performing joint optimization of laser-visual features and constructing an error minimization objective function. Finally, a high-precision external parameter matrix is ​​obtained through multiple iterations to improve the accuracy of multi-sensor fusion and SLAM systems.

2. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 1, characterized in that: The step S1 specifically includes: S101: extracting corner features of the calibration plate from the camera image, and using the PnP algorithm to calculate the plane equation, center position and normal vector of the calibration plate in the camera coordinate system; S102: extract multiple planes from the laser radar point cloud, select the plane where the calibration plate is located, and use the RANSAC method to optimize its plane parameters to obtain the center position and normal vector of the calibration plate in the laser radar coordinate system; S103: Calculate a preliminary extrinsic parameter matrix by matching the calibration plate planes in the two coordinate systems; S104: Based on least squares optimization, minimize the distance from the lidar point cloud to the plane detected by the camera to optimize a more accurate external parameter matrix.

3. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 1, characterized in that: The step S2 specifically includes: S201: calibrating the external parameters of multiple laser radars, extracting the overlapping point cloud areas of the repeated scanning or non-repeated scanning laser radars, performing point cloud matching, and calculating the transformation matrix between the laser radars; S202: In the visual-inertial SLAM system, the SuperPoint method is used to extract 2D feature points in the camera image, and the depth of some feature points is estimated in combination with the local laser point cloud data to generate a sparse visual feature map; S203: Matching the visual feature points in the multi-frame data to ensure the correlation between the feature points in different frames to improve the accuracy of visual SLAM; S204: Perform joint optimization of external parameters for laser-visual feature fusion. First, calculate the bidirectional reprojection error of the visual feature points and the matching error between the 3D visual feature points and the lidar point cloud map. Then, use SuperGlue to perform cross-frame 2D feature matching to enhance the matching accuracy of the visual feature points. On this basis, construct an error minimization objective function to optimize the external parameter matrix between the lidar, camera, and IMU. S205: After multiple iterations of optimization, a high-precision extrinsic parameter matrix is ​​obtained, including camera-lidar extrinsic parameters, lidar-IMU extrinsic parameters, inter-lidar extrinsic parameters, and a global extrinsic parameter matrix after joint optimization; that is, the extrinsic parameters are used for multi-sensor fusion to improve the accuracy of the SLAM system and automatic navigation tasks.

4. The method for off-line calibration of multi-modal multi-sensor external parameters from coarse to fine according to claim 2, characterized in that: The step of extracting corner point features of the calibration plate from the camera image comprises: Visual plane feature extraction uses a checkerboard pattern as a calibration plate; first, the gradient of the image is calculated to capture the edge information of the image, which is achieved through edge detection operators such as the Sobel operator or the Canny operator. Secondly, the gradient information is used to construct a structure tensor or Hessian matrix through the outer product of the gradient. Then, the response function is calculated through the Harris corner detector to measure the possibility of each pixel in the image becoming a corner point, and the response function is non-maximum suppressed, that is, only the point with the maximum response value is retained in the local area, and the response values ​​of other points will be suppressed; finally, the appropriate corner points are screened through thresholding.

5. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 2, characterized in that: The use of the PnP algorithm to calculate the plane equation, center position and normal vector of the calibration plate in the camera coordinate system includes: A series of 3D feature points on the calibration plate And the corresponding two-dimensional feature points have Where K is the camera's intrinsic parameter matrix, T ex =[R|t] is the camera's extrinsic matrix, including the rotation matrix R and the translation vector t. The PnP algorithm solves the camera's extrinsic matrix based on the corresponding points to obtain the center position and normal vector of the calibration plate plane in the camera coordinate system.

6. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 2, characterized in that: The RANSAC method optimizes its plane parameters, including: Randomly select three points in the plane point cloud and calculate the plane parameters according to the plane equation Ax+By+Cz+D=0; Use the remaining points to calculate the distance to the plane equation. If the distance is less than the set distance threshold, the point is an interior point; otherwise, it is an exterior point. Count the number of interior points under the plane parameters. Continue to execute the above two steps. If the number of inner points of the current plane parameter is greater than the set number threshold, update the plane parameter and save the plane parameter with the largest number of inner points. Repeat the above three steps and iterate continuously until the iteration threshold is reached, and the plane parameters with the largest number of inliers are found. Finally, the model parameters are estimated again using the inliers to obtain the final plane parameters.

7. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 2, characterized in that: In step S104, the following least squares extrinsic parameter optimization problem is established: Among them, R and t represent the rotation matrix and translation vector of the external parameters between the lidar and the camera, respectively. represents the number of lidar and camera data pairs, Indicates the number of points in the calibration plate point cloud detected by the lidar in the i-th pair of data, Indicates that the laser radar in the i-th pair of data detects the j-th point in the calibration plate point cloud, Indicates the position of the center point of the calibration plate detected by the image in the i-th pair of data, Indicates the normal vector of the calibration plate detected by the image.

8. The method for off-line calibration of multi-modal multi-sensor extrinsic parameters from coarse to fine according to claim 3, characterized in that: In step S204, the external parameter joint optimization of laser-visual feature fusion includes the following sub-steps: For including Lidar, The multi-sensor external parameter joint optimization problem of a visible light camera and an inertial measurement unit is defined as: Among them, x i It represents the state of IMU in the world coordinate system at time i, including position and attitude. T represents the external parameters between multiple sensors, including the external parameters between each camera and lidar. External parameters between LiDARs External reference between LiDAR and IMU During the offline optimization process, the external parameters between the two laser radars are fixed, and for the 3D visual feature points, the optimization objective function is established as: in represents the external parameter between the i-th camera and the lidar, represents the external parameter between the lidar and the IMU, ρ(·) is the kernel function, η1 and η2 represent the normalization factors of the visual bidirectional reprojection error and the laser-vision matching error, respectively. and denote the pixel coordinates of the lth feature point on the jth and kth cameras, respectively, π(·) is the camera projection function, and denote the normalized coordinates of the lth feature point on the jth and kth cameras, respectively, and Σ z represents the covariance matrix of camera observations, represents the coordinates of the lth feature point in the world coordinate system, Q l represents the nearest point of the lth feature point on the laser point cloud map, n l Indicates Q l Normal vectors of nearby point clouds, Σ Q represents the covariance matrix of the point, N P Indicates the number of visual feature points; For the matching problem of the same feature between different cameras, that is, the visual bidirectional reprojection error term in the above optimization objective function, SuperGlue is used to match 2D features and solve the optimization objective function to obtain the optimized external parameters.

Images