A Fast Joint Calibration Method for LiDAR and GPS-RTK

Abstract

The present invention relates to the technical field of multi-sensor joint calibration, and is a fast joint calibration method for lidar and GPS-RTK. The specific method includes: automatically obtaining the initialized structural external parameter value by using a structural external parameter self-checking device as the initial calibration value; then, extracting geometric features in the point cloud through a depth feature extraction network, performing front-back frame matching, and calculating the mileage; dynamically adjusting the data selection logic according to the GPS-RTK and mileage data, and assigning different weights to different points; finally, precisely solving the structural external parameter and the system external parameter by constructing a weighted minimization error optimization problem. The present invention reduces the dependence on manually calibrated points, reduces the complexity, significantly improves the calibration efficiency and accuracy, and is applicable to scenarios with high-precision positioning requirements such as autonomous driving and UAV navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor joint calibration, and particularly to a fast joint calibration method for lidar and GPS-RTK. Background Art

[0002] With the development of autonomous driving technology, accurate vehicle positioning and environmental perception have become crucial. Lidar (LiDAR) and Global Positioning System Real-Time Kinematic (GPS-RTK) are two sensor technologies widely used in vehicle positioning and environmental modeling. Lidar can provide high-precision three-dimensional environmental data, while GPS-RTK can provide centimeter-level positioning accuracy. However, existing calibration methods for lidar and GPS-RTK have some limitations. First, traditional calibration methods are time-consuming and usually require specific calibration sites and markers, which not only increase the calibration cost but also limit the flexibility of the calibration process. Second, existing calibration methods often require complex mathematical models and a large amount of manual adjustment, which not only increases the technical threshold but also affects the automation level of the calibration process.

[0003] In the existing disclosed invention technologies, for example, the patent with the publication number CN109597054A discloses a calibration method for lidar. The calibration method includes: obtaining distance measurement values and corresponding reflectivity measurement values of a calibration object at multiple calibration distance values by using the lidar; splitting the distance measurement values and corresponding reflectivity measurement values into at least two groups according to the magnitudes of the distance measurement values; further splitting the distance measurement values and corresponding reflectivity measurement values in each group into at least two subgroups according to the magnitudes of the reflectivity measurement values in each group; and respectively fitting an error correction function with the reflectivity measurement value as the input variable according to the error value between the distance measurement value in each subgroup and the corresponding calibration distance value and the corresponding reflectivity measurement value.

[0004] The above patent processing flow is relatively complex in data calculation. Especially when dealing with large-scale data, when multiple splits and fittings are performed, it will result in a high computational overhead and has the problems described in the background art. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a fast joint calibration method for lidar and GPS-RTK to solve the above technical problems. The technical solution of the fast joint calibration method for lidar and GPS-RTK includes the following steps:

[0006] S1: Automatically identify the positions of the lidar sensor and the GPS-RTK antenna through the structural external parameter self-checking device, and perform the initialization calculation of the structural external parameters;

[0007] S2: Obtain information of fixed features in the operating environment through lidar, and simultaneously record and analyze multi-modal data during the driving process of the moving platform;

[0008] S3: Extract the real-time driving environment conditions of the moving platform, and dynamically adjust the processing window and threshold of lidar data and GPS-RTK data according to the real-time driving environment conditions to obtain valid data in the real-time driving environment conditions;

[0009] S4: Select lidar mileage point - north-east-up coordinate point pairs in the valid data according to the distance selection algorithm, calculate the preliminary system extrinsic parameters through the plane calibration algorithm, construct an optimization problem for minimizing the error, and perform non-linear optimization to iteratively solve the system extrinsic parameters and the structural extrinsic parameters;

[0010] S5: According to the system extrinsic parameters and the structural extrinsic parameters, complete the joint calibration of lidar and GPS-RTK, obtain the structural extrinsic parameters from the lidar to the GPS-RTK installation position and the system extrinsic parameters of the north-east-up coordinate system initialized by GPS-RTK and the earth-centered earth-fixed coordinate system, and complete the mutual conversion processing between the absolute geographic coordinates and the lidar mileage;

[0011] The process of initializing and calculating the structural extrinsic parameters includes:

[0012] S131: Identify the lidar and the GPS-RTK antenna in the position image of the lidar sensor through a convolutional neural network, and calculate the three-dimensional coordinates of the lidar and the GPS-RTK through the pixel coordinates, the actual depth of the pixel, and the camera intrinsic parameters;

[0013] S132: Import the three-dimensional coordinates of the GPS-RTK into the relative pose calculation strategy to calculate and obtain the relative pose between the lidar and the GPS-RTK antenna;

[0014] S133: Extract the relative pose between the lidar and the GPS-RTK antenna to form the initial value of the structural extrinsic parameters.

[0015] Specifically, S1 includes the following specific steps:

[0016] S11: Install the lidar and the GPS-RTK on the moving platform, and operate the moving platform under the structural extrinsic parameter self-checking device, where the structural extrinsic parameter self-checking device consists of a depth camera suspended above the moving platform and its connected AI calculation unit;

[0017] Specifically, the suspension height of the depth camera is determined according to the height of the moving platform. Exemplarily, in this embodiment, when the position image captured by the depth camera covers both the lidar and the GPS-RTK antenna at the same time, it is the minimum suspension height of the depth camera.

[0018] S12: Obtain the position image of the lidar sensor on the moving platform through the depth camera;

[0019] S13: Automatically extract the position image of the lidar sensor through an AI algorithm and perform initialization calculation of the extrinsic parameters of the structure.

[0020] Specifically, in S11, the AI calculation unit includes: the convolutional neural network used is a trained object detection network or semantic segmentation network, which is used to detect or segment two types of objects, namely the lidar and the GPS-RTK antenna.

[0021] Furthermore, in S131, the calculation strategy of the three-dimensional coordinates is specifically as follows:

[0022]

[0023] Among them, z is the converted metric depth, obtained from the depth map of the depth camera;

[0024] (q, p) are the pixel coordinates of the lidar and the GPS-RTK;

[0025] (x, y, z) are the physical coordinates in the depth camera coordinate system, that is, the physical coordinates of the lidar and the GPS-RTK;

[0026] f x , f y are the focal lengths of the depth camera in the horizontal and vertical directions respectively;

[0027] C x , C y is the camera principal point.

[0028] Specifically, S2 includes the following steps:

[0029] S21: Extract the real-time driving environment conditions of the moving platform. When the moving platform is close to the feature object, if the GPS-RTK signal is interfered, execute step S3; otherwise, execute step S22;

[0030] S22: Move the moving platform and obtain the information of the fixed feature objects in the running environment through the lidar. Among them, the fixed feature object information includes: the edge, corner and surface data of the feature object;

[0031] Furthermore, the fixed feature object information also includes: the outer facade or outer corners of the building, the publicity signs, the utility poles, the trees, the stationary vehicles;

[0032] S23: Simultaneously record the multi-modal data during the driving process of the moving platform. The multi-modal data includes: the lidar point cloud, the GPS-RTK longitude and latitude information, and the motion trajectory data of the IMU;

[0033] S24: Extract the geometric features of the lidar point cloud in real time through a deep feature extraction network, and calculate the inter-frame lidar odometry data by combining the motion trajectory data of the IMU.

[0034] Furthermore, S24 includes the following specific steps:

[0035] S241: Introduce a feature extraction network to automatically extract the local and global geometric features of the point cloud from the geometric features of the lidar point cloud;

[0036] Among them, the feature extraction network is one of a deep convolutional neural network and a graph neural network;

[0037] S242: Use the Transformer network for global feature matching of inter-frame point clouds;

[0038] S243: Adopt a multi-scale feature fusion mechanism to extract point cloud features at different scales;

[0039] S244: Use GPU acceleration to perform the inference process of the deep learning network to complete the real-time matching of inter-frame point clouds;

[0040] S245: After completing the inter-frame matching, extract the six-degree-of-freedom transformation between two frames of point clouds as the lidar odometry data for subsequent external parameter calculation and optimization, where the six degrees of freedom include: three translation amounts and three rotation amounts.

[0041] Specifically, S3 includes the following specific steps:

[0042] S31: Extract the real-time driving environment conditions of the moving platform, where the real-time driving environment conditions include noise characteristics, laser characteristics, and GPS signal strength characteristics;

[0043] The noise characteristic is the lidar point cloud density;

[0044] The laser characteristic is the average distance between points in the feature area;

[0045] The GPS signal strength characteristic is the GPS status bit and the number of satellites;

[0046] S32: Preset the threshold range of the lidar point cloud density, the threshold range of the average distance between points in the feature area, the threshold of the GPS status bit, and the threshold of the number of satellites, synchronously preset the initial threshold range, and screen the real-time driving environment conditions according to the above threshold ranges, where the initial threshold range is 25 - 30;

[0047] S33: When the noise characteristic exceeds the threshold range of the lidar point cloud density, increase the time window of the lidar data;

[0048] When the laser feature exceeds the threshold range of the average distance between points within the feature region, it is determined that the data is unavailable and rejection processing is performed;

[0049] When the GPS signal strength feature is simultaneously less than the threshold of the GPS status bit and the threshold of the number of satellites, increase the time window of the GPS data and increase the threshold value when the lidar data is matched with the GPS-RTK data; the threshold value when the lidar data is matched with the GPS-RTK data includes: the distance threshold and the error threshold in the ICP algorithm;

[0050] S34: According to S32 - S33, mark the real-time driving environment conditions after the screening process as valid data, and adaptively adjust the time window size for the acquisition process of the valid data.

[0051] Furthermore, S3 also includes the following specific steps:

[0052] S35: When the GPS signal strength feature is simultaneously less than the threshold of the GPS status bit and the threshold of the number of satellites or the noise feature exceeds the threshold range of the lidar point cloud density, adjust and record the weight distribution of the sensor data in real time, and provide weight constraints for subsequent optimization calculations, where the weight distribution strategy specifically includes:

[0053] w = w gps × w lidar-noise × w feature ;

[0054] where, w is the total weight value; w gps is the GPS-RTK weight factor;

[0055] num star is the real-time number of GPS-RTK satellites in view;

[0056] w lidar-noise is the lidar noise weight factor; w feature is the lidar feature saliency weight factor.

[0057] Specifically, S4 includes the following specific steps:

[0058] S41: Through the coordinate transformation algorithm, complete the conversion of the longitude and latitude data to the northeast celestial coordinate system, obtain the lidar mileage point - northeast celestial coordinate point pair, and select the lidar mileage point - northeast celestial coordinate point pair in the valid data according to the distance selection algorithm; where, the distance selection algorithm specifically screens the points through the distance between point pairs, combined with the constraint condition that the three points are not collinear pairwise; the number of lidar mileage point - northeast celestial coordinate point pairs in the valid data is greater than or equal to 4;

[0059] S42: Calculate the preliminary external parameters of the system through the planar calibration algorithm.

[0060] Furthermore, in S41, the coordinate transformation algorithm specifically includes:

[0061] S411: Convert the longitude and latitude data to the Earth-centered Earth-fixed rectangular coordinate system as follows:

[0062]

[0063] where (X, Y, Z) are the coordinates in the Earth-centered Earth-fixed rectangular coordinate system after conversion;

[0064] (lon, lat, alt) are the longitude and latitude coordinates in the geodetic coordinate system; e is the eccentricity of the Earth ellipsoid; N is the radius of curvature of the reference ellipsoid;

[0065] S412: Extract the Earth-centered Earth-fixed rectangular coordinate system and convert it to the northeast-up coordinate system, specifically including: calculating the northeast-up coordinates (e, n, u) of (X, Y, Z) under the coordinate origin with coordinates (x 0 , y 0 , z 0 );

[0066]

[0067]

[0068] where the geodetic height of the coordinate origin with coordinates (x 0 , y 0 , z 0 ) is (lon0, lat0, alt0);

[0069] (Δx, Δy, Δz) is the coordinate transformation difference matrix.

[0070] Specifically, in S42, the planar calibration algorithm includes:

[0071] S421: Make assumptions before solving, where the assumptions before solving include: only considering the heading information, assuming that the origin of the translation amount coincides, and making a planar assumption for the rotation amount;

[0072] S422: Translate the mileage information and the northeast-up coordinates respectively through the de-centered coordinates.

[0073] S423: Use two segments of trajectory points to construct an iterative closest point problem, convert the optimization problem of minimizing the error to be solved into an optimization problem of point pair fitting, solve the rotation matrix R through SVD decomposition, and extract the heading angle from R.

[0074] Specifically, in S423, the pre - construction data for the optimization problem of minimizing the error to be solved includes: based on the lidar trajectory and the GPS - RTK trajectory, combined with the point weights determined in S3; the optimization algorithms used in the optimization problem include: the least - squares method and the Ceres optimization library.

[0075] A computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it causes the computer to execute the described fast joint calibration method for lidar and GPS - RTK.

[0076] An electronic device includes a memory, a processor, and instructions stored on the memory and executable on the processor. When the processor executes the instructions, it implements the above - described fast joint calibration method for lidar and GPS - RTK.

[0077] Compared with the prior art, the present invention according to the above - mentioned solution has the following beneficial effects: A fast joint calibration method for lidar and GPS - RTK uses lidar data and the Fast - LIO algorithm for dead reckoning to generate continuous relative displacement information. The lidar odometry information is converted into GPS - RTK odometry data through rough extrinsic parameters of the structure, and conditional planar calibration is performed with the north - east - down data converted by GPS - RTK to obtain the initial value of the system extrinsic parameters. Then, the lidar odometry data is matched with the north - east - down coordinates, and an optimization problem is constructed to iteratively approximate the accurate system extrinsic parameters and structural extrinsic parameters from the initial values of the system and structural extrinsic parameters, thereby achieving high - precision joint calibration of the two. Compared with traditional methods, the present invention reduces the dependence on manually marked points, reduces complexity, and significantly improves the efficiency and accuracy of calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0079] Among them:

[0080] Figure 1 is a schematic flowchart of a fast joint calibration method for lidar and GPS - RTK of the present invention;

[0081] Figure 2 is a right - hand view of the lidar installation in a fast joint calibration method for lidar and GPS - RTK of the present invention;

[0082] Figure 3It is a top view of the lidar installation and a schematic diagram of the motion trajectory of the moving platform in a fast joint calibration method of lidar and GPS-RTK according to the present invention.

[0083] Reference numerals:

[0084] 1. Fixed top surface; 2. Depth camera; 3. GPS antenna; 4. Lidar; 5. Moving platform; 6. Motion trajectory. Specific embodiments

[0085] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings of the specification.

[0086] Many specific details are set forth in the following description in order to fully understand the present invention, but the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0087] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The appearances of "in one embodiment" in different places in this specification do not all refer to the same embodiment, nor are they separate or alternative embodiments that exclude each other from other embodiments.

[0088] Embodiment 1

[0089] As Figure 1 shown, a fast joint calibration method of lidar and GPS-RTK according to an embodiment of the present invention, as Figure 1 shown, includes the following specific steps:

[0090] S1: Automatically identify the positions of the lidar sensor and the GPS-RTK antenna through the external structure self-checking device, and perform the initialization calculation of the external structure parameters;

[0091] S1 includes the following specific steps:

[0092] S11: Install the lidar and the GPS-RTK on the moving platform, and operate the moving platform under the external structure self-checking device. Among them, the external structure self-checking device consists of a depth camera suspended above the moving platform and its connected AI calculation unit;

[0093] Exemplarily, in this embodiment, the selected lidar model is MID360, which can perform 360° non-repetitive scanning and has an IMU integrated inside. It is installed on a passenger car, and the passenger car is placed in a scene with obvious fixed features; in this embodiment, the scene with obvious fixed features is selected as a scene with buildings, trees, and stationary vehicles;

[0094] Specifically, the suspension height of the depth camera is determined according to the height of the moving platform. Exemplarily, in this embodiment, when the position image captured by the depth camera covers both the lidar and the GPS-RTK antenna at the same time, it is the minimum suspension height of the depth camera. In this embodiment, it should be noted that the moving platform needs to be at a certain distance from buildings or trees that can significantly block the GPS signal to avoid interference with the GPS-RTK signal during the calibration process. In this embodiment, this distance is selected as 5m.

[0095] In addition, in this embodiment, during the installation of the lidar and the RTK antenna, it is necessary to ensure that the centers of the lidar and the RTK antenna are on the vehicle's central axis and at the same height plane;

[0096] S12: Obtain the position image of the lidar sensor on the moving platform through the depth camera;

[0097] S13: Automatically extract the position image of the lidar sensor through the AI algorithm and perform the initialization calculation of the extrinsic parameters of the structure.

[0098] In S11, the AI calculation unit includes: the convolutional neural network used is a trained object detection network or semantic segmentation network, which is used to detect or segment two types of objects, namely the lidar and the GPS-RTK antenna.

[0099] S13 also includes the following specific steps:

[0100] S131: Identify the lidar and the GPS-RTK antenna in the position image of the lidar sensor through the convolutional neural network, and calculate the three-dimensional coordinates of the lidar and the GPS-RTK through the pixel coordinates, the actual depth of the pixel, and the camera intrinsic parameters;

[0101] S132: Import the three-dimensional coordinates of the GPS-RTK into the relative pose calculation strategy to calculate and obtain the relative pose between the lidar and the GPS-RTK antenna;

[0102] S133: Extract the relative pose between the lidar and the GPS-RTK antenna to form the initial value of the extrinsic parameters of the structure.

[0103] In S131, the calculation strategy of the three-dimensional coordinates is specifically:

[0104]

[0105] Among them, z is the converted metric depth, which is obtained from the depth map of the depth camera;

[0106] (q, p) are the pixel coordinates of the lidar and GPS-RTK;

[0107] (x, y, z) are the physical coordinates in the depth camera coordinate system, that is, the physical coordinates of the lidar and GPS-RTK;

[0108] f x , f y are the focal lengths of the depth camera in the horizontal and vertical directions respectively;

[0109] C x , C y is the principal point of the camera.

[0110] S2: Obtain the information of fixed features in the operating environment through the lidar, and record and analyze multi-modal data during the driving process of the mobile platform;

[0111] S2 includes the following specific steps:

[0112] S21: Extract the real-time driving environment conditions of the mobile platform. When the mobile platform is close to the feature, if the GPS-RTK signal is interfered, execute step S3, otherwise execute step S22;

[0113] S22: Move the mobile platform to obtain the information of fixed features in the operating environment through the lidar. Among them, the fixed feature information includes: the edge, corner and surface data of the feature;

[0114] Furthermore, the fixed feature information also includes: the outer facade or outer corner of the building, the publicity sign, the telegraph pole, the tree, the stationary vehicle;

[0115] S23: Record the multi-modal data during the driving process of the mobile platform at the same time. The multi-modal data includes: lidar point cloud, GPS-RTK longitude and latitude information and the motion trajectory data of the IMU;

[0116] S24: Extract the geometric features of the lidar point cloud in real time through the depth feature extraction network, and assist in calculating the inter-frame lidar odometry data by combining the motion trajectory data of the IMU.

[0117] S24 includes the following specific steps:

[0118] S241: Introduce the feature extraction network to automatically extract the local and global geometric features of the point cloud from the geometric features of the lidar point cloud;

[0119] Among them, the feature extraction network is one of a deep convolutional neural network and a graph neural network;

[0120] Exemplarily, in this embodiment, the feature extraction network is FCGF or SuperPoint;

[0121] S242: Use a Transformer network to perform global feature matching of inter-frame point clouds; among them, the Transformer network is a deep learning architecture widely used in the field of natural language processing. The Transformer architecture ensures high matching accuracy in sparse, repetitive texture, or uneven point cloud density scenarios by capturing long-range dependencies between point cloud frames;

[0122] S243: Adopt a multi-scale feature fusion mechanism to extract point cloud features at different scales;

[0123] In this embodiment, a multi-scale feature fusion mechanism is adopted to extract point cloud features at different scales, ensuring that both low-resolution features at a long distance and high-resolution features at a short distance can be processed, thereby optimizing the detailed matching between frames and maintaining the accuracy and consistency of feature extraction in a large-scale scene;

[0124] S244: Use GPU acceleration to perform the inference process of the deep learning network to complete the real-time matching of inter-frame point clouds;

[0125] In this embodiment, through efficient parallel computing, the extraction and matching processes of point cloud features are accelerated. Especially in large-scale point cloud data, the overall calibration calculation efficiency can be significantly improved.

[0126] S245: After completing the inter-frame matching, extract the six-degree-of-freedom transformation between two frames of point clouds as the odometry data of the lidar for subsequent external parameter calculation and optimization, where the six degrees of freedom include: three translation amounts and three rotation amounts;

[0127] Exemplarily, in this embodiment, the acquisition method of the six-degree-of-freedom transformation includes: measuring the distance d between the lidar and the center of the RTK antenna with a tape measure, then the translation T = [d, 0, 0] and the rotation degree of freedom R = [0, 0, 0] can be determined;

[0128] S3: Extract the real-time driving environment conditions of the moving platform and dynamically adjust the processing window and threshold of the lidar data and GPS-RTK data according to the real-time driving environment conditions to obtain valid data in the real-time driving environment conditions;

[0129] S3 includes the following specific steps:

[0130] S31: Extract the real-time driving environment conditions of the moving platform, where the real-time driving environment conditions include noise characteristics, laser characteristics, and GPS signal strength characteristics;

[0131] The noise characteristic is the lidar point cloud density;

[0132] The laser characteristic is the average distance between points within the characteristic area;

[0133] The GPS signal strength characteristic is the GPS status bit and the number of satellites;

[0134] S32: Preset the threshold range of the lidar point cloud density, the threshold range of the average distance between points within the characteristic area, the threshold of the GPS status bit, and the threshold of the number of satellites, synchronize the initial threshold range, and screen the real-time driving environment conditions according to the above threshold ranges, where the initial threshold range is 25 - 30;

[0135] S33: When the noise characteristic exceeds the threshold range of the lidar point cloud density, increase the time window of the lidar data; in this embodiment, increasing the time window of the lidar data can increase the cumulative data volume to smooth the noise interference and ensure that the extracted features are more stable;

[0136] When the laser characteristic exceeds the threshold range of the average distance between points within the characteristic area, determine that the data is unavailable and perform rejection processing;

[0137] When the GPS signal strength characteristic is simultaneously less than the threshold of the GPS status bit and the threshold of the number of satellites, increase the time window of the GPS data and increase the threshold value when matching the lidar and GPS-RTK data; the threshold value when matching the lidar and GPS-RTK data includes: the distance threshold and the error threshold in the ICP algorithm. In this embodiment, increasing the threshold value when matching the lidar and GPS-RTK data can tolerate a large error between the data and ensure that a limited number of feature point pairs can still be matched in a harsh environment. When the environmental conditions improve, the system will automatically reduce the threshold value to improve the matching accuracy and optimize the external parameter calculation result.

[0138] S34: According to S32 - S33, mark the real-time driving environment conditions after the screening process as valid data, and adaptively adjust the time window size for the acquisition process of the valid data.

[0139] S3 also includes the following specific steps:

[0140] S35: When the GPS signal strength feature is simultaneously less than the thresholds of the GPS status bit and the number of satellites, or when the noise feature exceeds the threshold range of the lidar point cloud density, adjust and record the weight allocation of the sensor data in real time, and provide weight constraints for subsequent optimization calculations. The weight allocation strategy specifically includes:

[0141] w = w gps × w lidar-noise × w feature ;

[0142] where w is the total weight value; w gps is the GPS-RTK weight factor;

[0143] num star is the real-time number of GPS-RTK satellites in view. In this embodiment, the more real-time GPS-RTK satellites in view, the more stable the GPS signal;

[0144] w lidar-noise is the lidar noise weight factor; w feature is the lidar feature saliency weight factor.

[0145] In this embodiment, a method for obtaining lidar data and GPS-RTK data is provided, including: using the Fast-LIO algorithm to perform feature matching on the front and rear frames of lidar point clouds, and calculating the lidar odometry data. If the time interval between the current lidar odometry and the acquisition time of GPS-RTK data is less than the threshold value of 0.1 s, then the two data are combined into a data pair for processing;

[0146] In this embodiment, S3 further includes: determining whether the GPS-RTK data in the data pair is valid; if valid, continue; otherwise, continue to loop to obtain data and perform the processing of step S4; the GPS-RTK data validity determination conditions include: 1) The GPS-RTK initialization has been completed, that is, the longitude and latitude data are greater than 0; 2) The status bit in the GPS-RTK data is 4; 3) The number of satellites in the GPS-RTK data exceeds the set threshold value, and the threshold value is set to 25 - 30;

[0147] S4: Select the lidar odometry point - northeast celestial coordinate point pair in the valid data according to the distance selection algorithm, calculate the preliminary system extrinsic parameters through the plane calibration algorithm, construct an optimization problem that minimizes the error, and perform nonlinear optimization to iteratively solve the accurate system extrinsic parameters and structural extrinsic parameters;

[0148] S4 includes the following specific steps:

[0149] S41: Through the coordinate conversion algorithm, complete the conversion of longitude and latitude data to the northeast celestial coordinate system, obtain the lidar mileage point - northeast celestial coordinate point pairs, and select the lidar mileage point - northeast celestial coordinate point pairs in the valid data according to the distance selection algorithm; wherein, the distance selection algorithm specifically screens points through the distance between point pairs, combined with the constraint condition that no two of the three points are collinear; the number of lidar mileage point - northeast celestial coordinate point pairs in the valid data is greater than or equal to 4.

[0150] S42: Calculate the preliminary system extrinsic parameters through the plane calibration algorithm.

[0151] In S41, the coordinate conversion algorithm specifically includes:

[0152] S411: Convert the longitude and latitude data to the Earth-centered Earth-fixed rectangular coordinate system, specifically as follows:

[0153]

[0154] wherein, (X, Y, Z) are the coordinates converted to the Earth-centered Earth-fixed rectangular coordinate system;

[0155] (lon, lat, alt) are the longitude and latitude coordinates in the geodetic coordinate system; e is the eccentricity of the Earth ellipsoid; N is the curvature radius of the reference ellipsoid.

[0156] S412: Extract the Earth-centered Earth-fixed rectangular coordinate system and convert the Earth-centered Earth-fixed rectangular coordinate system to the northeast celestial coordinate system, specifically including: calculating the northeast celestial coordinates (e, n, u) of (X, Y, Z) under the coordinate origin with coordinates (x 0 , y 0 , z 0 ).

[0157]

[0158]

[0159] wherein, the longitude, latitude, and altitude of the coordinate origin with coordinates (x 0 , y 0 , z 0 ) are (lon0, lat0, alt0);

[0160] (Δx, Δy, Δz) is the coordinate conversion difference matrix.

[0161] In S42, the plane calibration algorithm includes:

[0162] S421: Make assumptions before solving, and the assumptions before solving include: only considering the heading information, assuming that the origin of the translation amount coincides, and making a plane assumption for the rotation amount.

[0163] S422: Translate the mileage information and the north-east-down coordinates respectively by removing the centroid coordinates; Exemplarily, in this embodiment, the mileage information and the north-east-down coordinates are translated respectively. Since the data are all in metric system and there is no scaling involved, there is only the rotation amount left.

[0164] S423: Use two segments of trajectory points to construct an Iterative Closest Point (ICP) problem, convert the optimization problem of minimizing the error to be solved into an optimization problem of point pair fitting, solve the rotation matrix R through Singular Value Decomposition (SVD), and extract the heading angle from R.

[0165] In S423, the pre - constructed data for the optimization problem of minimizing the error to be solved includes: based on the lidar trajectory and the GPS - RTK trajectory, combined with the point weights determined in S3; In this example, the point weights represent the quality of the collected data; The optimization algorithms used in the optimization problem include: the least - squares method and the Ceres optimization library.

[0166] Iteratively solve the accurate extrinsic parameters of the system and the extrinsic parameters of the structure through the optimization algorithms in the Ceres optimization library; In this example, the use of the optimization algorithm aims to minimize the error between the lidar trajectory and the GPS - RTK trajectory, and calculate the optimal rotation matrix and translation vector.

[0167] S5: According to the accurate extrinsic parameters of the system and the extrinsic parameters of the structure, complete the joint calibration of the lidar and the GPS - RTK, obtain the extrinsic parameters of the structure from the lidar to the GPS - RTK installation position and the extrinsic parameters of the system between the north - east - down coordinate system initialized by the GPS - RTK and the Earth Centered Earth Fixed (ECEF) coordinate system, and complete the mutual conversion process between the absolute geographic coordinates and the lidar mileage.

[0168] Embodiment 2

[0169] As Figure 2 、 Figure 3 shown, in this embodiment, the depth camera 2 is installed on the fixed top surface 1 for acquiring image data. The lidar 4 is installed at the front center of the moving platform 5. Two GPS antennas 3 are symmetrically arranged on both sides behind the lidar 4 and are located on the same plane of the moving platform 5. During the installation process, try to ensure that the line connecting the centers of the two GPS antennas 3 and the lidar 4 is collinear with the central axis of the moving platform 5. During the calibration process, try to control the moving platform 5 to move in a figure - eight motion to obtain the motion trajectory 6, and the selected scene needs to satisfy both having obvious fixed feature objects and having a certain distance from the occlusion objects to avoid signal interference affecting the iteration time and data validity during the calibration solution.

[0170] Embodiment 3

[0171] This embodiment provides an electronic device, including: a processor and a memory, wherein, instructions that can be called by the processor are stored in the memory;

[0172] The processor executes the above-mentioned fast joint calibration method for lidar and GPS-RTK by calling the instructions stored in the memory.

[0173] This electronic device can have relatively large differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPU) and one or more memories. Among them, at least one computer program is stored in the memory, and this computer program is loaded and executed by the processor to implement the fast joint calibration method for lidar and GPS-RTK provided by the above method embodiment. This electronic device can also include other components for implementing device functions. For example, this electronic device can also have components such as wired or wireless network interfaces and input / output interfaces for inputting and outputting data. This embodiment will not be elaborated here.

[0174] Embodiment 4

[0175] This embodiment provides a computer-readable storage medium, on which a rewritable computer program is stored;

[0176] When the computer program runs on a computer device, it causes the computer device to execute the above-mentioned fast joint calibration method for lidar and GPS-RTK.

[0177] For example, the computer-readable storage medium can be a read-only memory (Read-Only Memory, abbreviated as: ROM), a random access memory (Random Access Memory, abbreviated as: RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, abbreviated as: CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0178] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the sequence of execution. The execution sequence of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0179] It should be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0180] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more collections of available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0181] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0182] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0183] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only one way, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in an electrical, mechanical, or other form.

[0184] The unit described as a separating component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in various embodiments of the present invention can be integrated in a processing unit, can exist separately physically for each unit, or two or more units can be integrated in one unit.

[0186] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0187] In summary of the above embodiments, the present invention Figure 2 , Figure 3 For the calibration of the lidar 4 and the GPS antenna 3 shown in, requires fewer constraints and assumptions. By using existing buildings, signs, stationary vehicles, etc. as references, the rigid constraints brought by fixed calibration objects can be avoided; and it can be carried out online, and the optimal algorithm is used to continuously iterate and solve, so as to achieve fast and high-precision joint calibration of sensors.

[0188] Compared with the prior art, the present invention according to the above solution has the beneficial effect that: a fast joint calibration method for lidar and GPS-RTK uses lidar data and the Fast-LIO algorithm to perform odometry calculation to generate continuous relative displacement information, converts the lidar odometry information into GPS-RTK odometry data through rough extrinsic parameters of the structure, and performs conditional planar calibration with the north-east-down data converted by GPS-RTK to obtain the initial value of the system extrinsic parameters. Then, the lidar odometry data is matched with the north-east-down coordinates, and an optimization problem is constructed to iteratively approximate the accurate system extrinsic parameters and structure extrinsic parameters from the initial values of the system and structure extrinsic parameters, so as to achieve high-precision joint calibration of the two. Compared with the traditional method, the present invention reduces the dependence on artificial calibration points, reduces the complexity, and significantly improves the efficiency and accuracy of calibration.

[0189] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A rapid joint calibration method for laser radar and GPS-RTK, characterized in that: The method comprises the following specific steps: S1: Automatically identify the positions of the LiDAR sensor and GPS-RTK antenna through the structure external parameter self-checking device, and perform initialization calculation of the structure external parameters; S2: Obtain information about fixed features in the operating environment through LiDAR, and record and analyze multimodal data during the motion platform's travel; S3: extract the real-time driving environment conditions of the motion platform, and dynamically adjust the processing window and threshold value of the lidar data and GPS-RTK data according to the real-time driving environment conditions to obtain valid data in the real-time driving environment conditions; S4: Select the LiDAR mileage point-northeast celestial coordinate point pair in the valid data according to the distance selection algorithm, calculate the preliminary system extrinsic parameters through the plane calibration algorithm, construct the optimization problem of minimizing the error, perform nonlinear optimization, and iteratively solve the system extrinsic parameters and structural extrinsic parameters; S5: According to the system external parameters and structural external parameters, complete the joint calibration of the laser radar and GPS-RTK, obtain the structural external parameters of the laser radar to the GPS-RTK installation position and the system external parameters of the northeast sky coordinate system and the earth-centered earth-fixed coordinate system initialized by GPS-RTK, and complete the conversion between absolute geographic coordinates and laser mileage; The process of initializing the calculation of the structural extrinsic parameters includes: S131: Identify the laser radar and GPS-RTK antennas in the position image of the laser radar sensor through a convolutional neural network, and calculate the three-dimensional coordinates of the laser radar and GPS-RTK through pixel coordinates, actual depth of pixels and camera internal parameters; S132: importing the three-dimensional coordinates of GPS-RTK into the relative posture calculation strategy, and calculating and obtaining the relative posture between the laser radar and the GPS-RTK antenna; S133: extracting the relative position and posture between the laser radar and the GPS-RTK antenna to form the initial value of the structural external parameter.

2. The rapid joint calibration method of laser radar and GPS-RTK according to claim 1 is characterized in that: S1 includes the following specific steps: S11: Install the laser radar and GPS-RTK on the motion platform, and operate the motion platform to the bottom of the structure external parameter self-checking device, wherein the structure external parameter self-checking device is composed of a depth camera suspended above the motion platform and an AI computing unit connected thereto; S12: Acquire a position image of a laser radar sensor on the motion platform through the depth camera; S13: Automatically extract the position image of the lidar sensor through the AI ​​algorithm and perform initialization calculation of the structural external parameters.

3. The rapid joint calibration method of laser radar and GPS-RTK according to claim 2 is characterized in that: S2 includes the following specific steps: S21: extracting the real-time driving environment conditions of the motion platform. When the motion platform is close to the feature object, if the GPS-RTK signal is interfered, then execute step S3, otherwise execute step S22; S22: The mobile motion platform obtains fixed feature object information in the operating environment through a laser radar, wherein the fixed feature object information includes: edge, corner and surface data of the feature object; S23: simultaneously recording multimodal data of the motion platform during its travel, wherein the multimodal data includes: laser radar point cloud, GPS-RTK latitude and longitude information, and motion trajectory data of the IMU; S24: The geometric features of the LiDAR point cloud are extracted in real time through a deep feature extraction network, and the motion trajectory data of the IMU is combined to assist in calculating the inter-frame LiDAR mileage data.

4. The rapid joint calibration method of laser radar and GPS-RTK according to claim 3 is characterized in that: S24 includes the following specific steps: S241: Introduce a feature extraction network to automatically extract local and global geometric features of the point cloud from the geometric features of the LiDAR point cloud; S242: Global feature matching of point clouds between frames using Transformer network; S243: Use multi-scale feature fusion mechanism to extract point cloud features from different scales; S244: Use GPU to accelerate the inference process of deep learning network and complete real-time matching of point clouds between frames; S245: After completing the inter-frame matching, extract the six degrees of freedom transformation between the two frames of point clouds as the mileage data of the laser radar for subsequent external parameter calculation and optimization, wherein the six degrees of freedom include: three translation quantities and three rotation quantities.

5. The rapid joint calibration method of laser radar and GPS-RTK according to claim 4 is characterized in that: S3 includes the following specific steps: S31: extracting real-time driving environment conditions of the motion platform, wherein the real-time driving environment conditions include noise characteristics, laser characteristics and GPS signal strength characteristics; The noise feature is the density of the laser radar point cloud; The laser feature is the average distance between points in the feature area; The GPS signal strength characteristics are the GPS status bit and the number of satellites; S32: Preset the threshold range of the laser radar point cloud density, the threshold range of the average distance between points in the feature area, the threshold of the GPS status bit and the threshold of the number of satellites, simultaneously preset the initial range of the threshold, and screen the real-time driving environment conditions according to the above threshold ranges; S33: when the noise feature exceeds the threshold range of the lidar point cloud density, increase the time window of the lidar data; When the laser feature exceeds the threshold range of the average distance between points in the feature area, the data is judged to be unusable and is removed; When the GPS signal strength characteristic is less than the threshold of the GPS status bit and the threshold of the number of satellites at the same time, the time window of the GPS data is increased and the threshold value when the laser radar matches the GPS-RTK data is increased; the threshold value when the laser radar matches the GPS-RTK data includes: the distance threshold and the error threshold in the ICP algorithm; S34: According to S32-S33, the real-time driving environment conditions after the screening process are marked as valid data, and the time window size is adaptively controlled for the process of obtaining the valid data.

6. The rapid joint calibration method of laser radar and GPS-RTK according to claim 5, characterized in that: S3 also includes the following specific steps: S35: When the GPS signal strength feature is less than the threshold of the GPS status bit and the threshold of the number of satellites at the same time or the noise feature exceeds the threshold range of the laser radar point cloud density, the weight distribution of the sensor data is adjusted and recorded in real time, and weight constraints are provided for subsequent optimization calculations. The weight distribution strategy specifically includes: in=in gps ×in lidar-noise ×in feature ; Among them, w is the total weight value; w gps is the GPS-RTK weight factor; num star Search satellite numbers for real-time GPS-RTK; w lidar-noise is the laser radar noise weight factor; w feature is the LiDAR feature significance weight factor.

7. The rapid joint calibration method of laser radar and GPS-RTK according to claim 6 is characterized in that: S4 includes the following specific steps: S41: completing the conversion of the longitude and latitude data and the northeast celestial coordinate system through a coordinate conversion algorithm, obtaining a pair of laser radar mileage points and northeast celestial coordinate points, and selecting a pair of laser radar mileage points and northeast celestial coordinate points in the valid data according to a distance selection algorithm; wherein the distance selection algorithm specifically selects points by the distance between the point pairs and the constraint condition that the three points are not collinear with each other; the number of the laser radar mileage points and northeast celestial coordinate points in the valid data is greater than or equal to 4; S42: Calculate preliminary system extrinsic parameters through a plane calibration algorithm.

8. The rapid joint calibration method of laser radar and GPS-RTK according to claim 7, characterized in that: In S41, the coordinate conversion algorithm specifically includes: S411: Convert the longitude and latitude data to the Earth-centered Earth-fixed rectangular coordinate system, as follows: Among them, (X, Y, Z) are the coordinates converted to the Earth-centered Earth-fixed rectangular coordinate system; (lon, lat, alt) are the longitude and latitude coordinates in the geodetic coordinate system; e is the eccentricity of the earth ellipsoid; N is the radius of curvature of the reference ellipsoid; S412: extracting the geocentric earth-fixed rectangular coordinate system and converting the geocentric earth-fixed rectangular coordinate system to the northeast celestial coordinate system, specifically including: calculating the northeast celestial coordinates (e, n, u) of (X, Y, Z) at the coordinate origin with coordinates (x0, y0, z0); Among them, the latitude and longitude of the coordinate origin with coordinates (x0, y0, z0) is (lon0, lat0, alt0); (Δx, Δy, Δz) is the coordinate transformation difference matrix.

9. The rapid joint calibration method of laser radar and GPS-RTK according to claim 8, characterized in that: In S42, the plane calibration algorithm includes: S421: making assumptions before solving, wherein the assumptions before solving include: considering only heading information, assuming that the origins of the translation amounts coincide, and making a plane assumption for the rotation amounts; S422: translating the mileage information and the northeast celestial coordinates respectively by removing the centroid coordinates; S423: Using two segments of trajectory points, construct an iterative closest point problem, convert the optimization problem of minimizing the error to be solved into an optimization problem of point pair fitting, solve the rotation matrix R through SVD decomposition, and extract the heading angle from R.

10. The rapid joint calibration method of laser radar and GPS-RTK according to claim 9, characterized in that: In S423, the construction of the pre-data of the optimization problem of minimizing the error to be solved includes: based on the laser radar trajectory and the GPS-RTK trajectory, combined with the point weights determined in S3; The optimization algorithms used in the optimization problem include: least square method and Ceres optimization library.

11. The rapid joint calibration method of laser radar and GPS-RTK according to claim 10, characterized in that: In S11, the AI ​​computing unit includes: the convolutional neural network used is a trained target detection network or a semantic segmentation network, which is used to detect or segment two types of targets: lidar and GPS-RTK antenna.

12. The rapid joint calibration method of laser radar and GPS-RTK according to claim 11, characterized in that: In S131, the calculation strategy of the three-dimensional coordinates is specifically: Where z is the converted metric depth, obtained from the depth map of the depth camera; (q, p) are the pixel coordinates of LiDAR and GPS-RTK; (x, y, z) is the physical coordinate in the depth camera coordinate system, that is, the physical coordinate of the laser radar and GPS-RTK; f x ,f y are the focal lengths of the depth camera in the horizontal and vertical directions respectively; C x ,C y The main point of the camera.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a rapid joint calibration method of a laser radar and a GPS-RTK is implemented as described in any one of claims 1 to 12.

14. An electronic device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instruction so that the electronic device performs the operation of implementing a rapid joint calibration method of a laser radar and GPS-RTK as described in any one of claims 1 to 12.

Images