A method and device for calibrating external parameters from laser radar to IMU

Abstract

The present application relates to the field of laser radar calibration technology and provides a laser radar to IMU extrinsic parameter calibration method, including: obtaining N consecutive frames of point cloud data detected by the laser radar, where N ≥ 2; performing feature extraction on the N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data; projecting the N frames of point cloud data into the first coordinate system based on a preset conversion relationship between the IMU and the first coordinate system; and determining the laser radar to IMU extrinsic parameter matrix in the first coordinate system based on feature points corresponding to the Kth frame of point cloud data in the N frames of point cloud data and feature points corresponding to the first frame of point cloud data in the N frames of point cloud data, where 1 ≤ K < N. The laser radar to IMU extrinsic parameter calibration method provided in the embodiments of the present application can be based on two sensors, the laser radar and the IMU, and can provide accurate surrounding environment data for the autonomous driving system by optimizing and fusing the data obtained by the two sensors.

Description

Technical Field

[0001] The present application belongs to the field of laser radar calibration technology, and in particular relates to a method and device for calibrating external parameters of a laser radar to an IMU. Background Art

[0002] In an autonomous driving system, the accuracy with which sensors perceive the surrounding environment determines the stability of the system's operation. To improve the accuracy of sensors' perception of the surrounding environment, multi-model complementary sensor collaboration technology is often used to obtain more information about the surrounding environment. The data obtained from multiple sensors is then fused and processed to obtain reliable data about the surrounding environment.

[0003] Among them, LiDAR can obtain 3D point cloud data containing accurate depth information and reflection intensity information, and the inertial measurement unit (IMU) can stably observe the vehicle's main body motion state and output the vehicle's driving status data at high frequency. However, LiDAR will obtain fuzzy point cloud data for scenes with relatively similar environmental factors, and it will not be able to extract the features of the point cloud data well afterwards. The data output by IMU may be affected by temperature and noise, resulting in deviations in the output driving status data. Therefore, the fusion of the two can overcome the defects of a single sensor during use, but the data obtained by heterogeneous sensors are quite different. How to further optimize the fusion process based on the two sensors of LiDAR and IMU to provide accurate surrounding environment data has become an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for calibrating the external parameters of a lidar to an IMU. Based on the lidar and IMU sensors, the method can optimize and fuse the data obtained by the two sensors to provide accurate surrounding environment data for the autonomous driving system.

[0005] In a first aspect, an embodiment of the present application provides a method for calibrating external parameters from a laser radar to an IMU, the method comprising: obtaining N consecutive frames of point cloud data detected by the laser radar, where N ≥ 2; performing feature extraction on the N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data; projecting the N frames of point cloud data to the first coordinate system based on a preset conversion relationship between the IMU and the first coordinate system; in the first coordinate system, determining the external parameter matrix from the laser radar to the IMU based on the feature points corresponding to the Kth frame of point cloud data in the N frames of point cloud data and the feature points corresponding to the 1st frame of point cloud data, where 1 ≤ K < N.

[0006] In a possible implementation of the first aspect, obtaining N consecutive frames of point cloud data detected by a lidar, where N ≥ 2, includes: obtaining point cloud data detected by the lidar; dividing the point cloud data into M sliding windows, each sliding window including N frames of point cloud data, where M ≥ 1 and N ≥ 2.

[0007] In a possible implementation of the first aspect, feature extraction is performed on N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data, including: dividing each frame of point cloud data in the N frames of point cloud data into multiple first voxel grids according to a first preset resolution; and determining the feature points corresponding to each frame of point cloud data based on a covariance matrix corresponding to each first voxel grid in the multiple first voxel grids.

[0008] In a possible implementation of the first aspect, determining feature points corresponding to each frame of point cloud data according to a covariance matrix corresponding to each first voxel grid in a plurality of first voxel grids includes: calculating eigenvalues of the covariance matrix corresponding to each first voxel grid in the plurality of first voxel grids, and determining feature points corresponding to the point cloud data in each first voxel grid;

[0009] If the feature points corresponding to the point cloud data in each first voxel grid belong to the same feature point, retaining the first voxel grid;

[0010] If the feature points corresponding to the point cloud data in each first voxel grid do not belong to the same feature points, the first voxel grid is divided into multiple second voxel grids according to the second preset resolution, and the feature points corresponding to the point cloud data in each second voxel grid are determined until the feature points of the point cloud data in each first voxel grid are determined, and the second preset resolution is smaller than the first preset resolution.

[0011] In a possible implementation of the first aspect, the method further includes: if the amount of point cloud data within the first voxel grid is greater than a preset threshold, obtaining the mean of all point cloud data within the first voxel grid; and determining the feature points of the point cloud data within each first voxel grid.

[0012] In a possible implementation of the first aspect, after performing feature extraction on N consecutive frames of point cloud data and determining feature points corresponding to the N frames of point cloud data, the method further includes:

[0013] A hash table is established based on the association relationship between N frames of point cloud data and the feature points corresponding to the N frames of point cloud data.

[0014] In a possible implementation of the first aspect, in a first coordinate system, determining an extrinsic parameter matrix from a lidar to an IMU based on feature points corresponding to a Kth frame of point cloud data and feature points corresponding to a first frame of point cloud data among N frames of point cloud data, where 1≤K<N, includes:

[0015] In the first coordinate system, minimize the sum of the distances between the feature points corresponding to the Kth frame of point cloud data and the feature points corresponding to the 1st frame of point cloud data in the N frames of point cloud data to obtain the external parameter matrix from the lidar to the IMU, 1≤K<N.

[0016] In a second aspect, an embodiment of the present application provides an external parameter calibration device for a laser radar to an IMU, comprising: an acquisition unit for acquiring N consecutive frames of point cloud data detected by the laser radar, where N ≥ 2;

[0017] A first determining unit is configured to perform feature extraction on N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data;

[0018] A conversion unit, configured to project N frames of point cloud data into the first coordinate system according to a preset conversion relationship between the IMU and the first coordinate system;

[0019] The second determination unit is used to determine the external parameter matrix from the lidar to the IMU in the first coordinate system based on the feature points corresponding to the Kth frame point cloud data in the N frames of point cloud data and the feature points corresponding to the 1st frame point cloud data, where 1≤K<N.

[0020] In a third aspect, the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any optional method of the first aspect is implemented.

[0021] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in the first aspect or any optional manner of the first aspect is implemented.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the method described in the first aspect or any optional method of the first aspect.

[0023] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0024] The embodiment of the present application provides a method for calibrating the external parameters of a laser radar to an IMU. The method extracts features from multiple frames of point cloud data detected by the laser radar to obtain feature points corresponding to the multiple frames of point cloud data. Then, based on the conversion relationship between the first frame of point cloud data and the other frames of point cloud data in the multiple frames of point cloud data, the multiple frames of point cloud data are converted to the same first coordinate system. Finally, based on the feature points of each frame of point cloud data in the point cloud data in the same first coordinate system and the feature points corresponding to the first frame of point cloud data, an external parameter conversion matrix from the laser radar to the IMU is obtained. This method does not directly fuse the data obtained by the laser radar with the IMU data, thus overcoming the problem of inaccurate calibration results due to the accumulated error of the IMU data. It realizes the provision of accurate surrounding environment data to the autonomous driving system by optimizing the processing of the data obtained by the laser radar based on the two sensors of the laser radar and the IMU. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 is a schematic diagram of the sensing ranges of multiple sensors provided in one embodiment of the present application;

[0027] Figure 2 This is a flow chart of the process of estimating external parameters from a laser radar to an IMU based on filtering provided in one embodiment of the present application;

[0028] Figure 3 This is a flow chart of a method for calibrating external parameters from a laser radar to an IMU, provided in one embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of a scenario for obtaining multi-frame point cloud data detected by a laser radar, provided in one embodiment of the present application;

[0030] Figure 5 This is a schematic diagram of the effect after feature extraction provided by an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of a first coordinate system conversion process provided by an embodiment of the present application;

[0032] Figure 7 This is an illustration of the effect of processing the acquired point cloud data using an unoptimized extrinsic parameter matrix provided in one embodiment of the present application;

[0033] Figure 81 is a schematic diagram of the effect of processing the same point cloud data obtained using the optimized extrinsic parameter matrix provided in an embodiment of the present application;

[0034] Figure 9 This is a structural block diagram of an external parameter calibration device for a laser radar to an IMU provided in an embodiment of the present application;

[0035] Figure 10 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0037] In an autonomous driving system, the accuracy with which sensors perceive the surrounding environment determines the stability of the system. To improve the accuracy with which sensors perceive the surrounding environment, multi-sensor calibration technology is often used to obtain more information about the surrounding environment. The data obtained by multiple sensors is then fused and processed to obtain reliable data about the surrounding environment.

[0038] Among them, in the multi-sensor calibration technology, the ability and perception range of multiple sensors to perceive the surrounding environment are different. The multi-sensor calibration process refers to the process of determining the relative position relationship between multiple sensors. Figure 1 The diagram shows the sensing ranges of multiple sensors provided in the embodiment of the present application, see Figure 1 , Four-Layer Laserscanner represents a four-layer laser scanning sensor, Single-Layer Laserscanner represents a single-layer laser scanning sensor, Radar represents a millimeter-wave radar sensor, UItrasonic represents an ultrasonic sensor, and Imu represents an inertial measurement sensor.

[0039] Among them, LiDAR can obtain 3D point cloud data containing accurate depth information and reflection intensity information, and IMU can stably observe the vehicle's main body motion state and output the vehicle's driving status data at high frequency. However, LiDAR will obtain fuzzy point cloud data for scenes with relatively similar environmental factors, and it will not be able to extract the features of the point cloud data well afterwards. The data output by IMU may be affected by temperature and noise, resulting in deviations in the output driving status data. Therefore, the fusion of the two can overcome the defects of a single sensor during use, but the data obtained by heterogeneous sensors are quite different. How to further optimize the fusion process based on the two sensors of LiDAR and IMU to provide accurate surrounding environment data has become an urgent problem to be solved.

[0040] In the existing technology, there are two calibration technologies for lidar and IMU sensors. One is the external parameter calibration method based on filtering. This method solves the lidar pose data and the IMU pose data separately, and then uses the Kalman filter algorithm to fuse the obtained lidar pose data and the IMU pose data.

[0041] like Figure 2 The figure shows a flow chart of the external parameter estimation process of the laser radar to IMU based on filtering provided by an embodiment of the present application. The external parameter calibration method based on filtering is to determine the relationship between the laser radar coordinate system and the IMU coordinate system at two adjacent moments. Specifically, assuming that the first moment corresponding to the two adjacent moments is marked as k, the second moment is marked as k+1, L k represents the coordinate system of the laser radar at time k, I k Indicates the coordinate system of IMU at time k, L k+1 represents the coordinate system of the laser radar at time k+1, I k+1 Represents the coordinate system of IMU at time k+1, and the conversion matrix from IMU to lidar coordinate system is The rotation matrix of the above laser radar from time k to time k+1 is If you know the transformation matrix from IMU to lidar coordinate system Then the rotation matrix of the laser radar from time k to time k+1 is It can be determined by the following formula (1):

[0042]

[0043] The above formula (1) can be solved by referring to the following quaternion related formula. The above formula (1) is expressed as the following formula (2) using quaternions:

[0044]

[0045] By multiplying (2) on the left and right respectively, we can get the following formula (3):

[0046]

[0047] If there are multiple sets of corresponding relationships between the data detected by the lidar and the data obtained by the IMU, the overdetermined equation shown in the following formula (4) can be obtained:

[0048]

[0049] In the above formula (4), ω represents the weight value corresponding to different groups of data, which is determined by the difference between the transformation vector between the lidar at two adjacent moments and the transformation vector derived from the IMU coordinate system. Q represents the weight value in the above formula (2).

[0050] Finally, the singular value decomposition method is used to calculate the above formula (4), and the eigenvector corresponding to the minimum singular value is taken to obtain the transformation matrix from the laser radar to the IMU coordinate system: You can get Figure 2 shown

[0051] The above method only uses the data detected by the lidar and the data obtained by the IMU, which cannot obtain the external parameter calibration parameters between the two in the vehicle body coordinate system. In addition, there is a cumulative error in the data obtained by the IMU. This method does not optimize the data obtained by the IMU, resulting in the inability to obtain accurate calibration parameters.

[0052] The other is an external parameter calibration method based on optimization. This method uses the assumed six-degree-of-freedom parameters to convert the data detected by the lidar into the IMU coordinate system to form point cloud data. Then, based on the point cloud data obtained by two scans, the nearest neighbor data points are determined through a kd tree (k-dimensional tree) or an octree, and the distance corresponding to each nearest neighbor data point is calculated to obtain the total distance. When the total distance is the smallest, it is considered that the point cloud data obtained by the two scans, which are composed of the lidar detection data converted to the IMU coordinate system, overlap, and the corresponding six-degree-of-freedom parameters at this time are taken as the calibration parameters of the lidar to the IMU sensor. Although this method takes into account the situation where there are cumulative errors in the data obtained by the IMU, the process requires the establishment of a corresponding model and the calculation of the total distance of the nearest neighbor data points, which prolongs the calculation time and cannot guarantee the real-time performance of data processing.

[0053] Therefore, to solve the above problems, an embodiment of the present application provides a method for calibrating the external parameters of a laser radar to an IMU. The method extracts features from the multi-frame point cloud data detected by the laser radar to obtain feature points corresponding to the multi-frame point cloud data. Then, based on the conversion relationship between the first frame point cloud data and the other frames of point cloud data in the multi-frame point cloud data, the multi-frame point cloud data is converted to the same first coordinate system. Finally, based on the feature points of each frame of point cloud data in the point cloud data in the same first coordinate system and the feature points corresponding to the first frame of point cloud data, the laser radar to IMU external parameter conversion matrix is obtained. This method does not directly fuse the data obtained by the laser radar with the IMU data, overcomes the problem of inaccurate calibration results due to the accumulated error of the IMU data, and realizes the provision of accurate surrounding environment data for the autonomous driving system by optimizing the processing of the data obtained by the laser radar based on the two sensors of the laser radar and the IMU.

[0054] In addition, based on the same first coordinate system, this method minimizes the sum of the distances between the feature points corresponding to the Kth frame point cloud data and the feature points corresponding to the 1st frame point cloud data in the multi-frame point cloud data, and obtains the extrinsic parameter matrix from the lidar to the IMU, so that the obtained extrinsic parameter conversion matrix is the extrinsic parameter conversion matrix after optimization processing, further ensuring the accuracy of the extrinsic parameter conversion matrix.

[0055] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0056] like Figure 3 FIG. 1 is a flow chart of a method for calibrating an external parameter from a lidar to an IMU provided in an embodiment of the present application, the method comprising:

[0057] S101, obtain N consecutive frames of point cloud data detected by the laser radar, where N ≥ 2.

[0058] It should be understood that point cloud data refers to a set of vectors in a three-dimensional coordinate system. This step essentially obtains the three-dimensional coordinates corresponding to each point in the point cloud data.

[0059] In order to accelerate the processing of point cloud data, in one possible implementation, N consecutive frames of point cloud data detected by the lidar are obtained, where N ≥ 2, including: obtaining point cloud data detected by the lidar; dividing the point cloud data into M sliding windows, each sliding window including N frames of point cloud data, where M ≥ 1 and N ≥ 2.

[0060] For example, assuming that the point cloud data detected by the lidar includes 10 frames of point cloud data, the 10 frames of point cloud data can be divided into two sliding windows, each sliding window includes 5 frames of point cloud data, and the point cloud data in the sliding windows including 5 frames of point cloud data are processed in turn.

[0061] In the embodiment of the present application, the acquired point cloud data is stored as a file in *.pcd format to facilitate subsequent data calls.

[0062] In order to fully obtain point cloud data at different rotation angles, in the embodiment of the present application, the vehicle equipped with IMU and laser radar rotates around Figure 4 Driving along the figure-eight trajectory shown is used to obtain point cloud data with a rotation angle of 360°.

[0063] In one possible implementation, in order to avoid the influence of noise on the external parameter results, the point cloud data detected by the lidar can be filtered to remove the noise from the detected point cloud data. Exemplarily, the filtering method may include Kalman filtering, smoothing filtering and / or Laplace operator, etc.

[0064] S102 , performing feature extraction on N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data.

[0065] The feature extraction process is also the process of extracting the feature points corresponding to each frame of point cloud data in N frames of point cloud data, that is, judging whether the point cloud points in each frame of point cloud data belong to points on the edge line (hereinafter referred to as line points) or points on the plane (hereinafter referred to as surface points).

[0066] In one possible implementation, feature extraction is performed on N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data, including: dividing each frame of point cloud data in the N frames of point cloud data into multiple first voxel grids according to a first preset resolution; and determining the feature points corresponding to each frame of point cloud data based on a covariance matrix corresponding to each first voxel grid in the multiple first voxel grids.

[0067] Among them, according to the covariance matrix corresponding to each first voxel grid in multiple first voxel grids, the feature points corresponding to each frame of point cloud data are determined, including: calculating the eigenvalues of the covariance matrix corresponding to each first voxel grid in multiple first voxel grids, and determining the feature points corresponding to the point cloud data in each first voxel grid; if the feature points corresponding to the point cloud data in each first voxel grid belong to the same feature point, then retaining the first voxel grid; if the feature points corresponding to the point cloud data in each first voxel grid do not belong to the same feature point, then dividing the first voxel grid into multiple second voxel grids according to a second preset resolution, and determining the feature points corresponding to the point cloud data in each second voxel grid, until the feature points of the point cloud data in each first voxel grid are determined, and the second preset resolution is less than the first preset resolution.

[0068] In the embodiment of the present application, an adaptive voxel grid method is used to extract the feature points corresponding to each frame of point cloud data in N frames of point cloud data, such as Figure 5 The figure shows a schematic diagram of the effect after feature extraction provided by an embodiment of the present application.

[0069] The feature extraction process includes: for each frame of point cloud data in N frames of point cloud data, dividing it into multiple first voxel grids according to a first preset resolution, calculating the eigenvalues of the covariance matrix in each first voxel grid in the multiple first voxel grids, and judging whether the point cloud points in each first voxel grid fall on the same edge line or plane. If the points in the first voxel grid all fall on the same edge line or plane, retaining the current first voxel grid; otherwise, dividing the first voxel grid into multiple second voxel grids according to a second preset resolution, and the second preset resolution is smaller than the first preset resolution; repeating the step of determining whether the points in each second voxel grid fall on the same edge line or plane until the point cloud data in each first voxel grid all fall on the same edge line or plane.

[0070] For example, assuming that the three-dimensional coordinate values of the point cloud data collected by the lidar are measured in meters, the first preset resolution can be set to 1m, and a voxel grid with a resolution of 1m is selected to divide each frame of N frames of point cloud data.

[0071] In the actual design process, if the number of point cloud data in the first voxel grid is greater than the first preset threshold, the mean of the point cloud data in the first voxel grid is obtained to determine whether the point cloud points in the first voxel grid fall on the same edge line or plane.

[0072] Of course, if the number of point cloud data in the second voxel grid is greater than the second preset threshold, the mean of the point cloud data in the second voxel grid can also be obtained to determine whether the point cloud points in the second voxel grid fall on the same edge line or plane.

[0073] The first preset threshold and the second preset threshold can be designed according to actual needs, and this application does not limit this.

[0074] In another possible implementation, the acquired point cloud data is divided into M sliding windows, each of the M sliding windows includes N frames of point cloud data, where M≥1 and N≥2; feature extraction is performed on the N frames of point cloud data in each sliding window to determine the feature points corresponding to each frame of point cloud data in the N frames of point cloud data.

[0075] S103: Project N frames of point cloud data into a preset first coordinate system according to a conversion relationship between the preset IMU and the first coordinate system.

[0076] Among them, the first coordinate system refers to a fixed coordinate system. For example, in an embodiment of the present application, the first coordinate system can be a world coordinate system, which is the absolute coordinate system of the autonomous driving system where the lidar and IMU are located.

[0077] like Figure 6 The diagram shows a first coordinate system conversion process provided by an embodiment of the present application, see Figure 6 , the solid line coordinate system L represents the lidar coordinate system, the dotted line coordinate system I represents the IMU coordinate system, and the first coordinate system is the world coordinate system W.

[0078] Assume N = 5, Figure 6 The P moment in the middle represents the time when the first frame of point cloud data among the five frames of point cloud data is obtained. Figure 6 Where time O represents the time of acquiring the third frame of point cloud data among the five frames of point cloud data. The process of projecting the third frame of point cloud data to the world coordinate system W can be referred to the following formula (5):

[0079]

[0080] In the above formula (5), Represents the transformation matrix between the lidar coordinate system at time O and the lidar coordinate system at time P; Represents the conversion matrix between the lidar coordinate system and the IMU coordinate system, that is, the preset external parameter matrix, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time P, that is Represents the inverse matrix of the transformation matrix between the IMU coordinate system and the world coordinate system at time P, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time 0. It is not difficult to understand that the process of projecting the third frame of point cloud data to the world coordinate system W is essentially the process of projecting the second frame of point cloud data to the Nth frame of point cloud data into the first frame of point cloud data, and then converting them to the world coordinate system W.

[0081] In actual application, That is, the conversion relationship between the IMU and the world coordinate system W. Its specific value can be determined based on the data collected by the IMU. The specific value of can be determined based on the point cloud data detected by the lidar, and the preset external parameter matrix can be set according to the actual application, which is not limited in this application.

[0082] After N consecutive frames of point cloud data are converted to the first coordinate system (i.e., the world coordinate system) using the above formula (5), in order to facilitate the query and retrieval of point cloud data and its corresponding feature points, optionally, a hash table is established based on the association relationship between the N frames of point cloud data and the feature points corresponding to the N frames of point cloud data.

[0083] S104, in the first coordinate system, based on the feature points corresponding to the K-th frame of point cloud data and the feature points corresponding to the 1st frame of point cloud data in the N frames of point cloud data, determine the external parameter matrix from the lidar to the IMU, 1≤K<N.

[0084] In an embodiment of the present application, the following formula (6) is used to minimize the sum of the distances between the feature points corresponding to the Kth frame of point cloud data and the feature points corresponding to the 1st frame of point cloud data in the N frames of point cloud data in the first coordinate system to obtain the external parameter matrix from the lidar to the IMU, 1≤K<N.

[0085]

[0086] In the above formula (6), i represents the i-th feature point, q represents a feature point on an edge line or plane of the first frame of the point cloud data in the M-frame sliding window, and p i Represents the feature point p of the K-th frame point cloud data in the M-frame sliding window i , n T Represents the direction vector of the edge line or the normal vector of the plane.

[0087] Assume that the i-th feature point is from the S-th i Frame extraction, where i∈{1,…,N}, Si∈{1,…,M}, and the three-dimensional coordinates corresponding to the point cloud data in the M-frame sliding window are marked as T=(T1,…,T M ); then the feature point p of the Kth frame point cloud data in the M-frame sliding window i It can be expressed by the following formula (7):

[0088] p i =R si p fi +t si i=1,…,N (7)

[0089] In the above formula (7), p fi (i∈{1,…,N}) represents the feature points extracted from the M-frame sliding window, and the feature points are associated with the same plane or the same edge line; R si represents the rotation matrix, t si Represents the translation matrix.

[0090] According to the above formula (6) and formula (7), the external parameter matrix from the lidar to the IMU can be determined.

[0091] When dividing the point cloud data detected by the LiDAR into sliding windows, the sum of the distances between the Kth frame point cloud data and the corresponding updated 1st frame point cloud data in the M-frame sliding window is minimized to obtain the LiDAR-to-IMU extrinsic parameter matrix for each sliding window, where 1 ≤ K < N. As is easy to understand, given the unchanging deployment positions of the LiDAR and IMU on the vehicle, the LiDAR-to-IMU extrinsic parameter matrices obtained for each sliding window are equal.

[0092] like Figure 7 The figure shows the effect of processing the acquired point cloud data using the non-optimized extrinsic parameter matrix provided in the embodiment of the present application. Figure 8 The figure shows the effect of processing the same point cloud data obtained by using the optimized external parameter matrix provided in the embodiment of the present application. Figure 8 This not only verifies the feasibility of the external parameter calibration method from the laser radar to the IMU provided in the embodiment of the application, but also Figure 8 It can be seen that the results obtained by the external parameter calibration method for the lidar to IMU provided in the embodiment of the present application are clearer and more accurate. Compared with the existing technology, the accuracy of the external parameter calibration for the lidar to IMU can be significantly improved.

[0093] The embodiment of the present application provides a method for calibrating the external parameters of a laser radar to an IMU. The method extracts features from multiple frames of point cloud data detected by the laser radar to obtain feature points corresponding to the multiple frames of point cloud data. Then, based on the conversion relationship between the first frame of point cloud data and the other frames of point cloud data in the multiple frames of point cloud data, the multiple frames of point cloud data are converted to the same first coordinate system. Finally, based on the feature points of each frame of point cloud data in the point cloud data in the same first coordinate system and the feature points corresponding to the first frame of point cloud data, an external parameter conversion matrix from the laser radar to the IMU is obtained. This method does not directly fuse the data obtained by the laser radar with the IMU data, thus overcoming the problem of inaccurate calibration results due to the accumulated error of the IMU data. It realizes the provision of accurate surrounding environment data to the autonomous driving system by optimizing the processing of the data obtained by the laser radar based on the two sensors of the laser radar and the IMU.

[0094] In addition, based on the same first coordinate system, this method minimizes the sum of the distances between the feature points corresponding to the Kth frame point cloud data and the feature points corresponding to the 1st frame point cloud data in the multi-frame point cloud data, and obtains the extrinsic parameter matrix from the lidar to the IMU, so that the obtained extrinsic parameter conversion matrix is the extrinsic parameter conversion matrix after optimization processing, further ensuring the accuracy of the extrinsic parameter conversion matrix.

[0095] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] Corresponding to the external parameter calibration method from the lidar to the IMU described in the above embodiment, Figure 9 A structural block diagram of an external parameter calibration device for a laser radar to an IMU provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0097] Reference Figure 5 , the device 200 for calibrating external parameters of the laser radar to the IMU includes: an acquisition unit 201 for acquiring N consecutive frames of point cloud data detected by the laser radar, where N≥2;

[0098] A first determining unit 202 is configured to perform feature extraction on N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data;

[0099] The conversion unit 203 is used to project the N frames of point cloud data into the first coordinate system according to a preset conversion relationship between the IMU and the first coordinate system;

[0100] The second determination unit 204 is used to determine the external parameter matrix from the lidar to the IMU in the first coordinate system based on the feature points corresponding to the Kth frame of point cloud data and the feature points corresponding to the 1st frame of point cloud data in the N frames of point cloud data, where 1≤K<N.

[0101] Optionally, the acquisition unit 201 is further used to: acquire point cloud data detected by the laser radar; divide the point cloud data into M sliding windows, each sliding window includes N frames of point cloud data, M≥1, N≥2.

[0102] Optionally, the first determination unit 202 is further used to: divide each frame of point cloud data in N frames of point cloud data into multiple first voxel grids according to a first preset resolution; and determine the feature points corresponding to each frame of point cloud data based on the covariance matrix corresponding to each first voxel grid in the multiple first voxel grids.

[0103] Optionally, determining the feature points corresponding to each frame of point cloud data according to a covariance matrix corresponding to each first voxel grid in the plurality of first voxel grids includes: calculating eigenvalues of the covariance matrix corresponding to each first voxel grid in the plurality of first voxel grids, and determining the feature points corresponding to the point cloud data in each first voxel grid;

[0104] If the feature points corresponding to the point cloud data in each first voxel grid belong to the same feature point, retaining the first voxel grid;

[0105] If the feature points corresponding to the point cloud data in each first voxel grid do not belong to the same feature points, the first voxel grid is divided into multiple second voxel grids according to the second preset resolution, and the feature points corresponding to the point cloud data in each second voxel grid are determined until the feature points of the point cloud data in each first voxel grid are determined, and the second preset resolution is smaller than the first preset resolution.

[0106] Optionally, the method further includes: if the amount of point cloud data within the first voxel grid is greater than a preset threshold, obtaining the mean of all point cloud data within the first voxel grid; and determining feature points of the point cloud data within each first voxel grid.

[0107] Optionally, after performing feature extraction on N consecutive frames of point cloud data and determining feature points corresponding to the N frames of point cloud data, the method further includes:

[0108] A hash table is established based on the association relationship between N frames of point cloud data and the feature points corresponding to the N frames of point cloud data.

[0109] Optionally, the second determination unit 204 is also used to: minimize the sum of the distances between the feature points corresponding to the Kth frame point cloud data and the feature points corresponding to the 1st frame point cloud data in the N frames of point cloud data in the first coordinate system, and obtain the external parameter matrix from the lidar to the IMU, 1≤K<N.

[0110] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0111] Based on the same inventive concept, the embodiment of the present application further provides a terminal device, the terminal device 300 is as follows Figure 10 shown.

[0112] like Figure 10 As shown, the terminal device 300 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The computer program 303 can be executed by the processor 301 to generate instructions, and the processor 301 can implement the steps of each of the aforementioned rights authentication method embodiments according to the instructions. Alternatively, when the processor 301 executes the computer program 303, the functions of each module / unit in the aforementioned device embodiments are implemented.

[0113] Exemplarily, computer program 303 may be divided into one or more modules / units, one or more of which are stored in memory 302 and executed by processor 301 to implement the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of computer program 303 in terminal device 300.

[0114] Those skilled in the art will understand that Figure 10 It is merely an example of the terminal device 300 and does not constitute a limitation of the terminal device 300. The terminal device 300 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 300 may also include input and output devices, network access devices, buses, etc.

[0115] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0116] The memory 302 can be an internal storage unit of the terminal device 300, such as a hard disk or memory of the terminal device 300. The memory 302 can also be an external storage device of the terminal device 300, such as a plug-in hard disk equipped on the terminal device 300, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 302 can also include both the internal storage unit of the terminal device 300 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the terminal device 300. The memory 302 can also be used to temporarily store data that has been output or is about to be output.

[0117] The terminal device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be repeated here.

[0118] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the method of the above method embodiment when the computer program is executed by a processor.

[0119] An embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to implement the method of the above method embodiment.

[0120] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include at least: any entity or device capable of carrying computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0121] References to "one embodiment" or "some embodiments" in this application mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0122] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include at least one of such features.

[0123] In addition, in this application, unless otherwise clearly specified and limited, the terms "connection", "connected", etc. should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection between two elements or the interaction relationship between two elements. Unless otherwise clearly specified, ordinary technicians in this field can understand the specific meanings of the above terms in this application according to specific circumstances.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for calibrating external parameters from a laser radar to an IMU, characterized in that: include: Obtain N consecutive frames of point cloud data detected by the lidar, where N ≥ 2; Performing feature extraction on the N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data; Coordinate transformation is performed according to the preset transformation relationship between the IMU and the world coordinate system and the transformation matrix between the coordinate system of the laser radar and the coordinate system of the IMU, and the N frames of point cloud data are projected into the world coordinate system; wherein the process of projecting the N frames of point cloud data into the world coordinate system is implemented by the following formula: In the formula, Represents the transformation matrix between the laser radar coordinate system at time O and the laser radar coordinate system at time P; represents the transformation matrix between the laser radar coordinate system and the IMU coordinate system, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time P, that is, Represents the inverse matrix of the transformation matrix between the IMU coordinate system and the world coordinate system at time P, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time 0; The process of projecting the N frames of point cloud data into the world coordinate system is a process of projecting the second frame of point cloud data to the Nth frame of point cloud data into the first frame of point cloud data respectively, and then converting them into the world coordinate system W; In the world coordinate system, the external parameter matrix from the laser radar to the IMU is determined based on the feature points corresponding to the Kth frame of point cloud data in the N frames of point cloud data and the feature points corresponding to the 1st frame of point cloud data, where 1≤K<N.

2. The method according to claim 1, characterized in that The step of obtaining N consecutive frames of point cloud data detected by the laser radar, where N is greater than or equal to 2, includes: Obtain point cloud data detected by lidar; The point cloud data is divided into M sliding windows, each sliding window includes N frames of point cloud data, M≥1, N≥2.

3. The method according to claim 1, characterized in that The performing feature extraction on the N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data includes: Dividing each frame of the N frames of point cloud data into a plurality of first voxel grids according to a first preset resolution; Determine the feature points corresponding to each frame of point cloud data according to the covariance matrix corresponding to each of the first voxel grids in the plurality of first voxel grids.

4. The method according to claim 3, characterized in that The determining, according to the covariance matrix corresponding to each of the first voxel grids in the plurality of first voxel grids, the feature points corresponding to each frame of point cloud data includes: Calculating the eigenvalues of the covariance matrix corresponding to each of the first voxel grids in the plurality of first voxel grids, and determining the feature points corresponding to the point cloud data in each of the first voxel grids; If the feature points corresponding to the point cloud data in each of the first voxel grids belong to the same feature point, retaining the first voxel grid; If the feature points corresponding to the point cloud data in each of the first voxel grids do not belong to the same feature points, the first voxel grid is divided into multiple second voxel grids according to a second preset resolution, and the feature points corresponding to the point cloud data in each of the second voxel grids are determined until the feature points of the point cloud data in each of the first voxel grids are determined, and the second preset resolution is smaller than the first preset resolution.

5. The method according to claim 3 or 4, characterized in that The method further includes: if the amount of point cloud data in the first voxel grid is greater than a preset threshold, obtaining a mean value of all point cloud data in the first voxel grid; and determining feature points of each point cloud data in the first voxel grid.

6. The method according to claim 1, characterized in that After extracting features from the N consecutive frames of point cloud data and determining feature points corresponding to the N frames of point cloud data, the method further includes: A hash table is established according to the association relationship between the N frames of point cloud data and the feature points corresponding to the N frames of point cloud data.

7. The method according to claim 1, characterized in that In the world coordinate system, the extrinsic parameter matrix from the laser radar to the IMU is determined based on the feature points corresponding to the Kth frame of point cloud data in the N frames of point cloud data and the feature points corresponding to the first frame of point cloud data, where 1≤K<N, including: In the world coordinate system, the sum of the distances between the feature points corresponding to the K-th frame of point cloud data in the N frames of point cloud data and the feature points corresponding to the 1st frame of point cloud data is minimized to obtain an extrinsic parameter matrix from the lidar to the IMU, 1≤K<N.

8. A laser radar to IMU external parameter calibration device, characterized in that: include: The acquisition unit is used to obtain N consecutive frames of point cloud data detected by the laser radar, where N is greater than or equal to 2; A first determining unit is configured to perform feature extraction on the N consecutive frames of point cloud data to determine feature points corresponding to the N frames of point cloud data; A conversion unit is configured to perform coordinate conversion based on a preset conversion relationship between the IMU and the world coordinate system and a conversion matrix between the coordinate system of the laser radar and the coordinate system of the IMU, and project the N frames of point cloud data into the world coordinate system; wherein the process of projecting the N frames of point cloud data into the world coordinate system is implemented by the following formula: In the formula, Represents the transformation matrix between the laser radar coordinate system at time O and the laser radar coordinate system at time P; represents the transformation matrix between the laser radar coordinate system and the IMU coordinate system, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time P, that is, Represents the inverse matrix of the transformation matrix between the IMU coordinate system and the world coordinate system at time P, Represents the transformation matrix between the IMU coordinate system and the world coordinate system at time 0; The process of projecting the N frames of point cloud data into the world coordinate system is a process of projecting the second frame of point cloud data to the Nth frame of point cloud data into the first frame of point cloud data respectively, and then converting them into the world coordinate system W; The second determination unit is used to determine the external parameter matrix from the laser radar to the IMU in the world coordinate system based on the feature points corresponding to the Kth frame point cloud data in the N frames of point cloud data and the feature points corresponding to the 1st frame point cloud data, where 1≤K<N.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images