Laser radar-camera-inertial navigation combined calibration method and device
By acquiring the point cloud data and rotation translation information of the sensor, and using the hand-eye calibration method to coordinate calibration of lidar, camera and inertial navigation sensors, the problem of large errors in manual point selection in the existing technology is solved, and the rapid and accurate multi-dimensional Sensor calibration.
Patent Information
- Application Number
- CN202411231053.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-10
AI Technical Summary
The prior art requires manual point selection in multi-sensor calibration, resulting in large errors and the inability to quickly calibrate without a common field of view.
By obtaining point cloud data of lidar, camera and inertial navigation sensors, establishing local maps and obtaining rotation translation information, the hand-eye calibration method is used to perform joint calibration of multiple sensors to avoid external intervention.
Fast and accurate multi-sensor calibration without common field of view is achieved, which reduces errors caused by manual intervention and improves calibration efficiency.
Smart Images
Figure CN119104088A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for joint calibration of a laser radar, a camera, and an inertial navigation system. Background Art
[0002] In the field of autonomous driving, multiple sensors are generally used for data fusion to facilitate the application of subsequent perception algorithms. Each sensor is in its own coordinate system by default, and multiple sensors do not have a common field of view. When calibrating multiple sensors, the existing technology only calibrates them separately, such as calibrating the camera and lidar by selecting points first, or using inertial navigation and lidar for calibration. Each group of sensors needs to be calibrated separately and finally merged into one coordinate system. However, this method requires manual point selection, which may cause large errors due to human intervention. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a method and device for laser radar-camera-inertial navigation joint calibration, which can avoid external interference and thus quickly calibrate multiple sensors without a common field of view.
[0004] In a first aspect, an embodiment of the present invention provides a method for joint calibration of a lidar-camera-inertial navigation, the method comprising: acquiring point cloud data of multiple sensors; the sensors comprising a lidar, an image sensor and an inertial sensor; matching the lidar point cloud data with a first local map to generate a second local map; acquiring calibration data in the second local map; the calibration data being rotation and translation information; and jointly calibrating the multiple sensors using hand-eye calibration according to the calibration data.
[0005] Optionally, before acquiring the point cloud data of multiple sensors, the method further includes: synchronizing the time of the multiple sensors.
[0006] Optionally, the laser radar point cloud data format is a data frame format; matching the laser radar point cloud data with the first local map to generate a second local map includes: determining whether a first local map exists; the first local map is a pre-established local map; if so, fusing the laser radar point cloud data with the data of the first local map to generate a second local map; if not, using the first frame of laser radar point cloud data as the first local map.
[0007] Optionally, calibration data is obtained in the second local map; the calibration data is rotation and translation information, including: obtaining the rotation and translation matrix of each sensor in the second local map; obtaining rotation and translation information according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
[0008] Optionally, the joint calibration of the multiple sensors using hand-eye calibration according to the calibration data includes: using a calibration model to perform hand-eye calibration on the multiple sensors according to the calibration data, and saving calibration parameters.
[0009] In a second aspect, the present invention also provides an embodiment of a laser radar-camera-inertial navigation joint calibration device, the device comprising: a first acquisition module, used to acquire point cloud data of multiple sensors; the sensors include laser radars, image sensors and inertial sensors; a first generation module, used to match the laser radar point cloud data with a first local map to generate a second local map; a second acquisition module, used to acquire calibration data in the second local map; the calibration data is rotation and translation information; a joint calibration module, used to jointly calibrate multiple sensors using hand-eye calibration according to the calibration data.
[0010] Optionally, the device further includes: a time synchronization submodule, used to synchronize the time of multiple sensors before acquiring the point cloud data of the multiple sensors.
[0011] Optionally, the laser radar point cloud data format is a data frame format; the first generation module includes: a first judgment submodule, used to determine whether a first local map exists; the first local map is a pre-established local map; a first generation submodule, used to, if yes, fuse the laser radar point cloud data with the data of the first local map to generate a second local map; and a second generation submodule, used to, if no, use the first frame of laser radar point cloud data as the first local map.
[0012] Optionally, the second acquisition module is specifically used to acquire the rotation and translation matrix of each sensor in the second local map; acquire rotation and translation information according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
[0013] Optionally, the joint calibration module is specifically used to perform hand-eye calibration on multiple sensors according to calibration data using a calibration model, and save calibration parameters.
[0014] The embodiment of the present invention provides a method and device for joint calibration of a lidar, a camera and an inertial navigation system. The method acquires point cloud data of a plurality of sensors, establishes a local radar map, obtains rotation and translation information of each sensor in the local map, and further performs joint calibration of the plurality of sensors by adopting a hand-eye calibration method according to the rotation and translation information. This method can avoid external interference, thereby quickly performing calibration of multiple sensors without a common field of view. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 The figure is a flowchart of a method for laser radar-camera-inertial navigation joint calibration according to an embodiment of the present invention.
[0017] Figure 2 It is a schematic diagram of the interpolation method part of an embodiment of the present invention.
[0018] Figure 3 This is a schematic diagram of the architecture of a laser radar-camera-inertial navigation joint calibration device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0020] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example
[0021] Figure 1 FIG. 1 is a flow chart of a method for laser radar-camera-inertial navigation joint calibration according to an embodiment of the present invention; see FIG. Figure 1 As shown, the method includes: S110, acquiring point cloud data of multiple sensors; the sensors include laser radar, image sensor and inertial sensor; In this step, a suitable site is selected for the vehicle to drive automatically, so that the vehicle can drive automatically. At the same time, data from the lidar point cloud, image sensor and inertial sensor are collected.
[0022] S120, matching the laser radar point cloud data with the first local map to generate a second local map; In this step, the point cloud data obtained by the lidar is matched with the first local map through the matching method of NDT (Normal Distribution Transform) + ICP (Iterative Closest Point) to generate a second local map.
[0023] S130, acquiring calibration data in the second local map; the calibration data is rotation and translation information; In this step, the rotation and translation information of each sensor in the second local map is obtained. In view of the different sampling frame rates of each sensor or the different start times of the same sensor, in order to obtain the mean of the sensor rotation and translation matrix at the current moment, the rotation and translation information of a frame of point cloud in each time period can be obtained by interpolation.
[0024] like Figure 2 As shown in the figure, the specific interpolation method is, for example, a point Pi falling between the times T1 and T2 in the combined inertial navigation time series P1 and the time series P2 of a frame of point cloud, whose timestamp is Ti, then the velocity information at the time Ti is front_scale = (Ti - T1) / (T2 - T1); back_scale = (T2 - Ti) / (T2 - T1).
[0025] It should be noted that due to the motion distortion of the laser odometer, there are errors in the interpolation of the rotation and translation moments. Therefore, for the laser odometer, it is necessary to first convert the rotation and translation matrix of the laser odometer into the form of quaternion, and then perform linear interpolation according to the above method. This ensures that the interpolated quaternion is still a unit quaternion, thereby representing a valid rotation.
[0026] Furthermore, after calculating the rotation and translation information of each time period of a frame of point cloud using the interpolation method, the rotation and translation information of each time period is averaged to obtain the rotation and translation information of the sensor at the current moment.
[0027] S140. Jointly calibrate multiple sensors using hand-eye calibration according to the calibration data.
[0028] The method for joint calibration of lidar, camera and inertial navigation provided in an embodiment of the present invention acquires point cloud data of multiple sensors, establishes a local radar map, obtains rotation and translation information of each sensor in the local map, and further performs joint calibration of the multiple sensors by hand-eye calibration based on the rotation and translation information, thereby avoiding external interference and quickly calibrating multiple sensors without a common field of view.
[0029] Optionally, in some embodiments, before acquiring the point cloud data of multiple sensors, the method further includes: synchronizing the time of the multiple sensors.
[0030] Specifically, before performing multi-sensor data fusion, the time of multiple sensors must be synchronized first. GPS clock synchronization or PTP / gPTP (Precise Time protocol) clock synchronization protocol is used to synchronize the time between multiple sensors. The sensors include lidar, image sensors, inertial sensors, millimeter wave sensors, etc.
[0031] Optionally, the laser radar point cloud data format is a data frame format; the matching of the laser radar point cloud data with the first local map to generate a second local map includes: determining whether a first local map exists; the first local map is a pre-established local map; if so, fusing the laser radar point cloud data with the data of the first local map to generate a second local map; if not, using the first frame of laser radar point cloud data as the first local map.
[0032] Specifically, the point cloud data of the laser radar is continuously acquired, and the newly received point cloud data is matched with the first local map to generate a second local map; wherein the first local map is a local map of the area that has been pre-established. Before matching with the first local map, it is first determined whether the first local map exists. If so, the acquired laser radar point cloud data is fused with the data of the first local map to form a new second local map. It should be noted that since the laser radar's continuous point cloud is fused into the module of the local map, the amount of data in the local map can be infinite. However, in actual use, the local map cannot be infinite. Therefore, there are generally two cropping methods for cropping and fusion of local maps. One is based on the relationship of time, for example, sorting by time and deleting point clouds within a certain time interval. The other is based on the relationship of position, for example, the current absolute position and a certain radius value, deleting the point cloud data before the above-mentioned area.
[0033] Optionally, calibration data is obtained in the second local map; the calibration data is rotation and translation information, including: obtaining the rotation and translation matrix of each sensor in the second local map; obtaining rotation and translation information according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
[0034] Specifically, when obtaining calibration data in the second local map, taking the lidar data as an example, a certain radar is used as the main radar, and the displacement information of the main radar odometer is calculated by the translation of the rotation and translation matrix (step_master) of each sensor obtained in the second local map. When the displacement of the odometer information of the main radar reaches a certain distance, the hand-eye calibration parameters are calculated; wherein step_master is equal to the current odometer information of the main radar multiplied by the inverse matrix of the odometer information of the main radar at the previous moment; a certain distance is related to the sampling rate of the radar and the speed of the vehicle.
[0035] Furthermore, after the displacement of the odometer information of the master radar reaches a certain distance, the displacement information of the odometers of all slave radars is calculated and the preset operation is performed with the step of the master radar. The operation method is the same as (step_master).
[0036] Optionally, the preset operation includes: calculating the ratio of the displacement information of the master radar to the displacement information of each slave radar, the error of the ratio is less than 0.005, and calculating the ratio of the angle information of the master radar to the angle information of each slave radar, the error of the ratio is less than 0.1.
[0037] Optionally, the joint calibration of the multiple sensors using hand-eye calibration according to the calibration data includes: using a calibration model to perform hand-eye calibration on the multiple sensors according to the calibration data, and saving calibration parameters.
[0038] Specifically, after the calibration data is obtained through the preset operation and more than a certain amount of data is accumulated, hand-eye calibration calculations are performed on all the data, and the calibration parameters are saved, wherein the certain amount of data needs to be greater than 6 frames of data, preferably greater than 30 frames of data, and the above-mentioned displacement information ratio or angle information ratio can be used as the weight of the matrix in the hand-eye calibration, so as to more effectively screen out the matrix eigenvalues. Example
[0039] Figure 3 FIG. 1 is a schematic diagram of the architecture of a laser radar-camera-inertial navigation joint calibration device according to an embodiment of the present invention; Figure 3 As shown, the device comprises: A first acquisition module is used to acquire point cloud data from multiple sensors; the sensors include laser radars, image sensors, and inertial sensors; A first generating module, used for matching the laser radar point cloud data with the first local map to generate a second local map; A second acquisition module, used to acquire calibration data in the second local map; the calibration data is rotation and translation information; The joint calibration module is used to perform joint calibration on multiple sensors using hand-eye calibration according to the calibration data.
[0040] Optionally, the device further includes: a time synchronization submodule, used to synchronize the time of multiple sensors before acquiring the point cloud data of the multiple sensors.
[0041] Optionally, the laser radar point cloud data format is a data frame format; the first generation module includes: a first judgment submodule, used to determine whether a first local map exists; the first local map is a pre-established local map; a first generation submodule, used to, if yes, fuse the laser radar point cloud data with the data of the first local map to generate a second local map; and a second generation submodule, used to, if no, use the first frame of laser radar point cloud data as the first local map.
[0042] Optionally, the second acquisition module is specifically used to acquire the rotation and translation matrix of each sensor in the second local map; acquire rotation and translation information according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
[0043] Optionally, the joint calibration module is specifically used to perform hand-eye calibration on multiple sensors according to calibration data using a calibration model, and save calibration parameters.
[0044] In summary, the embodiments of the present invention provide a method and device for joint calibration of lidar, camera and inertial navigation. By acquiring point cloud data of multiple sensors, a radar local map is established to obtain rotation and translation information of each sensor in the local map, and further hand-eye calibration is adopted according to the rotation and translation information to jointly calibrate the multiple sensors, which can avoid external interference, thereby quickly calibrating multiple sensors without a common field of view.
[0045] It should be noted that in this article, relational terms such as first and second are only used to refer to One entity or operation is distinguished from another entity or operation without necessarily requiring or implying that these Any such actual relationship or order exists between entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.
[0046] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0047] For the convenience of description, if it involves a system, server, etc., it may be described separately by dividing the functions into various units / modules. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.
[0048] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A laser radar-camera-inertial navigation joint calibration method, characterized in that: Acquire point cloud data from multiple sensors; the sensors include laser radar, image sensor and inertial sensor; Matching the laser radar point cloud data with the first local map to generate a second local map; Acquire calibration data in the second local map; the calibration data is rotation and translation information; The multiple sensors are jointly calibrated using hand-eye calibration according to the calibration data.
2. The laser radar-camera-inertial navigation joint calibration method according to claim 1, characterized in that: Before acquiring the point cloud data of the multiple sensors, the method further includes: Synchronize multiple sensor times.
3. The laser radar-camera-inertial navigation joint calibration method according to claim 1, characterized in that: The laser radar point cloud data format is a data frame format; The step of matching the laser radar point cloud data with the first local map to generate a second local map includes: Determining whether a first local map exists; the first local map is a pre-established local map; If yes, the laser radar point cloud data is merged with the data of the first local map to generate a second local map; If not, the first frame of lidar point cloud data is used as the first local map.
4. The laser radar-camera-inertial navigation joint calibration method according to claim 1, characterized in that: The step of obtaining calibration data in the second local map; wherein the calibration data is rotation and translation information, includes: Obtaining the rotation and translation matrix of each sensor in the second local map; The rotation and translation information is acquired according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
5. The laser radar-camera-inertial navigation joint calibration method according to claim 1, characterized in that: The method of jointly calibrating the multiple sensors by using hand-eye calibration according to the calibration data includes: The calibration model is used to calibrate the hand-eye of multiple sensors according to the calibration data, and the calibration parameters are saved.
6. A laser radar-camera-inertial navigation joint calibration device, characterized in that: The device comprises: A first acquisition module is used to acquire point cloud data from multiple sensors; the sensors include laser radars, image sensors, and inertial sensors; A first generating module, used for matching the laser radar point cloud data with the first local map to generate a second local map; A second acquisition module, used to acquire calibration data in the second local map; the calibration data is rotation and translation information; The joint calibration module is used to perform joint calibration on multiple sensors using hand-eye calibration according to the calibration data.
7. The laser radar-camera-inertial navigation joint calibration device according to claim 6, characterized in that: The device further comprises: a time synchronization submodule, which is used to synchronize the time of multiple sensors before acquiring the point cloud data of the multiple sensors.
8. The laser radar-camera-inertial navigation joint calibration device according to claim 6, characterized in that: The laser radar point cloud data format is a data frame format; the first generation module includes: The first judgment submodule is used to judge whether a first local map exists; the first local map is a pre-established local map; A first generating submodule is used for fusing the laser radar point cloud data with the data of the first local map to generate a second local map; The second generation submodule is used to use the first frame of laser radar point cloud data as the first local map if no.
9. The laser radar-camera-inertial navigation joint calibration device according to claim 6, characterized in that: The second acquisition module is specifically used to acquire the rotation and translation matrix of each sensor in the second local map; The rotation and translation information is acquired according to the rotation and translation matrix; the rotation and translation information includes: displacement information, speed information, and angle information.
10. The laser radar-camera-inertial navigation joint calibration device according to claim 6, characterized in that: The joint calibration module is specifically used to perform hand-eye calibration on multiple sensors according to calibration data using a calibration model, and save calibration parameters.