Multi-LiDAR calibration methods, devices, terminal equipment and storage media

Abstract

This application provides a multi-LiDAR calibration method, apparatus, terminal equipment, and storage medium. First, based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each LiDAR to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each LiDAR are calibrated. Then, based on the feature points of each feature graphic of the artificial feature plate in the coordinate system of each LiDAR, the relative pose of each LiDAR is calibrated. This application has the advantages of high accuracy, fast calculation speed, low cost, simple device, and no complicated algorithm. It is suitable for the calibration of static multi-LiDAR, such as the calibration of multiple LiDAR in the vehicle production process.

Description

Technical Field

[0001] This application belongs to the field of security technology, and in particular relates to a multi-laser radar calibration method, device, terminal equipment and storage medium. Background Technology

[0002] With the continuous development of autonomous driving technology, the types and numbers of sensors equipped on autonomous vehicles are constantly increasing. Among them, LiDAR sensors, with their high distance perception accuracy, ability to perceive the reflectivity of object surfaces in a scene, and strong anti-interference capabilities, are widely used in many autonomous driving scenarios. However, a single LiDAR cannot acquire complete information in an autonomous driving scenario. Therefore, combining multiple LiDARs is essential. In this application scenario, obtaining the positional relationships and coordinate transformations between the various LiDARs becomes particularly important.

[0003] One calibration method for multiple lidar systems uses a black and white grid as a calibration board, but this requires knowing the precise position of the calibration board in the world coordinate system and moving the calibration board. This method is cumbersome and requires high computational performance from the equipment. Summary of the Invention

[0004] This application provides a multi-LiDAR calibration method, apparatus, terminal device, and storage medium, which can solve the problems of cumbersome external parameter calibration process and high equipment requirements among multiple LiDARs in the prior art.

[0005] In one aspect of this application, a multi-LiDAR calibration method is provided, comprising:

[0006] Based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated.

[0007] The relative pose of each lidar is determined based on the feature points of each feature pattern on the artificial feature plate in the coordinate system of each lidar.

[0008] In an optional implementation, it further includes:

[0009] Acquire the initial scan point cloud data in each lidar coordinate system to be calibrated, as well as the initial reflectivity data of the scanned artificial feature plate; the initial scan point cloud data includes the scan point cloud data of the artificial feature plate;

[0010] Using preset planar constraints, planar detection is performed on the initial scan point cloud data and the initial reflectivity data to generate point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated.

[0011] In an optional implementation, before performing planar detection on the initial scanned point cloud data and the initial reflectivity data using preset planar constraints, the method further includes:

[0012] The initial scanned point cloud data is subjected to initial visualization interface selection filtering.

[0013] In an optional implementation, the multi-LiDAR calibration method further includes, before acquiring the initial scan point cloud data:

[0014] For each lidar, the artificial feature plate is set parallel to the lidar's Z-axis and within a set distance from the lidar.

[0015] Acquire the initial scan point cloud data containing the artificial feature plate in the lidar coordinate system, wherein the number of point clouds on all feature patterns of the artificial feature plate in the initial scan point cloud data is higher than a first set point cloud threshold.

[0016] In an optional implementation, before calibrating the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar, the multi-lidar calibration method further includes:

[0017] The artificial feature plate is formed using two materials with different reflectivities.

[0018] In an optional implementation, based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated, including:

[0019] For each lidar, the edge points of each feature graphic in the point cloud data are determined based on the reflectivity data of the artificial feature plate.

[0020] Based on the second set point cloud threshold and the number of edge points, determine whether edge points of the feature graphic on the artificial feature plate have been detected;

[0021] If so, by combining the first graphic feature size and the second graphic feature size, random sampling feature graphic segmentation detection is performed on the edge points to obtain the feature points of each feature graphic;

[0022] If the number of detected feature points is higher than the number of feature graphics on the artificial feature board, the feature points are arranged and combined to form multiple sets of feature point combinations.

[0023] Based on the multiple sets of feature point combinations, the feature points of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system are calibrated.

[0024] In an optional implementation, based on the multiple sets of feature point combinations, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated, including:

[0025] Compare the differences between each set of feature point combinations and the actual feature point combinations of each feature graphic in the artificial feature plate, and determine the feature point combination with the smallest difference by setting a fault tolerance threshold.

[0026] Each feature point in the combination of feature points with the smallest difference is labeled as a feature point of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system.

[0027] In an optional implementation, it further includes:

[0028] If the number of detected feature points is less than the number of feature patterns on the artificial feature plate, or if the combination of feature points does not meet the set fault tolerance threshold, the steps of obtaining the initial scan point cloud data under each lidar coordinate system to be calibrated, and the initial reflectivity data of the artificial feature plate are re-executed until the number of detected feature points is higher than the number of feature patterns on the artificial feature plate.

[0029] In an optional implementation, the feature pattern is a circle or an arc, the circle is a solid structure, the arc is a hollow structure, and the feature point is the center of the circle.

[0030] In an optional implementation, the size of the first graphic feature is smaller than the size of the second graphic feature, and the arc-shaped circle is outside the artificial feature plate.

[0031] In an optional implementation, the step of calibrating the relative pose of each lidar based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar includes:

[0032] The relative poses of multiple lidar units are determined by using the nearest point search method and singular value decomposition algorithm.

[0033] Another embodiment of this application provides a multi-laser radar calibration device, comprising:

[0034] The first calibration module calibrates the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar to be calibrated, based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated.

[0035] The second calibration module calibrates the relative pose of each lidar based on the feature points of each feature pattern on the artificial feature plate in the coordinate system of each lidar.

[0036] In an optional embodiment, the artificial feature plate includes a solid circle and a hollowed-out arc, the center of which is located outside the artificial feature plate, and the arc-shaped edge of which forms the edge of the artificial feature plate.

[0037] Another embodiment of this application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-laser calibration method as described above.

[0038] Another aspect of this application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-laser radar calibration method as described above.

[0039] The beneficial effects of this application embodiment compared with the prior art are as follows: First, based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated. Then, based on the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar, the relative pose of each lidar is calibrated. This has the advantages of high accuracy, fast calculation speed, low cost, simple device, and no complicated algorithm. It is suitable for the calibration of multiple static lidars, such as the calibration of multiple lidars in the vehicle production process. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the system architecture of a multi-LiDAR calibration method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic flowchart of a multi-LiDAR calibration method provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram illustrating a further component of the multi-LiDAR calibration method provided in an embodiment of this application.

[0044] Figure 4 This is a schematic flowchart of step S1 in a multi-LiDAR calibration method provided in an embodiment of this application;

[0045] Figure 5This is a schematic diagram of the structure of an artificial feature plate provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the module structure of the multi-LiDAR calibration device in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0048] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0049] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0050] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0051] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0052] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0054] Current calibration methods for multiple lidar systems use a black and white grid as a calibration board. However, this requires knowing the precise position of the calibration board in the world coordinate system and moving the calibration board. The above methods are cumbersome and require high computational performance from the equipment, thus having shortcomings.

[0055] In view of this, this application provides a multi-LiDAR calibration method, apparatus, terminal device, and storage medium. First, based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each LiDAR to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each LiDAR are calibrated. Then, based on the feature points of each feature graphic of the artificial feature plate in the coordinate system of each LiDAR, the relative pose of each LiDAR is calibrated. This method has the advantages of high accuracy, fast calculation speed, low cost, simple device, and no complicated algorithm. It is suitable for the calibration of static multi-LiDAR, such as the calibration of multiple LiDAR in the vehicle production process.

[0056] The multi-laser calibration method, apparatus, terminal equipment, and storage medium provided in this application are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 The following is a schematic diagram of the architecture of the multi-lidar calibration device provided in the embodiments of this application, such as... Figure 1 As shown, the multi-LiDAR calibration architecture includes: multiple LiDARs 2, an artificial feature plate 1, and a multi-LiDAR calibration device 3. The multi-LiDAR calibration device 3 can calibrate the feature points of each feature pattern of the artificial feature plate in the coordinate system of each LiDAR to be calibrated based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each LiDAR to be calibrated; then, based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each LiDAR, the relative pose of each LiDAR is calibrated.

[0058] In the embodiments of this application, the lidar can be a lidar radar, that is, a laser as the emission light source, which uses photoelectric detection technology for ranging. Each lidar in this application includes a transmitting system, a receiving system, information processing, etc. The transmitting system is composed of various types of lasers, such as carbon dioxide lasers, neodymium-doped yttrium aluminum garnet lasers, semiconductor lasers, and wavelength-tunable solid-state lasers, as well as optical beam expanders; the receiving system uses a combination of a telescope and various types of photodetectors, such as photomultiplier tubes, semiconductor photodiodes, avalanche photodiodes, infrared and visible light multi-element detectors, etc.

[0059] In the specific embodiments of this application, the lidar adopts two working modes: pulse or continuous wave. The detection method can be divided into lidars such as Mie scattering, Rayleigh scattering, Raman scattering, Brillouin scattering, fluorescence, and Doppler, depending on the detection principle. This application does not limit the types of lidars.

[0060] Figure 2 A flowchart illustrating a multi-lidar calibration method is shown, as follows: Figure 2 As shown, it includes:

[0061] S1: Based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, calibrate the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar.

[0062] S2: Based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar, calibrate the relative pose of each lidar.

[0063] The multi-LiDAR calibration method provided in this application first calibrates the feature points of each feature pattern of the artificial feature plate in the coordinate system of each LiDAR to be calibrated based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each LiDAR. Then, based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each LiDAR, the relative pose of each LiDAR is calibrated. This method has the advantages of high accuracy, fast calculation speed, low cost, simple device, and no complicated algorithm. It is suitable for the calibration of static multi-LiDAR, such as the calibration of multiple LiDAR in the vehicle production process.

[0064] In the embodiments of this application, the artificial feature plate may include one or more feature graphics, such as circles, squares, etc., and this application does not limit them.

[0065] Furthermore, in the embodiments of this application, the feature points can be any preset point on the feature graphic, such as the center, centroid, or edge point. This application does not impose any restrictions on this. After the feature points are preset, the specific coordinates of the feature graphic can be traced back based on the coordinates of the feature points.

[0066] In the prior art, artificial feature plates are generally in the shape of a black and white checkerboard, that is, the artificial feature plate is composed of black squares and hollow squares, and then external parameter calibration is performed. Since this application uses feature points for calibration instead of using specific graphic boundaries, the calculation is simpler. In addition, when a circle is used as the feature graphic, the center of the circle and the circular pattern will not be displaced or deformed no matter how the circle is rotated. Therefore, it is most suitable as the feature graphic in this application, and it works in conjunction with the calibration of the feature points of this application.

[0067] The point cloud data of the artificial feature plate in the coordinate system of each lidar to be calibrated, specifically the point cloud data detected by each lidar itself through laser scanning, can be understood as follows: Figure 1 As shown, since the relative positions of the lidar and the artificial feature plate are different, the point cloud data obtained are also different.

[0068] Since the position of each lidar relative to the artificial feature plate is different, when the lidar scans the artificial feature plate, the artificial feature plate is often not parallel to the scanning plane of the lidar. Therefore, the reflectivity of the artificial feature plate is different. The reflectivity is the highest when the lidar is directly facing the artificial feature plate. This application will not elaborate on this point.

[0069] The steps described above in this application will be explained in detail below. In optional embodiments, such as... Figure 3 As shown, it also includes:

[0070] S01: Obtain the initial scanning point cloud data in each lidar coordinate system to be calibrated, as well as the initial reflectivity data of the artificial feature plate; the initial scanning point cloud data includes the scanning point cloud data of the artificial feature plate.

[0071] In this step, the artificial feature plate is placed within the sensing range of the lidar to obtain the true reflectivity data of the object surface within the lidar sensing range, that is, to obtain the initial scan point cloud data and initial reflectivity data of the next frame of the lidar coordinate system to be calibrated.

[0072] In this embodiment, the process of acquiring a frame of point cloud data and reflectivity data in the coordinate system of a lidar to be calibrated is as follows: First, an artificial feature plate made of two different reflectivities is placed perpendicular to the ground within the sensing range of the lidar to be calibrated, satisfying a distance threshold of 1 and parallel to the Z-axis of the lidar, so that the complete artificial feature plate can be displayed in the point cloud display module. The lidar data acquisition module then obtains a frame of point cloud data and reflectivity data scanned by the lidar to be calibrated.

[0073] S02: Using preset planar constraints, perform planar detection on the initial scan point cloud data and the initial reflectivity data to generate point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated.

[0074] Specifically, in this embodiment, the initial scanned point cloud data is subjected to an initial frame selection filtering operation, and the artificial feature plate point cloud data and artificial feature plate reflectivity are processed by random sampling plane detection algorithm and plane constraint condition 1 to obtain artificial feature plate point cloud data and reflectivity data.

[0075] In an optional implementation, before performing planar detection on the initial scanned point cloud data and the initial reflectivity data using preset planar constraints, the method further includes:

[0076] The initial scanned point cloud data is subjected to initial visualization interface selection filtering.

[0077] Specifically, this application can provide a point cloud display interface, manual input control, and manual selection to perform point cloud selection processing on the point cloud data with a minimum range including the manual feature plate, so as to obtain the initially filtered point cloud data.

[0078] In an optional implementation, before acquiring the initial scan point cloud data, the multi-LiDAR calibration method further includes:

[0079] For each lidar, the artificial feature plate is set parallel to the lidar's Z-axis and within a set distance from the lidar.

[0080] Acquire the initial scan point cloud data containing the artificial feature plate in the lidar coordinate system, wherein the number of point clouds on all feature patterns of the artificial feature plate in the initial scan point cloud data is higher than a first set point cloud threshold.

[0081] Specifically, the artificial feature calibration board can be placed within the sensing range of the lidar and positioned perpendicular to the ground, keeping the lidar's Z-axis parallel to the calibration board, so that the feature pattern on the artificial feature board has a number of point clouds that meet the point cloud threshold 1.

[0082] As can be seen, in the optional implementation, the method can be summarized as follows: before calibrating the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar, the multi-lidar calibration method further includes: using two materials with different reflectivities to form the artificial feature plate.

[0083] In the preferred embodiment of this application, the feature pattern is circular. The circle can ensure that the differences in various angles and directions are minimized, so it is not necessary to consider the relative tilt of the artificial feature plate and the lidar in the vertical plane. The prior art uses black and white squares for calibration. Due to the different technical principles, when the black and white squares of the prior art are used in this application, it is also necessary to consider whether the square is upright relative to the lidar. That is, two sides of the square are horizontal relative to the lidar and the other two sides are vertical relative to the lidar. Alternatively, the square and the lidar should be kept in the same orientation, that is, the point cloud of the square is in the "same posture" in the point cloud image of the artificial feature plate captured by any lidar.

[0084] Furthermore, in a preferred embodiment of this application, feature circle 1 and feature circle 2 can be provided. Feature circle 2 can be a virtual feature circle, for example, only a portion of the arc of feature circle 2 is retained on the artificial feature plate. The arc can be hollowed out. Feature circle 1 can be solid. The artificial feature plate includes a solid circle and a hollowed-out arc. The center of the hollowed-out arc is located outside the artificial feature plate, and the arc edge of the arc forms the edge of the artificial feature plate.

[0085] In a preferred embodiment of this application, the radii of the circles and the arcs can be different. In other embodiments, the radii of each circle can be the same or different. Generally, to simplify calculations, it is sufficient to simply set the radii of the circles and the arcs to be different.

[0086] The calibration process using artificial feature plates in this application will be described in detail below.

[0087] In an optional implementation, step S1 involves calibrating the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, such as... Figure 4 ,include:

[0088] S11: For each lidar, determine the edge points of each feature graphic in the point cloud data based on the reflectivity data of the artificial feature plate;

[0089] S12: Determine whether edge points of the feature graphic on the artificial feature plate are detected based on the second set point cloud threshold and the number of edge points;

[0090] S13: If so, combine the first graphic feature size and the second graphic feature size to perform random sampling feature graphic segmentation detection on the edge points to obtain the feature points of each feature graphic;

[0091] S14: If the number of detected feature points is higher than the number of feature graphics on the artificial feature board, the feature points are arranged and combined to form multiple sets of feature point combinations.

[0092] S15: Based on the multiple sets of feature point combinations, calibrate the feature points of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system.

[0093] Specifically, in this embodiment, the initial scanned point cloud data is defined as the first point cloud data, and the point cloud data of the artificial feature plate in the coordinate system of each lidar after planar constraint processing is defined as the second point cloud data. Similarly, the initial reflectivity data is defined as the first reflectivity data, and the reflectivity data of the artificial feature plate in the coordinate system of each lidar after planar constraint processing is defined as the second reflectivity data.

[0094] Furthermore, a first set point cloud threshold is defined as point cloud threshold 1, a second set point cloud threshold is defined as point cloud threshold 2, and the first feature size is smaller than the second feature size.

[0095] Specifically, in step S11, an artificial feature plate made of two different reflectivities is first placed perpendicular to the ground within the sensing range of the lidar to be calibrated, satisfying a distance threshold of 1 and parallel to the Z-axis of the lidar, so that the complete artificial feature plate can be displayed in the point cloud display module. The lidar data acquisition module obtains a frame of point cloud data and reflectivity data scanned by the lidar to be calibrated. The point cloud data and reflectivity data on the artificial feature plate are classified according to the lidar scanning beams to obtain all point cloud data and reflectivity data for each lidar beam.

[0096] Then, using the second point cloud data of the artificial feature plate and the second reflectivity data of the artificial feature plate and the reflectivity threshold 1, the feature circle point cloud and the feature plate point cloud are distinguished to obtain the edge point cloud of the feature circle.

[0097] After obtaining the edge point cloud of the feature circle, the reflectivity difference between the point clouds of each lidar beam is calculated, and the edge points of the feature circle are obtained by filtering through a reflectivity threshold of 1.

[0098] In step S12, it is determined whether the edge points of the feature circle meet the point cloud threshold 2. If not, the calibration is performed again, that is, the operation is restarted from S01.

[0099] In step S13, as Figure 5As shown, assuming there are 3 circles and 4 arcs on the artificial feature plate, the edge points are randomly sampled for feature graphic segmentation detection based on the first and second graphic feature dimensions to obtain the feature points of each feature graphic. Specifically, the edge points are randomly sampled for circular segmentation detection using a circle radius threshold of 1 to obtain a group of undetermined circle centers with a number of circle centers greater than or equal to 3. If the number of undetermined circle centers is less than 3, the edge points are randomly sampled for circular segmentation detection using a circle radius threshold of 2 to obtain a group of undetermined circle centers with a number of circle centers greater than or equal to 4. If the number of undetermined circle centers is less than 4, a frame of point cloud data and reflectivity data from the LiDAR scan to be calibrated is re-obtained.

[0100] That is, by using two different shapes, a circle and an arc, and placing the center of the arc outside the artificial feature plate, the center of the circle can be detected by two circle radius thresholds. When the number of the center of the two circles is less than the actual number, the calibration is reconfigured.

[0101] Then in step S14, if the number of detected feature points is higher than the number of feature graphics on the artificial feature board, the feature points are arranged and combined to form multiple sets of feature point combinations. Specifically, for the undetermined circle center group, multiple sets of circle center groups with a quantity of 3 or 4 are obtained by permutation and combination.

[0102] Then, in step S15, for the multiple sets of center groups, specifically, according to the combination of multiple sets of feature points, the feature points of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system are calibrated. First, a geometric judgment can be made using the fault tolerance threshold 1 to exclude center groups that do not meet the conditions until a set of center groups with the best geometric distribution is obtained. If a set of center groups with the best geometric distribution cannot be obtained, then a frame of point cloud data and reflectivity data of the lidar scan to be calibrated is obtained again, and the calculation is repeated until a set of center groups with the best geometric distribution is obtained.

[0103] In an optional implementation, based on the multiple sets of feature point combinations, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated, including:

[0104] Compare the differences between each set of feature point combinations and the actual feature point combinations of each feature graphic in the artificial feature plate, and determine the feature point combination with the smallest difference by setting a fault tolerance threshold.

[0105] Each feature point in the combination of feature points with the smallest difference is labeled as a feature point of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system.

[0106] In this embodiment, after obtaining the optimal set of center points, a lidar to be calibrated is manually selected as the main lidar. Using the pose transformation matrix estimation module, the center point sets of each lidar coordinate system other than the main lidar are registered with the center point set of the main lidar using ICP to estimate the pose transformation matrix, thereby completing the extrinsic parameter calibration.

[0107] It is understandable that, in computer implementation, if the number of detected feature points is less than the number of feature patterns on the artificial feature plate, or if the combination of feature points does not meet the set fault tolerance threshold, the steps of obtaining the initial scanning point cloud data under each lidar coordinate system to be calibrated, and scanning the initial reflectivity data of the artificial feature plate, can be re-executed until the number of detected feature points is higher than the number of feature patterns on the artificial feature plate. This application will not elaborate on this.

[0108] In an optional implementation, the step of calibrating the relative pose of each lidar based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar includes:

[0109] The relative poses of multiple lidar units are determined by using the nearest point search method and singular value decomposition algorithm.

[0110] In some specific embodiments, the ICP registration includes an SVD decomposition method, specifically...

[0111] In this application, ICP (Iterative Closest Point) is used, which has high accuracy and does not require feature point extraction. Furthermore, since this application selects the optimal combination for feature circle center combination, it is equivalent to completing coarse registration and will not lead to getting trapped in local optima.

[0112] This application utilizes the ICP algorithm to repeatedly select corresponding point pairs and calculate the optimal rigid body transformation until the convergence accuracy requirement for correct registration is met. Specifically, by finding rotation and translation parameters, the point clouds in two different coordinate systems are rotated and translated, with one point cloud coordinate system (the main lidar coordinate system) as the global coordinate system and the other point cloud (other lidar coordinate systems) so that the overlapping parts of the two sets of point clouds completely overlap. The input of the algorithm is the coordinates of one of the centers in the center group in the main lidar coordinate system and the target point cloud, i.e., the optimal center group in the main lidar coordinate system and the optimal center group in other lidar coordinate systems. The output of the algorithm is the rotation and translation matrix, i.e., the pose transformation matrix.

[0113] The specific process for calculating the pose transformation matrix is ​​as follows:

[0114] Assume that point cloud {Q} is the target point cloud (the coordinates of one of the centers in the center group under the main lidar coordinate system), {P} is the source point cloud (the coordinates of the center in the center group under the other lidar coordinate systems to be registered), pi (i∈1,2,...N), and qi is the point in {E} that is closest to pi.

[0115] Next, calculate the RT transformation matrix from {P} to {Q}, which is the rotation matrix R and the translation matrix T. If the transformation parameters are accurate, then every point pi in the point cloud {P} should completely coincide with a point qi in the point cloud {Q} after the transformation, i.e., qi = Rpi + T. However, due to the presence of noise, it is impossible for all points to completely coincide. Define an objective function, and the R and T that minimize the objective function are the desired transformation parameters.

[0116] First, for each point pi in {P}, find its nearest point qi in {Q} (this can be achieved using the kd-tree nearest neighbor search algorithm) to form a one-to-one point pair. Calculate the centroids of the two point clouds, denoted as Up and Uq, respectively. Decentrate the two point clouds and then construct a matrix H. Perform SVD decomposition on the H matrix to obtain R and T. After obtaining the R and T matrices, use them to perform spatial transformation on the space to be registered to obtain a new point set. Substitute this into the objective function. If the average distance between the new transformed point set and the reference point set is less than a given threshold, stop the iterative calculation. Otherwise, use the new transformed point set as the new {Pi} and continue iterating until the requirements of the objective function are met.

[0117] Then, the pose transformation matrix estimation module calculates the relative pose of each pair of lidars based on the feature circle center group, thereby completing the calibration. That is, since the above RT matrix can be used to know the coordinates of each point on the lidar in the corresponding main lidar, the relative pose of any two lidars can be determined by the coordinates of the corresponding points.

[0118] The specific scenarios described in this application will be explained in detail below.

[0119] Step 1: Place the artificial feature plate within the sensing range of the lidar, and use the lidar data acquisition module to acquire the next frame of point cloud data and reflectivity data of the lidar coordinate system to be calibrated.

[0120] Step 2 uses the operable point cloud display module and the artificial feature plate point cloud data and reflectance data extraction module to preprocess the point cloud data and reflectance data to obtain the point cloud data and reflectance data on the artificial feature plate.

[0121] Step 3 uses the feature circle edge point detection module to perform edge point detection processing on the point cloud data and reflectance data on the artificial feature plate to determine the edge point cloud of the feature circle.

[0122] Step 4: Use the feature center detection module to detect the center of the edge points and obtain the feature center group;

[0123] Step 5: Repeat steps 1-4 to obtain the feature circle center set in the coordinate system of each lidar to be calibrated, and save them respectively;

[0124] Step 6 uses the pose transformation matrix estimation module to calculate the relative pose of each pair of lidars based on the feature circle center group, thereby completing the calibration.

[0125] The process of obtaining a frame of point cloud data and reflectivity data in the coordinate system of a lidar to be calibrated in step 1 is as follows: First, an artificial feature plate made of two different reflectivities is placed perpendicular to the ground and within the sensing range of the lidar to be calibrated, satisfying the distance threshold of 1 and parallel to the Z-axis of the lidar, so that the complete artificial feature plate can be displayed in the point cloud display module. The lidar data acquisition module obtains a frame of point cloud data and reflectivity data scanned by the lidar to be calibrated.

[0126] Step 2 uses the point cloud display module and the artificial feature plate point cloud data and reflectance data extraction module to preprocess the point cloud data and reflectance data as follows:

[0127] Step 21 involves an operable point cloud display module, controlled by a mouse and keyboard, to manually select points within a minimum range, including the manual feature plate, to obtain initially filtered point cloud data.

[0128] Step 22: For the point cloud data of the first filtering, use the random sampling plane detection algorithm and plane constraint 1 in the data extraction module to perform data extraction processing on the artificial feature plate point cloud data and artificial feature plate reflectivity to obtain artificial feature plate point cloud data and reflectivity data.

[0129] The specific operation of step 3, which utilizes the feature circle edge point detection module to perform edge point detection processing on the point cloud data and reflectance data of the artificial feature plate, is as follows:

[0130] Step 31 classifies the point cloud data and reflectance data on the artificial feature plate according to the scanning beam of the lidar, and obtains all the point cloud data and reflectance data of each lidar beam.

[0131] Step 32 calculates the reflectivity difference between the point clouds of each lidar beam;

[0132] Step 33: Obtain the edge points of the feature circle by filtering through a reflectivity threshold of 1;

[0133] Step 34 determines whether the edge points of the feature circle satisfy the point cloud threshold 2. If not, proceed to step 1.

[0134] The specific process of step 4, which uses the feature circle center group detection module to detect the center of the edge points, is as follows:

[0135] Step 41 uses a circle radius threshold of 1 to perform random sampling circular segmentation detection on the edge points. If a group of undetermined circle centers with a number of circle centers greater than or equal to 3 is obtained, proceed to step 42. If the number of undetermined circle centers is less than 3, then a circle radius threshold of 2 is used to perform random sampling circular segmentation detection on the edge points. If a group of undetermined circle centers with a number of circle centers greater than or equal to 4 is obtained, proceed to step 42. If the number of undetermined circle centers is less than 4, proceed to step 1 to re-obtain a frame of point cloud data and reflectivity data from the LiDAR scan to be calibrated.

[0136] Step 42: For the undetermined circle center group, use permutations and combinations to obtain multiple circle center groups with a quantity of 3 or 4.

[0137] Step 43 uses the fault tolerance threshold 1 to make a geometric judgment on the multiple sets of circle center groups, and excludes circle center groups that do not meet the conditions until the set of circle center groups with the best geometric distribution is obtained; if the set of circle center groups with the best geometric distribution cannot be obtained, then proceed to step 1 to obtain a frame of point cloud data and reflectivity data of the LiDAR scan to be calibrated, and recalculate until the set of circle center groups with the best geometric distribution is obtained.

[0138] In step 5, a lidar to be calibrated is manually selected as the main lidar. Using the pose transformation matrix estimation module, the center groups of circles in the coordinate system of each lidar other than the main lidar are compared with the center group of the main lidar using ICP registration to estimate the pose transformation matrix, thereby completing the extrinsic parameter calibration. The ICP registration includes the SVD decomposition method.

[0139] It can be seen that the beneficial effects of the present invention are: the device is simple and there is no complicated algorithm; it uses reflectivity to find the edge point of the feature circle, which solves the problem that the trailing point cannot be correctly found; it can calibrate various types of multi-laser lidar; and it can quickly and accurately calibrate multi-laser lidar.

[0140] Based on the same inventive concept, a multi-laser radar calibration device 10, such as Figure 6 As shown, it includes: a first calibration module 1, which calibrates the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar according to the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar; and a second calibration module 2, which calibrates the relative pose of each lidar according to the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar.

[0141] The lidar calibration device provided in this application, through the setting of a first calibration module and a second calibration module, firstly calibrates the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar to be calibrated based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar. Then, based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar, the relative pose of each lidar is calibrated. Thus, it has the beneficial effects of high accuracy, fast calculation speed, low cost, simple device, and no complicated algorithm. It is suitable for the calibration of multiple static lidars, such as the calibration of multiple lidars in the vehicle production process.

[0142] In an optional embodiment, the artificial feature plate includes a solid circle and a hollowed-out arc, the center of which is located outside the artificial feature plate, and the arc-shaped edge of which forms the edge of the artificial feature plate.

[0143] In the preferred embodiment of this application, the feature pattern is circular. The circle can ensure that the differences in various angles and directions are minimized, so it is not necessary to consider the relative tilt of the artificial feature plate and the lidar in the vertical plane. The prior art uses black and white squares for calibration. Due to the different technical principles, when the black and white squares of the prior art are used in this application, it is also necessary to consider whether the square is upright relative to the lidar. That is, two sides of the square are horizontal relative to the lidar and the other two sides are vertical relative to the lidar. Alternatively, the square and the lidar should be kept in the same orientation, that is, the point cloud of the square is in the "same posture" in the point cloud image of the artificial feature plate captured by any lidar.

[0144] Furthermore, in a preferred embodiment of this application, feature circle 1 and feature circle 2 can be provided. Feature circle 2 can be a virtual feature circle, for example, only a portion of the arc of feature circle 2 is retained on the artificial feature plate. The arc can be hollowed out. Feature circle 1 can be solid. The artificial feature plate includes a solid circle and a hollowed-out arc. The center of the hollowed-out arc is located outside the artificial feature plate, and the arc edge of the arc forms the edge of the artificial feature plate.

[0145] In a preferred embodiment of this application, the radii of the circles and the arcs can be different. In other embodiments, the radii of each circle can be the same or different. Generally, to simplify calculations, it is sufficient to simply set the radii of the circles and the arcs to be different.

[0146] The multi-lidar calibration method and apparatus implemented by other entities in the above embodiments have the same or similar effects as those in the foregoing embodiments of this application, and will not be described in detail here.

[0147] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0149] To implement the above embodiments, this application also proposes a terminal device. Figure 7 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Figure 7 As shown, the terminal device 600 includes:

[0150] The system includes a memory 610 and at least one processor 620, and a bus 630 connecting different components (including the memory 610 and the processor 620). The memory 610 stores a computer program, which, when executed by the processor 620, implements the multi-laser calibration method described in the embodiments of this application.

[0151] Bus 630 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0152] Terminal device 600 typically includes various electronically readable media. These media can be any available media that can be accessed by terminal device 600, including volatile and non-volatile media, removable and non-removable media.

[0153] Memory 610 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 640 and / or cache memory 650. Terminal device 600 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 660 can be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 630 via one or more data media interfaces. Memory 610 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0154] A program / utility 680 having a set (at least one) of program modules 670 may be stored in, for example, memory 610. Such program modules 670 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 670 typically perform the functions and / or methods described in the embodiments of this application.

[0155] Terminal device 600 can also communicate with one or more external devices 690 (e.g., keyboard, pointing device, display 691, etc.), one or more devices that enable a user to interact with terminal device 600, and / or any device that enables terminal device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 692. Furthermore, terminal device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 693. As shown, network adapter 693 communicates with other modules of terminal device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] The processor 620 executes various functional applications and data processing by running programs stored in the memory 610.

[0157] It should be noted that the implementation process and technical principles of the terminal device in this embodiment are explained in the foregoing description of the multi-LiDAR calibration method in this application embodiment, and will not be repeated here.

[0158] This application also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0159] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calibrating multiple lidar sensors, characterized in that, include: Based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated; the feature graphic includes an arc shape, and the feature point is the center of the circle. The relative pose of each lidar is determined based on the feature points of each feature pattern on the artificial feature plate in the coordinate system of each lidar.

2. The method according to claim 1, characterized in that, Also includes: Acquire the initial scan point cloud data in each lidar coordinate system to be calibrated, as well as the initial reflectivity data of the scanned artificial feature plate; The initial scanned point cloud data includes the scanned point cloud data of the artificial feature plate; Using preset planar constraints, planar detection is performed on the initial scan point cloud data and the initial reflectivity data to generate point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated.

3. The method according to claim 2, characterized in that, Before acquiring the initial scan point cloud data, the multi-LiDAR calibration method further includes: For each lidar, the artificial feature plate is set parallel to the lidar's Z-axis and within a set distance from the lidar. Acquire the initial scan point cloud data containing the artificial feature plate in the lidar coordinate system, wherein the number of point clouds on all feature patterns of the artificial feature plate in the initial scan point cloud data is higher than a first set point cloud threshold.

4. The method according to claim 1, characterized in that, Based on the point cloud data and reflectivity data of the artificial feature plate in the coordinate system of each lidar to be calibrated, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated, including: For each lidar, the edge points of each feature graphic in the point cloud data are determined based on the reflectivity data of the artificial feature plate. Based on the second set point cloud threshold and the number of edge points, determine whether edge points of the feature graphic on the artificial feature plate have been detected; If so, by combining the first graphic feature size and the second graphic feature size, random sampling feature graphic segmentation detection is performed on the edge points to obtain the feature points of each feature graphic; If the number of detected feature points is higher than the number of feature graphics on the artificial feature board, the feature points are arranged and combined to form multiple sets of feature point combinations. Based on the combination of multiple sets of feature points, the feature points of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system are calibrated.

5. The method according to claim 4, characterized in that, Based on the multiple sets of feature point combinations, the feature points of each feature graphic of the artificial feature plate in the coordinate system of each lidar are calibrated, including: Compare the differences between each set of feature point combinations and the actual feature point combinations of each feature graphic in the artificial feature plate, and determine the feature point combination with the smallest difference by setting a fault tolerance threshold. Each feature point in the combination of feature points with the smallest difference is labeled as a feature point of each feature graphic of the artificial feature plate in the corresponding lidar coordinate system.

6. The method according to claim 5, characterized in that, Based on the reflectivity data of the artificial feature plate, the edge points of each feature shape in the point cloud data are determined, including: The point cloud data and reflectance data on the artificial feature plate are classified according to the scanning beams of the lidar to obtain all the point cloud data and reflectance data of each lidar beam. Calculate the reflectivity difference between the point clouds of the line beams of each lidar unit; Edge points of the feature image are obtained by filtering based on a preset reflectivity threshold.

7. The method according to claim 5, characterized in that, The feature graphic includes a circle, which is a solid structure, and an arc shape, which is a hollow structure.

8. The method according to claim 1, characterized in that, The step of calibrating the relative pose of each lidar based on the feature points of each feature pattern of the artificial feature plate in the coordinate system of each lidar includes: The relative poses of multiple lidar units are determined by using the nearest point search method and singular value decomposition algorithm.

9. A multi-laser radar calibration device, characterized in that, include: The first calibration module calibrates the feature points of each feature graphic of the artificial feature board in the coordinate system of each lidar to be calibrated, based on the point cloud data and reflectivity data of the artificial feature board in the coordinate system of each lidar to be calibrated; the feature graphic includes an arc, and the feature point is the center of the circle. The second calibration module calibrates the relative pose of each lidar based on the feature points of each feature pattern on the artificial feature plate in the coordinate system of each lidar.

10. The multi-laser radar calibration device according to claim 9, characterized in that, The artificial feature plate also includes a solid circle, and the arc is a hollow arc. The center of the hollow arc is located on the outside of the artificial feature plate, and the arc edge of the arc forms the edge of the artificial feature plate.

Images