Calibration method and device of multiple laser radars, electronic equipment and vehicle

Abstract

The application relates to the technical field of automatic driving, and provides a multi-laser-radar calibration method and device, electronic equipment and a vehicle. The method comprises the following steps: plane features of point cloud data are extracted, meanwhile, for each first plane feature, matching is performed in a second plane feature to obtain matching plane features, then, a plane feature matching pair between each first plane feature and the matching plane features is constructed; optimal rotation parameters and optimal translation parameters from a secondary laser-radar coordinate system to a primary laser-radar coordinate system are determined according to the plane feature matching pair, and the secondary laser-radar is calibrated according to the optimal rotation parameters and the optimal translation parameters. The application can solve the problems of limited local constraint capability of point features and insufficient constraint on translation degrees of freedom of line features, improve the robustness of the calibration process to noise interference, and make the final optimal rotation parameters and optimal translation parameters more accurate and more stable.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a calibration method, device, electronic device and vehicle for multiple lidar sensors. Background Technology

[0002] In the field of autonomous driving, LiDAR serves as a crucial environmental perception sensor, providing high-precision 3D point cloud data. As application scenarios become increasingly complex, a single LiDAR often cannot meet the demands of comprehensive environmental perception; therefore, multi-LiDAR systems are widely adopted. However, relative pose deviations exist between multiple LiDARs. Without precise calibration, this can lead to inaccurate alignment of point cloud data, severely impacting subsequent functions such as environmental modeling and target detection.

[0003] In related technologies, calibration methods for multi-LiDAR systems generally include point feature-based calibration methods and line feature-based calibration methods. However, point feature-based calibration methods are not robust enough in feature localization under noise interference and can only provide local geometric constraints, with limited global pose estimation capabilities. Line feature-based calibration methods have weak constraints on translational degrees of freedom along the straight line direction, and pose solving is subject to under-constraint risks. Summary of the Invention

[0004] In view of the shortcomings of the prior art, this application provides a calibration method, device, electronic device and vehicle for multiple lidars, to solve the problems that point feature-based calibration methods have insufficient robustness in feature localization under noise interference and can only provide local geometric constraints, with limited global pose estimation capabilities; line feature-based calibration methods have weak constraints on translational degrees of freedom along the straight line direction, and pose solving has the risk of underconstraint.

[0005] Firstly, this application provides a calibration method for multiple lidar systems, comprising: The first point cloud obtained by the main lidar scanning three mutually perpendicular planes and the second point cloud obtained by the secondary lidar scanning three mutually perpendicular planes are acquired. Based on the initial extrinsic parameters of the secondary lidar relative to the primary lidar, the second point cloud is transformed to the coordinate system of the primary lidar. Planar features are extracted from the first point cloud and the transformed second point cloud respectively to obtain the first planar features and the second planar features corresponding to three mutually perpendicular planes; For each first planar feature, a matching is performed in the second planar feature to obtain a matching planar feature, and a planar feature matching pair is constructed between each first planar feature and the matching planar feature; The external parameters to be optimized for the secondary lidar relative to the primary lidar are determined. These external parameters include rotation and translation parameters. Based on the planar feature matching pairs, rotation parameters, and translation parameters, the optimal rotation and translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system are determined. The secondary lidar is then calibrated based on the optimal rotation and translation parameters.

[0006] In one embodiment of this application, for each first planar feature, matching is performed on the second planar features to obtain matching planar features, including: For each first planar feature, determine the angle between the plane normal vector and the second planar feature, and select the second planar feature whose plane normal vector angle is less than a preset angle threshold as the initial matching planar feature; determine the bidirectional average point-to-plane distance between the first planar feature and the initial matching planar feature, and if the bidirectional average point-to-plane distance is less than a preset distance threshold, then use the initial matching planar feature as the matching planar feature of the first planar feature.

[0007] In one embodiment of this application, the first planar feature includes a first planar equation and a first planar point cloud, and the matching planar feature includes a second planar equation and a second planar point cloud; Based on the planar feature matching pairs, rotation parameters, and translation parameters, the optimal rotation parameters and optimal translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system are determined. This includes: for each planar feature matching pair, using the first plane equation as the reference plane, projecting each point in the second plane point cloud to the primary lidar coordinate system using the rotation and translation parameters; calculating the point-to-plane residual from the projected point to the reference plane, and constructing a cost function with the sum of squares of the point-to-plane residuals of all planar feature matching pairs as the objective; iteratively optimizing the cost function using the initial extrinsic parameters as the initial values ​​for iteration until the cost function converges, thus obtaining the optimal rotation parameters and optimal translation parameters.

[0008] In one embodiment of this application, the calibration of the secondary lidar is performed based on the optimal rotation parameters and the optimal translation parameters, including: for each plane feature matching pair, obtaining the second plane point cloud in the matching plane features; and transforming each point in the second plane point cloud to the coordinate system of the primary lidar using the optimal rotation parameters and the optimal translation parameters to calibrate the secondary lidar.

[0009] In one embodiment of this application, after calibrating the secondary lidar based on the optimal rotation parameters and the optimal translation parameters, the method further includes: for each planar feature matching pair, calculating the first average distance and the first standard deviation from each point in the first planar point cloud to the reference plane; calculating the second average distance and the second standard deviation from each point in the point cloud data obtained by the calibrated secondary lidar to the reference plane; and verifying the calibration result of the secondary lidar based on the first average distance, the second average distance, and a preset average distance threshold, as well as the first standard deviation, the second standard deviation, and a preset standard deviation threshold.

[0010] In one embodiment of this application, the initial extrinsic parameters include an initial rotation matrix and an initial translation vector. Before transforming the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the main lidar, the method further includes: obtaining the installation angle of the secondary lidar relative to the main lidar, and determining the initial rotation matrix based on the installation angle; obtaining the spatial distance between the origin of the secondary lidar and the origin of the main lidar, and determining the initial translation vector based on the spatial distance.

[0011] Secondly, this application also provides a calibration device for multiple lidar systems, comprising: The acquisition module is configured to acquire the first point cloud obtained by the main lidar scanning three mutually perpendicular planes and the second point cloud obtained by the secondary lidar scanning three mutually perpendicular planes; The conversion module is configured to convert the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the main lidar. The extraction module is configured to extract planar features from the first point cloud and the transformed second point cloud respectively, to obtain the first planar features and the second planar features corresponding to three mutually perpendicular planes; The matching module is configured to perform matching on the second plane feature for each first plane feature to obtain the matching plane feature, and to construct a plane feature matching pair between each first plane feature and the matching plane feature; The calibration module is configured to determine the external parameters to be optimized for the secondary lidar relative to the primary lidar. These external parameters include rotation and translation parameters. Based on the planar feature matching pairs, rotation parameters, and translation parameters, the module determines the optimal rotation and translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system, and calibrates the secondary lidar based on these optimal parameters.

[0012] Thirdly, this application also provides an electronic device, which includes: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the steps of the above-described method.

[0013] Fourthly, this application also provides a vehicle, the vehicle including: an electronic device, which, when executed, causes the vehicle to perform the steps of the above method.

[0014] The beneficial effects of this technical solution are as follows: First, a first point cloud is obtained by scanning three mutually perpendicular planes using a main lidar, and a second point cloud is obtained by scanning three mutually perpendicular planes using a secondary lidar. Based on the initial extrinsic parameters of the secondary lidar relative to the main lidar, the second point cloud is transformed to the coordinate system of the main lidar. Planar features are extracted from both the first and transformed point clouds to obtain first and second planar features corresponding to the three mutually perpendicular planes. Extracting planar features effectively solves the problem of unstable positioning of point and line features under noise interference. Simultaneously, for each first planar feature, matching is performed on the second planar features to obtain matching planar features. Then, a planar feature matching pair is constructed between each first planar feature and the matching planar feature. Global geometric constraints address the limited local constraint capability of point features and the insufficient constraint on translational degrees of freedom for line features. Finally, based on the planar feature matching pair, rotation parameters, and translation parameters, the optimal rotation and translation parameters from the secondary lidar coordinate system to the main lidar coordinate system are determined, and the secondary lidar is calibrated based on these optimal parameters. This improves the robustness of the calibration process to noise interference and also compensates for the constraints of traditional point and line feature calibration methods, resulting in higher accuracy and stronger stability of the final optimal rotation and translation parameters, thereby improving the accuracy of lidar calibration.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a schematic flowchart illustrating a calibration method for multiple lidar sensors, as shown in an exemplary embodiment of this application. Figure 2 This is a schematic diagram illustrating the structure of a calibration device for multiple lidar systems, as shown in an exemplary embodiment of this application. Figure 3 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0017] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0019] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0020] It is understood that the multi-LiDAR calibration method provided in this example is applied to vehicles, including new energy vehicles, passenger vehicles (such as cars, buses, coaches, minibuses, etc.), cargo vehicles (such as ordinary trucks, box trucks, trailer trucks, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks), special vehicles (such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, excavators, bulldozers, loaders, road rollers, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.), recreational vehicles (such as amusement vehicles, amusement park autonomous driving devices, balance bikes, etc.), and rescue vehicles (such as fire trucks, ambulances, power repair vehicles, engineering emergency rescue vehicles, etc.).

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating a calibration method for multiple lidar systems, as shown in an exemplary embodiment of this application. Figure 1 As shown, in an exemplary embodiment, the calibration method for multiple lidars includes steps S110 to S150, and each step is described in detail below.

[0022] S110, acquire the first point cloud obtained by the main lidar scanning three mutually perpendicular planes and the second point cloud obtained by the secondary lidar scanning three mutually perpendicular planes; Specifically, prior to executing S110, lidar can be installed in appropriate locations on the vehicle. For example, the lidar installed in the vehicle includes a main lidar (e.g., 128 lines), a left secondary lidar (e.g., 32 lines), and a right secondary lidar (e.g., 32 lines). The main lidar can be installed at the front of the roof, in the middle position near the top of the windshield, with the left and right secondary lidars distributed on either side of the main lidar.

[0023] It is understandable that the main LiDAR can also be installed in other locations within the vehicle, such as... The number and installation position of the secondary LiDAR can be adjusted according to the actual application scenario, either on the left or right side of the front bumper. For example, they can be installed below the side mirrors on both sides of the vehicle, or on the left and right fenders of the vehicle body. As long as the primary and secondary LiDARs can cover the three mutually perpendicular planes required for calibration, the installation position will not affect the calibration logic of this method. The calibration parameters can be adjusted according to the actual installation position.

[0024] After installing the main and secondary lidars at their respective locations on the vehicle, the measurement center of the main lidar can be used as the origin. The vertical axis is upward. Positive axis direction The positive axis direction is consistent with the longitudinal direction of the vehicle body and is determined according to the right-hand Cartesian coordinate system criterion. Establish the main radar coordinate system along the positive axis. ; Meanwhile, the measurement center of the left secondary lidar is taken as the origin. , The axis is perpendicular to the lidar base and points upwards, in the ideal installation state. shaft and The axes are parallel and in the same direction, determined according to the right-hand Cartesian coordinate system criterion. Establish the left sub-laser radar coordinate system along the positive axis. The coordinate system definition method for the right secondary lidar can refer to that of the left secondary lidar, and will not be elaborated further here.

[0025] When acquiring point cloud data, the three mutually perpendicular planes can be selected from regular wall corners in an indoor space, such as a three-wall corner structure with two perpendicular walls in a room. The goal is to ensure that the perpendicularity between the three planes meets the calibration requirements. The main and secondary lidars then scan these three planes respectively to obtain the corresponding first and second point clouds.

[0026] It is understandable that, when acquiring the first and second point clouds, hardware time synchronization of each lidar can be performed using the second pulse signal output by the Global Navigation Satellite System receiver and the Recommended Minimum Specific GPS / TRANSIT Data (GPRMC). Based on the synchronization clock triggering data acquisition, time-aligned point cloud data sets of the main, left, and right lidars are obtained. ,in , and These represent the point cloud data collected by the main lidar, left sub-library lidar, and right sub-library lidar within the same time period.

[0027] In addition, in order to reduce computing power and improve fitting accuracy, after the initial point cloud obtained by scanning three mutually perpendicular planes by the main lidar and the secondary lidar, redundant point clouds in the initial point cloud can be removed to filter out invalid and repetitive information in the initial point cloud, retain key features, and obtain a cleaner first point cloud and second point cloud.

[0028] In some examples, after the main and secondary lidars scan three mutually perpendicular planes to obtain the initial point cloud, the point cloud data sets of the main, left secondary, and right secondary lidars can be combined. Convert to Point Cloud Data (PCD) format for data preprocessing.

[0029] Select three frames of raw point cloud data corresponding to the same timestamp t. As a set of calibration data, open the main LiDAR point cloud file using CloudCompare software. By using the polygon selection function, redundant points in the point cloud that are unrelated to the three mutually perpendicular planes are deleted one by one, leaving only the point cloud containing the three perpendicular planes, thus obtaining the processed main radar segmentation point cloud. (First point cloud). Similarly, for the left sub-laser point cloud file... Perform the same redundant point removal operation, retaining only the point cloud containing only three vertical planes, to obtain the processed left sub-radar segmented point cloud. (Second point cloud); similarly, for the right secondary lidar point cloud file Perform the same redundant point removal operation, retaining only the point cloud containing three vertical planes, to obtain the processed right sub-radar segmented point cloud. (Second point cloud).

[0030] S120, based on the initial extrinsic parameters of the secondary lidar relative to the primary lidar, transform the second point cloud to the coordinate system of the primary lidar; In some embodiments, the initial extrinsic parameters include an initial rotation matrix and an initial translation vector. Before transforming the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the primary lidar, the method further includes: Obtain the installation angle of the secondary lidar relative to the primary lidar, and determine the initial rotation matrix based on the installation angle; obtain the spatial distance between the origin of the secondary lidar and the origin of the primary lidar, and determine the initial translation vector based on the spatial distance.

[0031] Specifically, taking the left sub-library lidar as an example, the installation angle of the left sub-library lidar relative to the main lidar can be measured using an inclinometer. Based on the installation angle, the initial rotation matrix of the left sub-library lidar to the main lidar can be determined. The initial translation vector was determined by measuring the spatial distance between the origin of the left secondary lidar and the origin of the main lidar using a steel tape measure. Then based on the initial rotation matrix and initial translation vector The left secondary lidar segmented the point cloud. Transform to the main lidar coordinate system to obtain the transformed point cloud. : ,in Segmenting point clouds for left-side secondary lidar Similarly, at any point in the diagram, the installation angle of the right secondary lidar relative to the main lidar can be measured using an inclinometer. Based on this installation angle, the initial rotation matrix from the right secondary lidar to the main lidar can be determined. The initial translation vector was determined by measuring the spatial distance between the origin of the right secondary lidar and the origin of the main lidar using a steel tape measure. Then, the point cloud segmented by the right secondary lidar is transformed into the coordinate system of the main lidar.

[0032] S130, perform planar feature extraction on the first point cloud and the transformed second point cloud respectively to obtain the first planar feature and the second planar feature corresponding to the three mutually perpendicular planes; Specifically, when extracting planar features from the first point cloud and the transformed second point cloud respectively, a random sample consensus algorithm can be used. For example, the random sample consensus algorithm can be used to extract features from the first point cloud. Planar feature extraction is performed to obtain three sets of planar features corresponding to three mutually perpendicular planes. Each set of planar features includes the plane equation and the corresponding point cloud, denoted as . ,in, The equation of a plane can be expressed as: , This is a subset of the point cloud corresponding to the plane. Let T be the unit normal vector in the main radar coordinate system, and T be the transpose matrix. Let be any point on the plane. The distance from the plane to the origin of the main radar coordinate system. This represents the i-th plane; similarly, the same method is used for the second point cloud of the left sub-laser. Planar feature extraction was performed, resulting in three sets of planar features, denoted as... ,in, The equation of a plane can be expressed as: , This is a subset of the point cloud corresponding to the plane. Describes the j-th plane. The unit normal vector in the main radar coordinate system. Let be any point on the plane. The distance from the plane to the origin of the main radar coordinate system; the same method is used for the second point cloud of the right secondary lidar. Planar feature extraction was performed, resulting in three sets of planar features, denoted as... ,in, The equation of a plane can be expressed as: , This is a subset of the point cloud corresponding to the plane. Describes the k-th plane. The unit normal vector in the main radar coordinate system. Let be any point on the plane. This represents the distance from the plane to the origin of the main radar coordinate system. Thus, a total of nine sets of planar features were extracted from the three frames of lidar data.

[0033] S140, for each first planar feature, match it in the second planar feature to obtain the matching planar feature, and construct a planar feature matching pair between each first planar feature and the matching planar feature; In some embodiments, for each first planar feature, matching is performed on the second planar features to obtain matching planar features, including: For each first planar feature, determine the angle between the plane normal vector and the second planar feature, and select the second planar feature whose plane normal vector angle is less than a preset angle threshold as the initial matching planar feature; determine the bidirectional average point-to-plane distance between the first planar feature and the initial matching planar feature, and if the bidirectional average point-to-plane distance is less than a preset distance threshold, then the initial matching planar feature is used as the matching planar feature of the first planar feature.

[0034] Specifically, taking the left secondary lidar as an example, during planar matching, the plane of the primary lidar is calculated. With the left sub-laser plane The angle between the normal vectors:

[0035] This represents the angle between the normal vector of the i-th plane of the main lidar and the j-th plane of the left sub-library lidar. The angles are calculated pairwise between the three planes of the main lidar and the three planes of the left sub-library lidar, resulting in a total of 9 angle values.

[0036] Then, an angle threshold is given in advance. For each plane of the main lidar In the three planes of the left sub-LiDAR, search for the condition that satisfies... The candidate planes are selected, and the one with the smallest included angle is chosen as the corresponding initial matching plane. ,Right now Thus, an initial matching relationship is established. ,right The above process is executed sequentially, and each plane i corresponds to a unique plane. This yields three initial matching relationships.

[0037] Then, the bidirectional average point-to-plane distance in the three initial matching relationships is calculated respectively. :

[0038] Verify the correctness of the initial matching relationship. Given a distance threshold based on experience. ,like Then the initial matching relationship is considered to be verified correctly. If the match is invalid, the point cloud data needs to be collected again.

[0039] Based on the initial matching relationship, a set of planar feature matching pairs is established between the main lidar and the left sub-library lidar, wherein each matching pair Includes the i-th plane of the main lidar and the i-th plane of the left sub-library lidar The planar equations of each plane and the complete correspondence of their corresponding point clouds were obtained, resulting in three pairs of matching planar features. These were used for accurate extrinsic parameter calibration of the left sub-library radar relative to the main lidar. Similarly, planar feature matching pairs were established between the main lidar and the right sub-library radar. Similarly, three pairs of matching planar features were obtained for accurate extrinsic parameter calibration of the right secondary lidar relative to the main lidar.

[0040] S150, determine the external parameters to be optimized for the secondary lidar relative to the primary lidar. The external parameters to be optimized include rotation parameters and translation parameters. Based on the planar feature matching pair, rotation parameters, and translation parameters, determine the optimal rotation parameters and optimal translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system. Then, calibrate the secondary lidar based on the optimal rotation parameters and optimal translation parameters.

[0041] Specifically, taking the left sub-LiDAR as an example, the external parameters to be optimized for the left sub-LiDAR relative to the main LiDAR are defined, and the rotation parameter adopts a unit quaternion. This indicates that the unit constraint is satisfied simultaneously: Translation parameters are expressed as three-dimensional vectors. This means that the optimized variable vector is... ,in This indicates that the vector of variables to be optimized belongs to a 7-dimensional Euclidean space.

[0042] In some embodiments, the first planar feature includes a first planar equation and a first planar point cloud, and the matching planar feature includes a second planar equation and a second planar point cloud; Based on the planar feature matching pairs, rotation parameters, and translation parameters, the optimal rotation parameters and optimal translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system are determined, including: For each planar feature matching pair, using the first planar equation as the reference plane, each point in the second planar point cloud is projected onto the main lidar coordinate system using rotation and translation parameters. The point-to-plane residual from the projected point to the reference plane is calculated, and a cost function is constructed with the sum of squares of the point-to-plane residuals of all planar feature matching pairs as the objective. The initial extrinsic parameters are used as the initial values ​​for iteration to iteratively optimize the cost function until the cost function converges, thus obtaining the optimal rotation and translation parameters.

[0043] Specifically, for each matching pair in the set of planar feature matching pairs Based on the plane equation of the main lidar Using the reference plane (first plane equation), the corresponding point cloud in the left sub-laser radar coordinate system is... Each point in (Second plane point cloud), after rotation and translation parameters, is projected onto the main lidar coordinate system: ,in Let be the projection point. For quaternions The rotation matrix obtained after transformation.

[0044] Then, calculate the projection points. to reference plane Point-to-plane residual :

[0045] Right now

[0046] Next, construct a cost function that targets the sum of squared residuals from all matching pairs to the plane:

[0047] st

[0048] That is, the problem of finding the optimal extrinsic parameters can be expressed as: .

[0049] The Ceres Solver library is used to solve the above nonlinear least squares problem. An initial rotation matrix is ​​used... The corresponding quaternion and initial translation vector As the initial values ​​for iteration, the optimization continues iteratively until the cost function converges, ultimately yielding the optimal rotation and translation parameters of the left secondary lidar relative to the primary lidar. Optimal rotation parameters Optimal translation parameters m, i.e. .

[0050] Similarly, define the extrinsic parameters to be optimized for the right secondary lidar. Based on the set of matching pairs Each matching pair in the equation of the main lidar plane For the reference plane (first plane equation), the corresponding point cloud in the right sub-laser coordinate system Perform the same residual calculation and cost function construction as described above, and use the Ceres Solver library with the initial extrinsic parameters as the initial values ​​for iteration to solve the problem. Finally, obtain the optimal rotation parameters and optimal translation parameters of the right sub-library radar relative to the main lidar.

[0051] Optimal rotation parameters Optimal translation parameters m, .

[0052] In some embodiments, the secondary lidar is calibrated based on optimal rotation parameters and optimal translation parameters, including: For each plane feature matching pair, obtain the second plane point cloud in the matching plane features; transform each point in the second plane point cloud to the main lidar coordinate system using the optimal rotation parameters and optimal translation parameters to calibrate the secondary lidar.

[0053] In some embodiments, after calibrating the secondary lidar based on the optimal rotation parameters and the optimal translation parameters, the method further includes: For each planar feature matching pair, calculate the first average distance and first standard deviation of each point in the first planar point cloud to the reference plane; calculate the second average distance and second standard deviation of each point in the point cloud data obtained by the calibrated secondary lidar to the reference plane; verify the calibration results of the secondary lidar based on the first average distance, the second average distance and the preset average distance threshold, as well as the first standard deviation, the second standard deviation and the preset standard deviation threshold.

[0054] Specifically, taking the left lidar as an example, the optimal rotation and translation parameters of the left sub-library lidar are obtained through optimization. ,against For each pair of correspondences, the planar features of the left and right sub-LiDAR are taken. The corresponding point cloud subset , each of the points Transform to the main lidar coordinate system: , to obtain the transformed first A subset of planar point clouds ,right The above transformation is performed sequentially on the three matching pairs to obtain .

[0055] against Each pair of correspondences With the main lidar plane Using the reference plane, calculate the subset of the main lidar point cloud. The first average distance from each point p in the (first plane point cloud) to the reference plane and the first standard deviation :

[0056]

[0057] Then, the second average distance from each point in the point cloud data obtained from the calibrated left sub-laser scan to the reference plane is calculated. Second standard deviation : That is, to calculate the transformed left sub-laser point cloud subset. various points in the middle To the same reference plane Second average distance Second standard deviation

[0058]

[0059]

[0060] Then, based on the given calibration accuracy threshold... and ,right The difference in average distance between the three sets of corresponding relationships was calculated separately. Difference from standard deviation If all correspondences satisfy: and , Then the left secondary lidar is considered to have been successfully calibrated.

[0061] For example, a calibration accuracy threshold is given based on experience. =0.03m and =0.02m. Therefore, in this embodiment, the above verification was performed on three sets of matching pairs using the optimal rotation and translation parameters. The quantitative evaluation results are shown in Table 1. The values ​​of all matching pairs are... and All values ​​meet the set thresholds, indicating that the calibration of the left secondary lidar has been verified.

[0062] Table 1 Quantitative Evaluation Results of Calibration Accuracy of Left Sub-Lidar

[0063] Similarly, a calibration accuracy threshold is given based on experience. =0.03m and =0.02m. The same verification process was performed on the right secondary lidar, and the quantitative evaluation results are shown in Table 2. All matching pairs... and All values ​​meet the set thresholds, indicating that the calibration of the right secondary lidar has been verified.

[0064] Table 2 Quantitative Evaluation Results of Right Sub-LiDAR Calibration Accuracy

[0065] Finally, CloudCompare software can be used to simultaneously display the main LiDAR segmented point cloud, the calibrated left sub-radar segmented point cloud, and the right sub-radar segmented point cloud in different colors. Visually check the color overlap and alignment of the three point clouds in the same plane area. If the boundaries of the point clouds in each plane area match and there is no obvious misalignment, it will corroborate the quantitative evaluation results and complete the comprehensive evaluation of the calibration accuracy.

[0066] In summary, the method provided by this invention robustly extracts planar features using the RANSAC algorithm, effectively overcoming the problem of unstable localization of point features under noise interference. It utilizes the plane normal vector and the distance from the point to the plane to construct global geometric constraints, solving the problems of limited local constraint capabilities of point features and insufficient constraint on translational degrees of freedom for line features. Using indoor corner scenarios as the calibration environment eliminates the need for dedicated calibration equipment, improving the system's versatility and deployment flexibility. Experimental results show that it meets the requirements of autonomous driving applications, the optimized algorithm converges quickly and stably, and it has good practical value.

[0067] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of this application.

[0068] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0069] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0070] Figure 2 This is a schematic diagram illustrating the structure of a calibration device for multiple lidar systems, as shown in an exemplary embodiment of this application. Figure 2 As shown, the exemplary calibration device for multiple lidar systems includes: The acquisition module 210 is configured to acquire the first point cloud obtained by the main lidar scanning three mutually perpendicular planes and the second point cloud obtained by the secondary lidar scanning three mutually perpendicular planes. The conversion module 220 is configured to convert the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the main lidar. The extraction module 230 is configured to extract planar features from the first point cloud and the transformed second point cloud respectively, to obtain first planar features and second planar features corresponding to three mutually perpendicular planes; The matching module 240 is configured to perform matching on the second plane feature for each first plane feature to obtain the matching plane feature, and to construct a plane feature matching pair between each first plane feature and the matching plane feature; The calibration module 250 is configured to determine the external parameters to be optimized for the secondary lidar relative to the primary lidar. The external parameters to be optimized include rotation parameters and translation parameters. Based on the planar feature matching pairs, rotation parameters, and translation parameters, the optimal rotation parameters and optimal translation parameters from the secondary lidar coordinate system to the primary lidar coordinate system are determined, and the secondary lidar is calibrated based on the optimal rotation parameters and optimal translation parameters.

[0071] In some embodiments, the matching module 240 is further configured to, for each first planar feature, determine the angle between the plane normal vectors of the first and second planar features, select the second planar feature whose angle between the plane normal vectors is less than a preset angle threshold as the initial matching planar feature; determine the bidirectional average point-to-plane distance between the first planar feature and the initial matching planar feature, and if the bidirectional average point-to-plane distance is less than a preset distance threshold, then use the initial matching planar feature as the matching planar feature of the first planar feature.

[0072] In some embodiments, the first planar feature includes a first planar equation and a first planar point cloud, and the matching planar feature includes a second planar equation and a second planar point cloud; the calibration module 250 is further configured to, for each planar feature matching pair, use the first planar equation as a reference plane and utilize rotation and translation parameters to project each point in the second planar point cloud onto the main lidar coordinate system; calculate the point-to-plane residual from the projected point to the reference plane, and construct a cost function with the sum of squares of the point-to-plane residuals of all planar feature matching pairs as the objective; iteratively optimize the cost function using the initial extrinsic parameters as the initial values ​​for iteration until the cost function converges, thereby obtaining the optimal rotation parameters and the optimal translation parameters.

[0073] In some embodiments, the calibration module 250 is further configured to obtain a second planar point cloud in the matching planar features for each planar feature matching pair; and to transform each point in the second planar point cloud to the main lidar coordinate system using optimal rotation parameters and optimal translation parameters to calibrate the secondary lidar.

[0074] In some embodiments, the calibration module 250 is further configured to calculate, for each planar feature matching pair, a first average distance and a first standard deviation from each point in the first planar point cloud to the reference plane; calculate a second average distance and a second standard deviation from each point in the point cloud data obtained by the calibrated secondary lidar to the reference plane; and verify the calibration results of the secondary lidar based on the first average distance, the second average distance and a preset average distance threshold, as well as the first standard deviation, the second standard deviation and a preset standard deviation threshold.

[0075] In some embodiments, the conversion module 220 is further configured to obtain the installation angle of the secondary lidar relative to the primary lidar, and determine an initial rotation matrix based on the installation angle; obtain the spatial distance between the origin of the secondary lidar and the origin of the primary lidar, and determine an initial translation vector based on the spatial distance.

[0076] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the methods provided in the above embodiments.

[0077] Figure 3 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 3 The computer system 300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0078] like Figure 3 As shown, the computer system 300 includes a CPU 301, which can perform various appropriate actions and processes according to a program stored in ROM 302 or a program loaded from storage portion 308 into RAM 303, such as executing the methods in the above embodiments. The CPU 301 is a Central Processing Unit, ROM 302 is a Read-Only Memory, and RAM 303 is a Random Access Memory.

[0079] RAM 303 also stores various programs and data required for system operation. CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304. I / O interface 305 is an input / output interface.

[0080] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0081] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 303, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs various functions defined in the system of this application.

[0082] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0085] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0086] Another aspect of this application provides a vehicle including the aforementioned electronic equipment.

[0087] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various embodiments above.

[0088] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the steps of this application.

Claims

1. A calibration method for multiple lidar sensors, characterized in that, include: The first point cloud obtained by the main lidar scanning three mutually perpendicular planes and the second point cloud obtained by the secondary lidar scanning the three mutually perpendicular planes are acquired. Based on the initial extrinsic parameters of the secondary lidar relative to the primary lidar, the second point cloud is transformed to the coordinate system of the primary lidar. Planar features are extracted from the first point cloud and the transformed second point cloud respectively to obtain the first planar features and the second planar features corresponding to the three mutually perpendicular planes; For each first planar feature, the angle between the planar normal vector and the second planar feature is determined. The second planar feature whose angle between the planar normal vector and the second planar feature is less than a preset angle threshold is selected as the initial matching planar feature. The bidirectional average point-to-plane distance between the first planar feature and the initial matching planar feature is determined. If the bidirectional average point-to-plane distance is less than a preset distance threshold, the initial matching planar feature is used as the matching planar feature of the first planar feature, and a planar feature matching pair between each first planar feature and the matching planar feature is constructed. The extrinsic parameters to be optimized for the secondary lidar relative to the primary lidar are determined. These extrinsic parameters include rotation and translation parameters. The first planar feature includes a first planar equation and a first planar point cloud. The matching planar feature includes a second planar equation and a second planar point cloud. For each planar feature matching pair, using the first planar equation as a reference plane, each point in the second planar point cloud is projected onto the primary lidar coordinate system using the rotation and translation parameters. The point-to-plane residual from the projected point to the reference plane is calculated, and a cost function is constructed with the sum of squares of the point-to-plane residuals for all planar feature matching pairs as the objective. The initial extrinsic parameters are used as the initial values ​​for iteration to iteratively optimize the cost function until it converges, obtaining the optimal rotation and translation parameters. The secondary lidar is then calibrated based on the optimal rotation and translation parameters.

2. The method according to claim 1, characterized in that, The calibration of the secondary lidar based on the optimal rotation parameters and the optimal translation parameters includes: For each of the planar feature matching pairs, obtain the second planar point cloud from the matched planar features; Each point in the second planar point cloud is transformed to the main lidar coordinate system using the optimal rotation parameters and the optimal translation parameters in order to calibrate the secondary lidar.

3. The method according to claim 1 or 2, characterized in that, After calibrating the secondary lidar based on the optimal rotation parameters and the optimal translation parameters, the process further includes: For each of the planar feature matching pairs, calculate the first average distance and the first standard deviation from each point in the first planar point cloud to the reference plane; Calculate the second average distance and second standard deviation of each point in the point cloud data obtained by the calibrated secondary lidar scanning to the reference plane; The calibration results of the secondary lidar are verified based on the first average distance, the second average distance, and the preset average distance threshold, as well as the first standard deviation, the second standard deviation, and the preset standard deviation threshold.

4. The method according to claim 1, characterized in that, The initial extrinsic parameters include an initial rotation matrix and an initial translation vector. Before transforming the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the primary lidar, the process further includes: Obtain the installation angle of the secondary lidar relative to the primary lidar, and determine the initial rotation matrix based on the installation angle; Obtain the spatial distance between the origin of the secondary lidar and the origin of the primary lidar, and determine the initial translation vector based on the spatial distance.

5. A calibration device for multiple lidar systems, characterized in that, include: The acquisition module is configured to acquire a first point cloud obtained by the main lidar scanning three mutually perpendicular planes and a second point cloud obtained by the secondary lidar scanning the three mutually perpendicular planes; The conversion module is configured to convert the second point cloud to the coordinate system of the main lidar based on the initial extrinsic parameters of the secondary lidar relative to the main lidar. The extraction module is configured to extract planar features from the first point cloud and the transformed second point cloud respectively, to obtain first planar features and second planar features corresponding to the three mutually perpendicular planes; The matching module is configured to, for each first planar feature, determine the angle between the plane normal vectors of the first and second planar features, select the second planar feature whose angle between the plane normal vectors is less than a preset angle threshold as the initial matching planar feature; determine the bidirectional average point-to-plane distance between the first planar feature and the initial matching planar feature; if the bidirectional average point-to-plane distance is less than a preset distance threshold, then use the initial matching planar feature as the matching planar feature of the first planar feature, and construct a planar feature matching pair between each first planar feature and the matching planar feature; The calibration module is configured to determine the extrinsic parameters to be optimized for the secondary lidar relative to the primary lidar. These extrinsic parameters include rotation and translation parameters. The first planar feature includes a first planar equation and a first planar point cloud. The matching planar feature includes a second planar equation and a second planar point cloud. For each planar feature matching pair, using the first planar equation as a reference plane, each point in the second planar point cloud is projected onto the primary lidar coordinate system using the rotation and translation parameters. The point-to-plane residual from the projected point to the reference plane is calculated, and a cost function is constructed with the sum of squared point-to-plane residuals for all planar feature matching pairs as the objective. The initial extrinsic parameters are used as the initial values ​​for iteration to iteratively optimize the cost function until it converges, obtaining the optimal rotation and translation parameters. The secondary lidar is then calibrated based on the optimal rotation and translation parameters.

6. An electronic device, characterized in that, include: One or more processors and a memory, wherein a computer program is stored in the memory, and when the one or more processors execute the computer program, the device performs the steps of the method as described in any one of claims 1 to 4.

7. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 6.

Images