Camera and LiDAR pose calibration method and device based on point cloud registration

Through the point cloud registration method, the problem of insufficient accuracy of camera and lidar position calibration is solved, and a higher accuracy and robust camera-lidar position calibration is achieved, avoiding dependence on calibrators.

CN113625288BActive Publication Date: 2024-12-03INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202110661361.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-15
Publication Date
2024-12-03
Estimated Expiration
2041-06-15

AI Technical Summary

Technical Problem

The position calibration methods of existing cameras and lidars are not accurate, and there are internal and external parameter coupling errors in the calibration error of multiple sensors, which affects the calibration results.

Method used

Using the camera and lidar position calibration method based on point cloud registration, multiple cameras and lidars collect image sequences and laser point cloud data of preset scenes, perform sparse reconstruction and point cloud registration, and obtain lidar and Position calibration information between cameras and does not depend on the calibration object.

Benefits of technology

It improves the accuracy and robustness of the camera-lidar relative posture calibration results, reduces calibration errors, and enhances calibration accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113625288B_ABST
    Figure CN113625288B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for calibrating the pose of a camera and a laser radar based on point cloud registration, wherein the method comprises: based on multiple cameras and laser radars, respectively collecting image sequences and laser point cloud data of a preset scene; wherein the multiple cameras are rigidly fixed; performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; aligning the laser point cloud data with the sparse three-dimensional point cloud data to obtain pose calibration information between the laser radar and a reference camera among the multiple cameras, and determining pose calibration information between the laser radar and other cameras based on the relative pose information between the multiple cameras. The present invention can obtain higher-precision camera-laser radar relative pose calibration results and camera intrinsic parameter calibration results, and is independent of calibration objects, and has higher robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of sensor calibration technology, and in particular to a camera and laser radar posture calibration method and device based on point cloud registration. Background Art

[0002] In recent years, driverless technology has attracted more and more attention. Sensors play an important role in driverless technology. Currently, the most widely used sensors include cameras, lidar, GPS, and IMU. The fusion of multiple sensors can bring out the advantages of each sensor. Among them, the two most important sensors: cameras and lidar, have good complementary advantages.

[0003] When performing multi-sensor fusion, posture calibration is usually required. However, the current camera and lidar calibration methods require the placement of calibration objects in a specific environment in advance, and the placement angle and distance of the calibration objects will affect the calibration results. In addition, there is a time difference in the triggering of multiple sensors, and the relative postures between the sensors will change during this time difference, resulting in poor calibration accuracy. In addition, the calibration error of multiple sensors also has internal and external parameter coupling errors. Summary of the invention

[0004] The present invention provides a camera and laser radar posture calibration method and device based on point cloud registration, which are used to solve the defect of poor calibration accuracy in the prior art.

[0005] The present invention provides a camera and laser radar posture calibration method based on point cloud registration, comprising:

[0006] Based on multiple cameras and laser radars, image sequences and laser point cloud data of preset scenes are collected respectively; wherein the multiple cameras are rigidly fixed to each other;

[0007] Performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene;

[0008] The laser point cloud data is aligned with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and a reference camera among the multiple cameras, and based on the relative pose information between the multiple cameras, the pose calibration information between the laser radar and other cameras is determined.

[0009] According to the present invention, a camera and laser radar pose calibration method based on point cloud registration is provided, which is based on multiple cameras and laser radars, respectively collecting image sequences and laser point cloud data of preset scenes, and specifically includes:

[0010] The vehicle makes a figure-8 circle in the preset scene, stops at intervals during the circle, and collects data based on multiple cameras and laser radars at the time of parking to obtain the image sequence and the laser point cloud data;

[0011] Wherein, the multiple cameras and the laser radar are fixed on the vehicle.

[0012] According to a method for calibrating a camera and a laser radar based on point cloud registration provided by the present invention, the laser point cloud data is registered with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and a reference camera among the multiple cameras, specifically comprising:

[0013] Determine the pose calibration information between the laser radar and the reference camera based on the single pose calibration information of the laser radar and the reference camera at each parking moment;

[0014] Among them, the single pose calibration information of the laser radar and the reference camera at any parking moment is obtained by aligning the laser point cloud data collected by the laser radar at any parking moment with the sparse three-dimensional point cloud data.

[0015] According to a camera and laser radar pose calibration method based on point cloud registration provided by the present invention, the pose calibration information between the laser radar and the reference camera is determined based on the single pose calibration information of the laser radar and the reference camera at each parking moment, specifically comprising:

[0016] Taking the pose calibration average value calculated in the previous round as the initial value, determining the single pose calibration information of the laser radar and the reference camera at each parking moment in the current round, and calculating the pose calibration average value of the current round, until the variance of the single pose calibration information at each parking moment currently determined is not greater than a preset threshold or the maximum number of iterations is reached;

[0017] The posture calibration average value is the average value of the single posture calibration information at each parking moment in the corresponding round.

[0018] According to a camera and laser radar pose calibration method based on point cloud registration provided by the present invention, the single pose calibration information of the laser radar and the reference camera at any parking moment is determined based on the following steps:

[0019] Based on each laser 3D point in the laser point cloud data collected at any parking moment, determining an image 3D point in the sparse 3D point cloud data that matches each laser 3D point in the laser point cloud data collected at any parking moment;

[0020] Relative posture estimation is performed based on each laser three-dimensional point and its matched image three-dimensional point to obtain single posture calibration information at any parking moment.

[0021] According to a camera and lidar pose calibration method based on point cloud registration provided by the present invention, sparse reconstruction is performed based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene, which specifically includes:

[0022] Extracting features from each image in the image sequence respectively, and performing feature matching on every two images to obtain matching point pairs between every two images;

[0023] A matching point trajectory is constructed based on the matching point pairs between every two images, the matching point trajectory is triangulated, and a bundling adjustment is performed to reconstruct the sparse three-dimensional point cloud data.

[0024] According to a camera and lidar pose calibration method based on point cloud registration provided by the present invention, the reprojection error function used in the bundling adjustment process is:

[0025] E1=E 1,1 +E 1,2

[0026] Among them, E 1,1 is the reference camera C i1 The reprojection error function, E 1,2 For other cameras C ij , reprojection error function for j = 2, 3, 4;

[0027]

[0028]

[0029] Where k is the number of the three-dimensional point in the image, j is the number of the image, i is the number of the auxiliary camera, and x is the number of the image. j,k is the image feature point, K j is the camera internal parameter, R j and t j The transformation matrix from the image 3D point in space to the reference camera, and is the transformation matrix from the reference camera to other cameras, X k is the three-dimensional point of the image, and r represents the projection process.

[0030] The present invention also provides a camera and laser radar posture calibration device based on point cloud registration, comprising:

[0031] A data acquisition unit, used to respectively acquire image sequences and laser point cloud data of a preset scene based on a plurality of cameras and a laser radar; wherein the plurality of cameras are rigidly fixed to each other;

[0032] A sparse reconstruction unit, configured to perform sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene;

[0033] A point cloud registration unit is used to align the laser point cloud data with the sparse three-dimensional point cloud data, obtain the pose calibration information between the laser radar and a reference camera among the multiple cameras, and determine the pose calibration information between the laser radar and other cameras based on the relative pose information between the multiple cameras.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the camera and lidar pose calibration method based on point cloud registration as described in any one of the above are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the camera and lidar pose calibration method based on point cloud registration as described in any one of the above.

[0036] The camera and lidar pose calibration method and device based on point cloud registration provided by the present invention obtain sparse three-dimensional point cloud data by sparsely reconstructing image sequences collected by multiple cameras, and perform point cloud registration on the laser point cloud data and the sparse three-dimensional point cloud data to obtain pose calibration information between the lidar and the camera. Higher-precision camera-lidar relative pose calibration results and camera intrinsic parameter calibration results can be obtained, and they are independent of calibration objects and have higher robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of the process of the camera and lidar pose calibration method based on point cloud registration provided by the present invention;

[0039] Figure 2 A schematic diagram of a generalized camera model provided by the present invention;

[0040] Figure 3A schematic diagram of the 8-shaped orbiting motion provided by the present invention;

[0041] Figure 4 A schematic diagram of a camera and laser radar posture calibration method provided by the present invention;

[0042] Figure 5 A schematic diagram of the structure of a camera and laser radar posture calibration device based on point cloud registration provided by the present invention;

[0043] Figure 6 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Figure 1 A schematic diagram of a process flow of a camera and lidar pose calibration method based on point cloud registration provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0046] Step 110, based on multiple cameras and laser radars, respectively collect image sequences and laser point cloud data of a preset scene; wherein the multiple cameras are rigidly fixed to each other;

[0047] Step 120, performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of a preset scene;

[0048] Step 130, align the laser point cloud data with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and a reference camera among multiple cameras, and determine the pose calibration information between the laser radar and other cameras based on the relative pose information between the multiple cameras.

[0049] Specifically, multiple cameras and laser radars are used for data collection, wherein the multiple cameras are used to collect image sequences of preset scenes, and the laser radar is used to collect laser point cloud data of preset scenes. Here, in order to improve the accuracy of the calibration method, scenes with rich textures and geometric structures can be selected for data collection, such as intersections, thereby providing rich scene structures, which can provide rich texture information for image feature extraction.

[0050] It should be noted that the multiple cameras are rigidly fixed. Here, the embodiment of the present invention uses a generalized camera model. The generalized camera model means that the cameras are rigidly fixed, so the relative positions of the cameras do not change. Figure 2 A schematic diagram of a generalized camera model provided by an embodiment of the present invention, such as Figure 2 As shown, a camera is selected from multiple cameras as a reference camera, such as Figure 2 C i1 is the reference camera, and the other cameras are C ij , j = 2, 3, 4 represents the camera number. The error function uses the reprojection error function, camera C ij The pose of the reference camera C i1 The above optimization model setting can ensure that the relative pose between cameras remains unchanged in the subsequent sparse reconstruction optimization process, which conforms to the objective physical model.

[0051] After data collection is completed, sparse reconstruction can be performed based on the image sequences collected by multiple cameras to obtain sparse 3D point cloud data of the preset scene. Among them, incremental sparse reconstruction can be used for sparse reconstruction. If there is a good global initial pose, a global sparse reconstruction method can also be selected.

[0052] Subsequently, the laser point cloud data is aligned with the sparse three-dimensional point cloud data to obtain the position and pose of the laser radar in the sparse three-dimensional point cloud data, and the relative position and pose between the reference camera and the laser radar in the multiple cameras is calculated based on this, and the position and pose calibration information between the laser radar and the reference camera in the multiple cameras is obtained. Since the relative position and pose between the multiple cameras is fixed, the position and pose calibration information between the laser radar and the other cameras can be determined based on the position and pose calibration information between the laser radar and the reference camera in the multiple cameras, and the relative position and pose information between the multiple cameras. Among them, the position and pose calibration information between the laser radar and any camera contains the relative relationship between the laser radar and the position and pose of the camera.

[0053] The method provided by the embodiment of the present invention performs sparse reconstruction on image sequences captured by multiple cameras to obtain sparse three-dimensional point cloud data, and performs point cloud registration on the laser point cloud data and the sparse three-dimensional point cloud data to obtain the posture calibration information between the lidar and the camera. A higher-precision camera-lidar relative posture calibration result and a camera intrinsic parameter calibration result can be obtained, and it does not rely on the calibration object and has higher robustness.

[0054] Based on the above embodiment, step 110 specifically includes:

[0055] The vehicle makes a figure-of-eight circle in the preset scene, stops at intervals during the circle, and collects data based on multiple cameras and lidar at the time of parking to obtain image sequences and laser point cloud data;

[0056] Among them, multiple cameras and lidars are fixed on the vehicle.

[0057] Specifically, since each camera in the multi-camera system has a different orientation, there is no overlapping area between the perspectives of the multiple cameras during data collection. In order to improve the accuracy of sparse reconstruction, the embodiment of the present invention installs a multi-camera acquisition device and a laser radar on a vehicle, such as an unmanned vehicle, and sets the vehicle to collect data in a preset scene in an "8" surround manner, so that there is an overlapping area between the perspectives of the multiple cameras, thereby increasing the constraints between different cameras during optimization, bundling multiple cameras together for optimization, and obtaining more stable and accurate camera parameters.

[0058] Figure 3 A schematic diagram of the 8-shaped orbiting motion provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown in the figure, the vehicle carrying multi-camera acquisition equipment starts from the starting position. At this time, camera 1 can collect data from the "+" area on the right. As the position of the multi-camera system acquisition equipment changes, camera 2 collects data from the "+" area. In the process of continuous collection, camera 3 and camera 4 will also collect data from the "+" area. Therefore, when the vehicle carrying multi-camera system collects data in the way of "8", the overlapping area of ​​multiple cameras can be obtained, which improves the accuracy of sparse reconstruction.

[0059] Secondly, the vehicle can stop at intervals during the data collection process, and collect data based on multiple cameras and lidars at the parking moment to obtain image sequences and laser point cloud data. The intermittent parking method can ensure that the spatial posture does not change when different sensors collect data, and there will be no posture changes between multiple sensors due to asynchronous triggering. At this time, the phase position relationship between multiple sensors is fixed (rigid).

[0060] In addition, since circling the figure of 8 is also a common way of driving for vehicles when calibrating inertial navigation equipment, one data collection can simultaneously realize the calibration of multiple sensors without adding too much extra data collection workload. Moreover, data collection is carried out in a smaller scenario, which reduces the amount of data and the amount of calculation for sparse reconstruction.

[0061] Based on any of the above embodiments, the laser point cloud data is registered with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and the reference camera among the multiple cameras, specifically including:

[0062] Based on the single pose calibration information of the laser radar and the reference camera at each parking moment, determine the pose calibration information between the laser radar and the reference camera;

[0063] Among them, the single pose calibration information of the laser radar and the reference camera at any parking moment is obtained by aligning the laser point cloud data collected by the laser radar at the parking moment with the sparse three-dimensional point cloud data.

[0064] Specifically, during the data collection process, the vehicle will stop at intervals, and multiple sensors will collect data at each parking moment. Therefore, for any parking moment, the laser point cloud data collected by the laser radar at the parking moment can be aligned with the sparse three-dimensional point cloud data to obtain the single pose calibration information of the laser radar and the reference camera at the parking moment. Among them, the single pose calibration information includes the relative pose between the laser radar and the corresponding camera, including the rotation matrix and the translation vector. During the alignment, the correspondence between the initial point pairs between the laser point cloud data p and the sparse three-dimensional point cloud data p' can be obtained according to the nearest neighbor. Assume that the i-th point p of the laser point cloud data p at this time i , the corresponding point in the sparse 3D point cloud data p' is p i '. Then the transformation relationship between the two sets of point clouds, that is, the single pose calibration information, can be solved by minimizing the following formula:

[0065]

[0066] Among them, (R, t) is the single pose calibration information, R is the rotation matrix, and t is the translation vector.

[0067] Using the same registration method, the single pose calibration information corresponding to each parking moment can be obtained. Since the single pose calibration information calculated at different parking moments will not be strictly equal, and due to factors such as matching errors, the single pose calibration information of some parking moments may contain large deviations. Therefore, in order to obtain stable and consistent calibration results, the single pose calibration information corresponding to each parking moment can be averaged, and the average value can be used as the pose calibration information between the lidar and the reference camera.

[0068] Based on any of the above embodiments, based on the single pose calibration information of the laser radar and the reference camera at each parking moment, determining the pose calibration information between the laser radar and the reference camera specifically includes:

[0069] The pose calibration average value calculated in the previous round is used as the initial value to determine the single pose calibration information of the lidar and the reference camera at each parking moment in the current round, and the pose calibration average value of the current round is calculated until the variance of the single pose calibration information at each parking moment currently determined is no greater than the preset threshold or the maximum number of iterations is reached;

[0070] The average value of the posture calibration is the average value of the single posture calibration information at each parking moment in the corresponding round.

[0071] Specifically, when there is a large registration error (such as a large initial laser radar position error, etc.), the fluctuation between the single pose calibration information at each parking moment will be relatively large. Therefore, the variance of the single pose calibration information at each parking moment can be used to measure the stability of the current solution. If the variance of the single pose calibration information at each parking moment in the previous round is large, the pose calibration average value calculated in the previous round can be used as the initial value to update all laser radar poses, and the single pose calibration information of the laser radar and the reference camera at each parking moment in the current round is iteratively determined, and the pose calibration average value of the current round is calculated, until the variance of the single pose calibration information at each parking moment currently determined is not greater than the preset threshold (for example, 10 -6 ) or reaches the maximum number of iterations, usually 3-5 times to converge.

[0072] Among them, the posture calibration average value is the average value of the single posture calibration information of each parking moment of the corresponding round. Since the single posture calibration information includes the rotation matrix and the translation vector, the average value of the rotation matrix and the average value of the translation vector can be calculated respectively. In order to calculate the average value of the rotation matrix R, the rotation matrix can be converted into an axis-angle form, and then the axis-angle representation of all rotation matrices can be averaged. After obtaining the mean value in the axis-angle form, it is converted into a rotation matrix form through the Rogrigius formula to obtain the average value of the rotation matrix. For the translation vector t, all translation vectors can be directly averaged.

[0073] Based on any of the above embodiments, the single pose calibration information of the laser radar and the reference camera at any parking moment is determined based on the following steps:

[0074] Based on each laser 3D point in the laser point cloud data collected at the parking moment, determine an image 3D point in the sparse 3D point cloud data that matches each laser 3D point in the laser point cloud data collected at the parking moment;

[0075] Relative pose estimation is performed based on each laser 3D point and its matching image 3D point to obtain the single pose calibration information at the parking moment.

[0076] Specifically, the iterative closest point algorithm (ICP) can be used for point cloud registration. Specifically, for a certain laser 3D point in the laser point cloud data collected at the parking moment, the Euclidean distance between it and all the image 3D points in the sparse 3D point cloud data is calculated, and the image 3D point with the closest Euclidean distance is taken as the matching point of the laser 3D point. Repeat the above steps to obtain the image 3D points in the sparse 3D point cloud data that match each laser 3D point in the laser point cloud data, forming two sets of matching point clouds.

[0077] Relative pose estimation is performed based on each laser 3D point and its matching image 3D point to obtain the single pose calibration information of the parking moment. Among them, the relative pose estimation method can adopt the relative pose estimation method based on RANSAC. The RANSAC algorithm is used to solve the rotation matrix R and translation vector t between the laser radar and the reference camera pose at the parking moment, perform the laser point spatial pose transformation, and reconstruct and calculate the error of the two sets of point clouds.

[0078] If the error is reduced significantly compared to the previous iteration or exceeds the maximum number of iterations, the iteration is terminated. Otherwise, the above three-dimensional point matching and relative pose estimation steps are iterated with the calculation result of this iteration as the initial value.

[0079] Based on any of the above embodiments, step 120 specifically includes:

[0080] Extract features from each image in the image sequence respectively, and perform feature matching between every two images to obtain matching point pairs between every two images;

[0081] The matching point trajectory is constructed based on the matching point pairs between every two images, the matching point trajectory is triangulated, and the bundle adjustment is performed to reconstruct the sparse 3D point cloud data.

[0082] Specifically, the SfM (Structured From Motion) algorithm can be used for sparse reconstruction. Specifically, feature extraction can be performed on each image in the image sequence, key points with stable properties in the image can be found as feature points, and feature matching can be performed on every two images to obtain matching point pairs between every two images. Subsequently, a matching point trajectory is constructed based on the matching point pairs between every two images, the matching point trajectory is triangulated, and bundled adjustment is performed to reconstruct sparse 3D point cloud data.

[0083] Based on any of the above embodiments, the reprojection error function used in the bundling adjustment process is:

[0084] E1=E 1,1 +E 1,2

[0085] Among them, E 1,1 is the reference camera C i1 The reprojection error function, E 1,2 For other cameras C ij , reprojection error function for j = 2, 3, 4;

[0086]

[0087]

[0088] Where k is the number of the three-dimensional point in the image, j is the number of the image, i is the number of the auxiliary camera, and x is the number of the image. j,k is the image feature point, K j is the camera internal parameter, R j and t j The transformation matrix from the image 3D point in space to the reference camera, and is the transformation matrix from the reference camera to other cameras, X k is the three-dimensional point of the image, and r represents the projection process.

[0089] Based on any of the above embodiments, Figure 4 A schematic diagram of a camera and laser radar pose calibration method provided by an embodiment of the present invention, such as Figure 4 As shown, the method includes: an SfM sparse reconstruction step and an ICP registration step. The SfM sparse reconstruction step includes feature extraction, feature matching, matching point trajectory triangulation and bundle adjustment steps, so as to obtain the intrinsic parameters and posture of each camera in the multi-camera system, as well as the reconstructed three-dimensional point cloud. The ICP registration step includes ICP for the image and laser point cloud at each parking moment, that is, ICP registration of the laser point cloud at each parking moment with the reconstructed three-dimensional point cloud, to obtain the external parameter calibration results corresponding to each parking moment, the external parameter calibration results are the posture calibration information between the laser radar and the reference camera at the corresponding moment, and the average of multiple external parameter calibration results, and iterative solution until convergence.

[0090] Based on any of the above embodiments, Figure 5 A schematic diagram of the structure of a camera and laser radar pose calibration device based on point cloud registration provided by an embodiment of the present invention, such as Figure 5 As shown, the device includes: a data acquisition unit 510, a sparse reconstruction unit 520 and a point cloud registration unit 530.

[0091] The data acquisition unit 510 is used to respectively acquire image sequences and laser point cloud data of a preset scene based on multiple cameras and laser radars; wherein the multiple cameras are rigidly fixed to each other;

[0092] The sparse reconstruction unit 520 is used to perform sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of a preset scene;

[0093] The point cloud registration unit 530 is used to align the laser point cloud data with the sparse three-dimensional point cloud data, obtain the pose calibration information between the laser radar and a reference camera among multiple cameras, and determine the pose calibration information between the laser radar and other cameras based on the relative pose information between the multiple cameras.

[0094] The device provided by the embodiment of the present invention performs sparse reconstruction on image sequences acquired by multiple cameras to obtain sparse three-dimensional point cloud data, and performs point cloud registration on the laser point cloud data and the sparse three-dimensional point cloud data to obtain the posture calibration information between the lidar and the camera. It can obtain higher-precision camera-lidar relative posture calibration results and camera intrinsic parameter calibration results, and is independent of calibration objects and has higher robustness.

[0095] Based on any of the above embodiments, the data acquisition unit 510 is specifically used for:

[0096] The vehicle makes a figure-of-eight circle in the preset scene, stops at intervals during the circle, and collects data based on multiple cameras and lidar at the time of parking to obtain image sequences and laser point cloud data;

[0097] Among them, multiple cameras and lidars are fixed on the vehicle.

[0098] Based on any of the above embodiments, the laser point cloud data is registered with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and the reference camera among the multiple cameras, specifically including:

[0099] Based on the single pose calibration information of the laser radar and the reference camera at each parking moment, determine the pose calibration information between the laser radar and the reference camera;

[0100] Among them, the single pose calibration information of the laser radar and the reference camera at any parking moment is obtained by aligning the laser point cloud data collected by the laser radar at the parking moment with the sparse three-dimensional point cloud data.

[0101] Based on any of the above embodiments, based on the single pose calibration information of the laser radar and the reference camera at each parking moment, determining the pose calibration information between the laser radar and the reference camera specifically includes:

[0102] The pose calibration average value calculated in the previous round is used as the initial value to determine the single pose calibration information of the lidar and the reference camera at each parking moment in the current round, and the pose calibration average value of the current round is calculated until the variance of the single pose calibration information at each parking moment currently determined is no greater than the preset threshold or the maximum number of iterations is reached;

[0103] The average value of the posture calibration is the average value of the single posture calibration information at each parking moment in the corresponding round.

[0104] Based on any of the above embodiments, the single pose calibration information of the laser radar and the reference camera at any parking moment is determined based on the following steps:

[0105] Based on each laser 3D point in the laser point cloud data collected at the parking moment, determine an image 3D point in the sparse 3D point cloud data that matches each laser 3D point in the laser point cloud data collected at the parking moment;

[0106] Relative pose estimation is performed based on each laser 3D point and its matching image 3D point to obtain the single pose calibration information at the parking moment.

[0107] Based on any of the above embodiments, the sparse reconstruction unit 520 is specifically used for:

[0108] Extract features from each image in the image sequence respectively, and perform feature matching between every two images to obtain matching point pairs between every two images;

[0109] The matching point trajectory is constructed based on the matching point pairs between every two images, the matching point trajectory is triangulated, and the bundle adjustment is performed to reconstruct the sparse 3D point cloud data.

[0110] Based on any of the above embodiments, the reprojection error function used in the bundling adjustment process is:

[0111] E1=E 1,1 +E 1,2

[0112] Among them, E 1,1 is the reference camera C i1 The reprojection error function, E 1,2 For other cameras C ij , reprojection error function for j = 2, 3, 4;

[0113]

[0114]

[0115] Where k is the number of the three-dimensional point in the image, j is the number of the image, i is the number of the auxiliary camera, and x is the number of the image. j,k is the image feature point, K j is the camera internal parameter, R j and t j The transformation matrix from the image 3D point in space to the reference camera, and is the transformation matrix from the reference camera to other cameras, X k is the three-dimensional point of the image, and r represents the projection process.

[0116] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the camera and laser radar pose calibration method based on point cloud registration, the method comprising: based on multiple cameras and laser radars, respectively collecting image sequences and laser point cloud data of a preset scene; wherein the multiple cameras are rigidly fixed; performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; registering the laser point cloud data with the sparse three-dimensional point cloud data to obtain pose calibration information between the laser radar and a reference camera among the multiple cameras, and determining pose calibration information between the laser radar and other cameras based on the relative pose information between the multiple cameras.

[0117] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the camera and lidar pose calibration method based on point cloud registration provided by the above methods, the method including: based on multiple cameras and lidars, respectively collecting image sequences and laser point cloud data of preset scenes; wherein the multiple cameras are rigidly fixed; sparse reconstruction is performed based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; the laser point cloud data is registered with the sparse three-dimensional point cloud data to obtain the pose calibration information between the lidar and a reference camera among the multiple cameras, and based on the relative pose information between the multiple cameras, the pose calibration information between the lidar and other cameras is determined.

[0119] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned camera and lidar pose calibration methods based on point cloud registration, the method comprising: based on multiple cameras and lidars, respectively collecting image sequences and laser point cloud data of preset scenes; wherein the multiple cameras are rigidly fixed; performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; aligning the laser point cloud data with the sparse three-dimensional point cloud data to obtain pose calibration information between the lidar and a reference camera among the multiple cameras, and determining the pose calibration information between the lidar and other cameras based on the relative pose information between the multiple cameras.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camera and lidar pose calibration method based on point cloud registration, characterized in that: include: Based on multiple cameras and laser radars, image sequences and laser point cloud data of preset scenes are collected respectively; wherein the multiple cameras are rigidly fixed to each other; Performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; The laser point cloud data at multiple moments are respectively registered with the sparse three-dimensional point cloud data to obtain single pose calibration information between the laser radar and a reference camera among the multiple cameras at each moment; Based on the single pose calibration information at each moment, the pose calibration information between the laser radar and a reference camera among the multiple cameras is determined, and based on the relative pose information between the multiple cameras, the pose calibration information between the laser radar and other cameras is determined.

2. The camera and laser radar pose calibration method based on point cloud registration according to claim 1, characterized in that: The method of collecting image sequences and laser point cloud data of preset scenes based on multiple cameras and laser radars specifically includes: The vehicle makes a figure-8 circle in the preset scene, stops at intervals during the circle, and collects data based on multiple cameras and laser radars at the time of parking to obtain the image sequence and the laser point cloud data; Wherein, the multiple cameras and the laser radar are fixed on the vehicle.

3. The camera and laser radar pose calibration method based on point cloud registration according to claim 2, characterized in that: The registering the laser point cloud data with the sparse three-dimensional point cloud data to obtain the pose calibration information between the laser radar and a reference camera among the multiple cameras specifically includes: Determine the pose calibration information between the laser radar and the reference camera based on the single pose calibration information of the laser radar and the reference camera at each parking moment; Among them, the single pose calibration information of the laser radar and the reference camera at any parking moment is obtained by aligning the laser point cloud data collected by the laser radar at any parking moment with the sparse three-dimensional point cloud data.

4. The camera and laser radar pose calibration method based on point cloud registration according to claim 3, characterized in that: The determining of the pose calibration information between the laser radar and the reference camera based on the single pose calibration information of the laser radar and the reference camera at each parking moment specifically includes: Taking the pose calibration average value calculated in the previous round as the initial value, determining the single pose calibration information of the laser radar and the reference camera at each parking moment in the current round, and calculating the pose calibration average value of the current round, until the variance of the single pose calibration information at each parking moment currently determined is not greater than a preset threshold or the maximum number of iterations is reached; The posture calibration average value is the average value of the single posture calibration information at each parking moment in the corresponding round.

5. The camera and laser radar pose calibration method based on point cloud registration according to claim 3, characterized in that: The single pose calibration information of the laser radar and the reference camera at any parking moment is determined based on the following steps: Based on each laser 3D point in the laser point cloud data collected at any parking moment, determining an image 3D point in the sparse 3D point cloud data that matches each laser 3D point in the laser point cloud data collected at any parking moment; Relative posture estimation is performed based on each laser three-dimensional point and its matched image three-dimensional point to obtain single posture calibration information at any parking moment.

6. The camera and laser radar pose calibration method based on point cloud registration according to claim 1, characterized in that: The performing sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene specifically includes: Extracting features from each image in the image sequence respectively, and performing feature matching on every two images to obtain matching point pairs between every two images; A matching point trajectory is constructed based on the matching point pairs between every two images, the matching point trajectory is triangulated, and a bundling adjustment is performed to reconstruct the sparse three-dimensional point cloud data.

7. The camera and laser radar pose calibration method based on point cloud registration according to claim 6, characterized in that: The reprojection error function used in the bundling adjustment process is: E1=E 1,1 +E 1,2 Among them, E 1,1 is the reference camera C i1 The reprojection error function, E 1,2 For other cameras C ij , reprojection error function for j = 2, 3, 4; Where k is the number of the three-dimensional point in the image, j is the number of the image, i is the number of the auxiliary camera, and x is the number of the image. j,k is the image feature point, K j is the camera internal parameter, R j and t j The transformation matrix from the image 3D point in space to the reference camera, and is the transformation matrix from the reference camera to other cameras, X k is the three-dimensional point of the image, and r represents the projection process.

8. A camera and laser radar pose calibration device based on point cloud registration, characterized in that: include: A data acquisition unit, used to respectively acquire image sequences and laser point cloud data of a preset scene based on a plurality of cameras and a laser radar; wherein the plurality of cameras are rigidly fixed to each other; A sparse reconstruction unit, configured to perform sparse reconstruction based on the image sequence to obtain sparse three-dimensional point cloud data of the preset scene; A point cloud registration unit, used to respectively register the laser point cloud data at multiple moments with the sparse three-dimensional point cloud data, to obtain single pose calibration information between the laser radar and a reference camera among the multiple cameras at each moment; Based on the single pose calibration information at each moment, the pose calibration information between the laser radar and a reference camera among the multiple cameras is determined, and based on the relative pose information between the multiple cameras, the pose calibration information between the laser radar and other cameras is determined.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the camera and lidar pose calibration method based on point cloud registration as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the camera and lidar pose calibration method based on point cloud registration as described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Laser radar and camera combined external parameter calibration system and method

    CN114994647A